HIGH-QUALITY IMAGE AND VIDEO RESTORATION AND ENHANCEMENT BY MINING RAW SENSOR DATA

PhD Thesis Proposal Defence


Title: "HIGH-QUALITY IMAGE AND VIDEO RESTORATION AND ENHANCEMENT BY MINING RAW
SENSOR DATA"

by

Mr. Yazhou XING


Abstract:

Image and video restoration and enhancement have long posed significant
challenges in the fields of computer vision and computational photography. The
task of recovering high-fidelity images and videos from corrupted or
low-quality pure RGB signals is complex and ill-posed. However, leveraging
camera RAW sensor data, which captures unprocessed signals with a linear
relationship to scene irradiance and typically ranges from 12 to 14 bits, can
greatly enhance the performance of restoration and enhancement tasks.

This thesis aims to extend existing solutions for image and video restoration
and enhancement by focusing on the recovery of RAW sensor data. Firstly, we
propose an Invertible Image Signal Processing (InvISP) pipeline that accurately
recovers high-fidelity RAW sensor data from sRGB images. Unlike synthesizing
RAW data from sRGB images, our innovative approach enables the rendering of
visually appealing sRGB images while also facilitating the recovery of nearly
perfect RAW data.

Secondly, we present a learning-based system designed to reduce overexposure
artifacts in high dynamic range (HDR) imaging. Our system leverages the
temporal instabilities of autoexposure, eliminating the need for complex
acquisition mechanisms such as alternating exposures or costly processing
commonly associated with HDR imaging.

Lastly, we explore the realistic compositing of portrait photographs or videos
onto raw input backgrounds. By unifying foreground alpha matte generation and
post-blending harmonization, we enable the realistic composition of portrait
images and deliver temporally stable results in videos.

Through these proposed solutions, we aim to advance the field of image and
video restoration and enhancement by leveraging the power of RAW sensor data.
Our contributions include the development of an Invertible Image Signal
Processing pipeline, a learning-based system for reducing overexposure
artifacts, and techniques for realistic compositing of portrait photographs or
videos.


Date:                   Friday, 2 February 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 5501
                        Lifts 25/26

Committee Members:      Dr. Qifeng Chen (Supervisor)
                        Prof. Chiew-Lan Tai (Chairperson)
                        Prof. Pedro Sander
                        Dr. Dan Xu