High-fidelity Image and Video Restoration and Enhancement by Recovering RAW Sensor Data

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "High-fidelity Image and Video Restoration and Enhancement by Recovering 
RAW Sensor Data"

By

Mr. Yazhou XING


Abstract:

Image and video restoration and enhancement have long posed significant 
challenges in computer vision and computational photography. The task of 
recovering high-fidelity images and videos from corrupted or low-quality pure 
RGB signals is highly complex and ill-posed. Alternatively, leveraging camera 
RAW sensor data, which captures unprocessed signals with a linear relationship 
to scene irradiance and typically ranges from 12 to 14 bits, can greatly 
enhance the performance of restoration and enhancement tasks. Accessing RAW 
data, however, can be quite hard due to their memory-demanding property: RAW 
images may be discarded during the process of data storing, transferring, and 
sharing.

This thesis aims to close this research gap by the recovery of RAW data for 
robust image and video enhancement and restoration. We begin this thesis with a 
general solution to recover RAW sensor data from sRGB images, dubbed Invertible 
Image Signal Processing (InvISP). Unlike synthesizing RAW data from sRGB 
images, our innovative InvISP enables the rendering of visually appealing sRGB 
images while also facilitating the recovery of nearly perfect RAW data.

Then, we study another fundamental problem in RAW data recovery: high dynamic 
range (HDR) videos reconstruction. We present an online learning-based system 
designed to reduce overexposure artifacts in HDR video imaging. Our system 
leverages the temporal instabilities of autoexposure, eliminating the need for 
complex acquisition mechanisms such as alternating exposures or costly 
processing commonly associated with HDR imaging.

Lastly, we explore the effect of a special form of raw images, uncorrupted 
complete backgrounds, for the realistic compositing of portrait photographs or 
videos. By unifying foreground alpha matte generation and post-blending 
harmonization, we enable the realistic composition of portrait images and 
deliver temporally stable results in videos.


Date:                   Thursday, 11 April 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 5501
                        Lifts 25/26

Chairman:               Prof. Shiheng WANG (ACCT)

Committee Members:      Prof. Qifeng CHEN (Supervisor)
                        Prof. Pedro SANDER
                        Prof. Long CHEN
                        Prof. Ling SHI (ECE)
                        Prof. Jinwei GU (CUHK)