学术顶刊

QILIN SUN, King Abdullah University of Science and Technology, Saudi Arabia, Point Spread Technology, China

CONGLI WANG, King Abdullah University of Science and Technology, Saudi Arabia

QIANG FU, King Abdullah University of Science and Technology, Saudi Arabia

XIONG DUN, Point Spread Technology, China, Tongji University, China

WOLFGANG HEIDRICH, King Abdullah University of Science and Technology, Saudi Arabia

Qilin Sun1 Ethan Tseng2 Qiang Fu1 Wolfgang Heidrich1 Felix Heide2 1KAUST 2Princeton University

Abstract:

High-dynamic-range (HDR) imaging is an essential

imaging modality for a wide range of applications in uncontrolled environments, including autonomous driving,

robotics, and mobile phone cameras. However, exist�ing HDR techniques in commodity devices struggle with

dynamic scenes due to multi-shot acquisition and post-processing time, e.g. mobile phone burst photography, making such approaches unsuitable for real-time applications.

In this work, we propose a method for snapshot HDR imag�ing by learning an optical HDR encoding in a single image

which maps saturated highlights into neighboring unsatu-rated areas using a diffractive optical element (DOE). We

propose a novel rank-1 parameterization of the DOE which

drastically reduces the optical search space while allowing

us to efficiently encode high-frequency detail. We propose a

reconstruction network tailored to this rank-1 parametrization for the recovery of clipped information from the encoded measurements. The proposed end-to-end framework

is validated through simulation and real-world experiments

and improves the PSNR by more than 7 dB over state-of-the-art end-to-end designs.

Qilin Sun

Abstract:

Imaging systems have long been designed in separated steps: the experience-driven optical design followed by sophisticated image processing. Such a general-propose approach achieves success in the past but left the question open for specific tasks and the best compromise between optics and post-processing, as well as minimizing costs. Driven by this, a series of works are proposed to bring the imaging system design into end-to-end fashion step by step, from joint optics design, point spread function (PSF) optimization, phase map optimization to a general end-to-end complex lens camera.

Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang

Abstract

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking.

Wenbo Bao , Student Member, IEEE, Xiaoyun Zhang, Member, IEEE,

Li Chen, Member, IEEE, Lianghui Ding, and Zhiyong Gao

Abstract:

This paper proposes a novel frame rate-up conversion method through high-order model and dynamic filtering (HOMDF) for video pixels. Unlike the constant brightness and linear motion assumptions in traditional methods, the intensity and position of the video pixels are both modeled with high-order polynomials in terms of time. Then, the key problem of our method is to estimate the polynomial coefficients that represent the pixel’s intensity variation, velocity, and acceleration. We propose to solve it with two energy objectives: one minimizes the auto-regressive prediction error of intensity variation by its past samples, and the other minimizes the video frame’s reconstruction error along the motion trajectory. To efficiently address the optimization problem for these coefficients, we propose a dynamic filtering solution inspired by the video’s temporal coherence. The optimal estimation of these coefficients is reformulated into a dynamic fusion of the prior estimate from the pixel’s temporal predecessor and the maximum likelihood estimate from the current new observation. Finally, frame rate up-conversion is implemented using motion-compensated interpolation by pixel-wise intensity variation and motion trajectory.

包文博

摘要：

近年来随着高品质显示设备的快速发展，为它们提供具有更高时

间和空间分辨率的高质量视频源成为学术界和工业界迫在眉睫的研究

课题。然而受制于高质量视频对视频采集、编码、传输、解码等各阶段

所需要的计算能力和传输带宽的苛刻要求，将现有海量的中低质量视

频通过数字信号处理的方式合成转换为高质量视频成为最可行的方案

之一。在视频质量提升的研究中，从时间域上对视频的帧率进行提升是

其中最具挑战的问题之一，也是最能直接给用户带来沉浸式视觉体验

的关键技术。具体来讲，视频帧率上变换的目标就是在低帧率如 30Hz

的视频源中插入一些额外的图像帧而得到高帧率如 60Hz 或 120Hz 的

高帧率视频。插入的额外的视频图像帧能够使帧间物体位移更精细，物

体运动过渡的视觉效应更为平滑，从而显著改善运动视频的观赏体验。