Intelligent-ISP for Video Enhancement under Extreme Light Conditions (ARD/259)

The image sensor market is a rapidly growing market with a compound annual growth rate of about 12% from 2018 to 2025, with the largest growth in the automotive industry at about 36%. The total revenue of image sensors in 2018 is approximately $13.7 billion. Video/image quality under extreme lighting conditions (such as dim light, large dynamic range (WDR)) is an important technical indicator in surveillance, drone and advanced driver assistance systems (ADAS) but is not well addressed. In addition, the fixed pipelines, separate functional modules and limited functions in the existing ISP (Image Signal Processor) require a lot of manpower and time in system tuning, and it is difficult to adapt to rapidly developing applications. In order to tackle the current technical difficulties, we propose a learning-based ConvNet to combine the key functional modules of traditional ISP and application-driven enhancement modules (such as video image enhancement under extreme lighting conditions) to form an intelligent ISP. The model parameters of this intelligent ISP are obtained through end-to-end automatic learning method. This approach can reduce the amount of time and labor required in ISP tuning and new algorithm development, thus can meet the rapidly evolving needs of AI applications. In this project, we will design a novel ConvNet to integrate the traditional ISP function modules into an intelligent ISP. The performance of traditional ISP and intelligent ISP will be compared quantitatively through Peak Signal-to-Noise Ratio(PSNR)/Structural SIMilarity(SSIM), noise suppression, etc. in the research report. In addition, a light intensity quantitatively controlled image data acquisition system will be set up to collect training and evaluation data for dim light image/video enhancement. Finally, we will port and optimize the intelligent ISP algorithm on embedded GPU, build the product prototype, and provide the hardware feasibility evaluation report.