Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement

Chia Hung Yeh, Hsin Fu Yang, Yu Yang Lin, Wan Jen Huang, Feng Hsu Tsai, Li Wei Kang*


研究成果: 雜誌貢獻期刊論文同行評審


This paper addresses the problem for fine-grained video super-resolution (FGVSR) to suppress temporal flickering caused by separately processed consecutive frames and enhance the quality of restored video frame details when upsizing videos. Some existing video SR methods fail to sufficiently utilize spatial-temporal information from input low-resolution (LR) videos, while others may generate undesirable artifacts or cannot well reconstruct image details. To overcome these problems, we present a novel deep learning framework for FGVSR, which takes a set of consecutive LR video frames and generate the corresponding super-resolved frames. Our deep FGVSR framework focuses on reconstructing missing information from the LR sources based on the proposed multi-frame alignment and refinement strategies. More specifically, we propose an alignment module, where multiple frames are aligned at feature level, to prevent the output videos from flickering. Then, we introduce a feature fusion module, where aligned features generated from our alignment module are fused and refined in a multi-scale manner. Finally, the proposed refinement module is used to reconstruct missing information based on the fused features. In addition, we also embed an image enhancement module on the skip connection from the input layer to the output layer of our network for further enhancing the SR results. Experimental results show that the proposed deep FGVSR, compared with existing deep learning-based VSR methods, achieves state-of-the-art performances on the three well-known benchmarks, including REDS, Vid4, and Vimeo90k. More specifically, compared with the state-of-the-art VSR methods in our experiments, our FGVSR achieves quantitative improvements from 0.70 dB to 9.54 dB in PSNR. On the other hand, our method has also been shown to be efficient to other image restoration tasks, such as image inpainting.

期刊Engineering Applications of Artificial Intelligence
出版狀態已發佈 - 2024 5月

ASJC Scopus subject areas

  • 控制與系統工程
  • 人工智慧
  • 電氣與電子工程


深入研究「Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement」主題。共同形成了獨特的指紋。