Neural Rendering and Its Hardware Acceleration: A Review

TL;DR

This paper surveys neural rendering as a synthesis of deep learning with physics-based graphics, emphasizing the hardware acceleration necessary to scale such methods in forward and inverse rendering as well as post-processing. It covers theoretical foundations (physics-based rendering, neural scene representations), representative neural rendering applications, and the hardware landscape, including CPUs, GPUs, and DSAs. The work highlights common computational bottlenecks (MLP queries, hash encoding, ray marching, volume rendering) and surveys accelerator designs (NRPU, ICARUS, NGPC, Gen-NeRF) and collaborative software–hardware strategies. By outlining platform capabilities, design challenges, and development trends, the paper argues for integrated neural rendering processors tailored to cloud and edge scenarios to support AR/VR, digital humans, and metaverse applications.

Abstract

Neural rendering is a new image and video generation method based on deep learning. It combines the deep learning model with the physical knowledge of computer graphics, to obtain a controllable and realistic scene model, and realize the control of scene attributes such as lighting, camera parameters, posture and so on. On the one hand, neural rendering can not only make full use of the advantages of deep learning to accelerate the traditional forward rendering process, but also provide new solutions for specific tasks such as inverse rendering and 3D reconstruction. On the other hand, the design of innovative hardware structures that adapt to the neural rendering pipeline breaks through the parallel computing and power consumption bottleneck of existing graphics processors, which is expected to provide important support for future key areas such as virtual and augmented reality, film and television creation and digital entertainment, artificial intelligence and the metaverse. In this paper, we review the technical connotation, main challenges, and research progress of neural rendering. On this basis, we analyze the common requirements of neural rendering pipeline for hardware acceleration and the characteristics of the current hardware acceleration architecture, and then discuss the design challenges of neural rendering processor architecture. Finally, the future development trend of neural rendering processor architecture is prospected.