A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

TL;DR

The work tackles data movement and energy bottlenecks in CNN-based vision systems by moving the first-layer convolution into a reconfigurable Processing-in-Pixel CMOS image sensor. It introduces a PWM-driven MAC built from 2.5T-per-pixel units with in-array convlink wiring and kernel-splicing to realize $3\times3$ to $9\times9$ kernels, enabling parallel, in-sensor first-layer computation. Simulation results in a $128\times128$ array show high linearity ($R^2>0.98$), substantial readout reduction, and a computing efficiency up to $11.65$ TOPS/W for a $7\times7$ kernel at $60$ FPS, with energy per frame far lower than traditional CIS+DLA systems. The approach improves fill-factor, reduces power, and supports scalable kernel sizes for IoT and surveillance scenarios, where low-latency, low-power vision is critical.

Abstract

The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited end sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based CMOS sensor architecture, which allows convolution operation before the column readout circuit to significantly improve the image reading speed with much lower power consumption. The simulation results show that the proposed architecture could support the computing efficiency up to 11.65 TOPS/W at the 8-bit weight configuration, which is three times as high as the conventional schemes. The transistors required for each pixel are only 2.5T, significantly improving the fill-factor.

Figures (18)

• Figure 1: The first layer of CNN is performed by PIP CMOS image sensor and the rest is done with DLA.
• Figure 2: Different architectures of visual systems. (a)Traditional architecture. (b)PNS architecture. (c)PIS architecture. (d)PIP architecture. Blue boxes represent the pixel, grey boxes mean the sensors, and green boxes show where the computing is conducted.
• Figure 3: The overview of the PIP architecture. (a) The system architecture, (b) The structure diagram of the pixel array, and (c) The pixel circuits. The red lines represent the column readout wires, the green lines represent the convlink wires between adjacent pixels, and the golden line shows the direction of the control signals.
• Figure 4: The computing flow diagram of the proposed architecture.
• Figure 5: Proposed pixel circuits in the (a) reset stage, (b) exposure stage in Traditional mode, (c) readout stage in Traditional mode, (d) exposure stage in Computing mode, (e) readout stage in Computing mode. The red lines represent the column readout wires and the green lines represent the convlink wires between adjacent pixels.