Photon-efficient camera with in-sensor computing

Image sensors with internal computing capabilities fuse sensing and computing to significantly reduce the power consumption and latency of machine vision tasks. Linear photodetectors such as 2D semiconductors with tunable electrical and optical properties enable in-sensor computing for multiple functions. In-sensor computing at the single-photon level is much more plausible but has not yet been achieved. Here, we demonstrate a photon-efficient camera with in-sensor computing based on a superconducting nanowire array detector with four programmable dimensions including photon count rate, response time, pulse amplitude, and spectral responsivity. At the same time, the sensor features saturated (100%) quantum efficiency in the range of 405–1550 nm. Benefiting from the multidimensional modulation and ultra-high sensitivity, a classification accuracy of 92.22% for three letters is achieved with only 0.12 photons per pixel per pattern. Furthermore, image preprocessing and spectral classification are demonstrated. Photon-efficient in-sensor computing is beneficial for vision tasks in extremely low-light environments such as covert imaging, biological imaging and space exploration. The single-photon image sensor can be scaled up to construct more complex neural networks, enabling more complex real-time vision tasks with high sensitivity.