Sensor - ISET/isetcam GitHub Wiki

• Core Pipeline overview
• Sensor tutorials
• Built-in sensor models
• Color filter arrays
• Sensor noise model

The sensor structure models an image sensor array: the photodetectors, color filter array (CFA), pixel electronics, and noise sources that convert optical-image irradiance into an array of digital pixel values. It is the third stage of the ISETCam pipeline, after scene radiance and optical-image irradiance are computed. With the development of CMOS image sensors in the 1990s, sensor structures and electronics have become increasingly sophisticated; modern sensors combine many components with a wide range of geometric and electronic properties, and ISETCam's sensor model accounts for them.

Each pixel contains one or more photodetectors that convert photons into electrons. Pixels spatially sample the irradiance and normally sit behind a microlens and a color filter. ISETCam groups the IR filter, color filters, and photodetector parameters as part of the sensor model, covering both geometric and electrical properties. Sophisticated microlens geometry, however, is part of the optics calculations in ISET3D — see Light field.

Scene (radiance) --oiCompute--> Optical Image (irradiance) --sensorCompute--> Sensor --ipCompute--> Image Processor --> Display

sensorCompute takes a sensor definition and an optical image and produces the pixel voltages:

scene  = sceneCreate;                scene  = sceneSet(scene,'fov',10);
oi     = oiCreate('wvf');            oi     = oiCompute(oi,scene,'crop',true);
sensor = sensorCreate('imx363');     sensor = sensorSet(sensor,'fov',10,oi);
sensor = sensorCompute(sensor,oi);
sensorWindow(sensor);

The left side of the sensor window shows pixel properties — size, fill factor, electrical noise, and so on. The right side summarizes properties of the sensor as a whole: pixel count, color filter pattern, dark signal nonuniformity (DSNU), photoresponse nonuniformity (PRNU), and more.

sensorCompute reads the spectral irradiance stored in the optical image and calculates the expected number of electrons at each pixel, accounting for:

• the spatial sampling of the irradiance data relative to the size and position of the pixels in the sensor array;
• the color filter covering each pixel — see Color filter arrays;
• the light sensitivity of each pixel through its filter, and its well capacity;
• shot noise and other electrical noise sources — see Sensor noise model;
• spatial nonuniformity of the pixels across the array (DSNU, PRNU).

Like other ISETCam structures, the sensor has the standard set of methods: sensorCreate, sensorSet/sensorGet, sensorCompute, sensorWindow, and sensorPlot. Pixel-level parameters live in a pixel sub-structure (sensorGet(sensor,'pixel')) and are read and written with the same sensorSet/sensorGet calls, using a 'pixel ...' parameter name.

sensorGet(sensor,'pixel size','um')      % ans = 1.4000  1.4000
e = sensorGet(sensor,'electrons');       % Per-pixel electron count
size(e)                                  % ans = 362  484

sensor = sensorSet(sensor,'exp time',0.050);   % Exposure time, seconds
sensorGet(sensor,'exp time','ms')              % ans = 50

ISETCam sets a deliberately small number of parameters directly; the much larger set of values you can get includes quantities derived from those you set, so that dependent properties never fall out of sync. See Programming Conventions for the general *Get/*Set philosophy shared by all ISETCam structures. Typing sensor<TAB> or pixel<TAB> at the MATLAB prompt lists the full set of sensor functions.

Start a simulation with a generic Bayer array, a calibrated vendor part, a multi-channel sensor, or another predefined configuration. See Built-in sensor models for the available sensorCreate options and short descriptions of each model category.

• Sensors and camera systems — sensor tutorials covering color filters, exposure, noise, and spatial resolution.
• Examples — s_sensorSNR works through the sensor SNR formula and how pixel and sensor parameters determine it.

The online Foundations of Image Systems Engineering (FISE) develops sensor and pixel physics, and signal-to-noise ratio, as conceptual background.

For many years ISETCam also modeled human visual encoding directly. Around 2015, Dave Brainard, Joyce Farrell, and Brian Wandell decided the many specializations required for human encoding needed their own implementation, and ISETCam and ISETBio were maintained as parallel repositories until about 2023. Since the 2023-2024 refactor, ISETCam is the base and ISETBio is a specialization built on it: human visual calculations use ISETBio with both ISETCam and ISETBio on the MATLAB path, rather than the sensor structure's legacy 'human' cone-mosaic option.