• Scene Radiance
• Optics and Optical Images

---

The light field

These are notes about the historical development of the concept of the light field. They will continue to be updated over time.

Remarkably, it was Leonardo in 1509 who first recognized that the rays of light from an object fill space in all directions. He noticed this while building a pinhole camera by poking a small hole in a wall that faced a brightly illuminated piazza. In addition to seeing the (inverted) image of the piazza on the wall of the room, he noticed that the pinhole could be placed almost anywhere in the wall and still produce the image. He explicitly stated that this implied the image of the piazza must be present in all directions, much as dropping a rock in a pond sends out waves in all directions.

Over the next four hundred years, physicists and biologists made many advances in understanding the nature of light (Newton, Planck) and the interaction of light with the eye (Young, Helmholtz, Maxwell). The formal description of radiometry and photometry followed from these developments (Troland, Wright). It was the Russian physicist Gershun, and others, who created formal mathematical representations of the light field — what is sometimes summarized simply as the scene spectral radiance. The light field modeled scene radiance using geometric concepts, setting aside the many details irrelevant to the critical elements needed to understand illumination engineering and photography.

Gershun described the scene as filled with light rays traveling in all directions, never intersecting or interfering with one another. He argued that each ray could be thought of as having a brightness that is constant along its path, even if that path originated on the other side of the galaxy. He introduced the mathematics for quantifying the photometry (or radiometry) with respect to small surface elements and ray direction. Each ray travels from a location (3 parameters) in a direction (2 parameters) and has a wavelength and polarization, for a total of 7 parameters; knowing the brightness of every such ray is knowing the light field.

As a practical matter, we hardly ever need the entire light field. Rather, like Leonardo, we place an imaging device — a camera or an eye — at particular locations within the light field and sample the rays incident at that device. The light field incident at the eye or camera was named the plenoptic function by Adelson and Bergen in a widely celebrated paper. It is convenient to distinguish this from the general light field, though the distinction is not always made in the literature. The plenoptic function describes the position of the rays incident at the aperture of the sensing device (2 parameters) and the angle of each ray (2 more parameters); for many devices we can ignore polarization and keep wavelength, for a total of five parameters.

Adelson and Bergen made the important point that many visual functions can be conceived of as calculations acting on the time-varying plenoptic function. This mathematical representation of the physical stimulus provides a unifying framework for thinking about the neural mechanisms that estimate color, motion, and form — a foundation they argued was better suited to understanding vision than the then-prevailing approach of conceiving of images as features (edges, corners, generalized cylinders).

The value of understanding vision from the mathematical perspective of the light field and the plenoptic function has proven to be both important and practical.

In ISETCam

The ordinary ISETCam scene and optical image structures (see Scene Radiance and Optics and Optical Images) simplify the light field to a single spectral radiance or irradiance value per spatial position — the planar-scene approximation used throughout the core pipeline. Modeling the light field more completely — light field cameras with microlens arrays above the sensor, multiple viewpoints, and related plenoptic-sampling designs — is handled by ISET3D, which simulates such cameras using physically based ray tracing. The examples/optics/s_opticsMicrolens.m example (see Examples) illustrates the microlens object ISETCam uses at the sensor surface in these designs.