Lightfield - ISET/isetcam GitHub Wiki

These are notes about the historical development of the concepts of the lightfield. They will continue to be updated over time.

Remarkably, it was Leonardo in 1509 who first recognized that the rays of light from an object fill space in all directions. He noticed that when he built a pinhole camera by poking a small hole in a wall that faced on a brightly illuminated piazza. In addition to seeing the image of the piazza (inverted) on the wall of the room, he noticed that the position of the pinhole in the wall could be neatly anywhere. He explicitly stated that this implied the image of the piazza must be present in all directions, much as dropping a rock in a pond sent out waves in all directions.

Over the next four hundred years physicists and biologists made many advances in understanding the nature of light (Newton, Planck) and the interaction of light with the eye (Young, Helmholtz, Maxwell). The formal description of radiometry and photometry followed on these developments (Troland, Wright). It was the Russian physicist, Gershwin, and others who created formal mathematical representations of the light field, or what is sometimes summarized as simply the scene spectral radiance. The light field modeled the scene radiance using geometric concepts, eschewing the many details as being irrelevant for modeling critical elements of the scene needed to understand illumination engineering and photography.

Gershun described the scene as being filled with light rays traveling in all directions but never intersecting or interfering with one another. He argued that each ray might be thought of as having brightness that was constant along its path - even if that path originated on the other side of the galaxy. He further introduced the mathematics of calculating how to quantify the photometry (or radiometry) with respect to small surface elements and the direction of the rays. Each ray travels from a location (3 parameters) in a direction (two parameters) and has a wavelength and polarization, for a total of 7 parameters. To know the brightness of each of these rays is to know the light field.

As a practical matter, we hardly ever need to know the entire light field. Rather, like Leonardo we place an imaging device at different locations within the light field and sample the rays incident at the device - either a camera or the eye. The light field incident at the eye or camera was named the plenoptic function by Adelson and abet gen in a recent and widely celebrated paper. It is very convenient to distinguish this concept from the general light field, though the distinction is not always made in the literature. The plenoptic function can be described as the position of the rays incident at the aperture of the sensing device, which comprises 2 parameters. The angle of each day requires an additional two parameters. For many devices we can ignore the polarization and keep the wavelength, for a total of five parameters. Hence, in practice for many cameras and the human eye the plenoptic function is a slightly simplified description of the local light field.

Adelson and Bergen make the important point that many visual functions can be conceived of as calculations that act on the time-varying plenoptic function. This mathematical representation of the physical stimulus provides a unifying framework to guide how we conceive of the neural mechanisms that estimate color, motion, and form. They suggest that this formulation, grounded in the fundamental mathematical representation of the physical stimulus, serves as a better foundation for understanding vision than the prevailing approach at the time which was to conceive of images as features (edges, corners, generalized cylinders).

Better or worse is a judgment we can leave to others. But the value of understanding vision from the mathematical perspective of light fields and the plenoptic function has proven to be important and practical. In this chapter we build on the representation through a series of formulae and algorithms and fundamental experimental observations.