To develop a high‑quality image signal processor (ISP), imaging professionals must have a deep understanding of the entire imaging pipeline, as every algorithm in the system influences and interacts with the others. Only through seamless coordination among these algorithms can an exceptional ISP processing chain be achieved.

Digital cameras feature a mature, end-to-end ISP pipeline. By capturing an image with the camera and examining it, we can gain insight into the roles of each module within the ISP algorithm. Furthermore, by working with the RAW images produced by the camera, we can design our own ISP processing pipeline. The design of digital cameras draws direct inspiration from the imaging principles of the human eye; therefore, we begin by exploring human vision.

Human Vision and Digital Cameras

People perceive objects through their eyes via the following process: First, light reflected from an object passes through the cornea—where it is refracted to focus precisely on the retina—and the pupil, which regulates the amount of light entering the eye. The light is then focused onto the retina by the lens. In the retina, rod cells (responsible for scotopic vision) and cone cells (responsible for photopic vision and color perception) convert light signals into electrical signals, which are transmitted to the brain via the optic nerve, thereby forming visual perception.

Here, using a cross-sectional diagram of a DSLR camera, we’ll introduce the structure of a digital camera:

Lens group: By adjusting the camera’s focal length, the field of view can be changed.

Aperture: By adjusting the size of the lens opening, you can control depth of field, image quality, and, in coordination with the shutter, the amount of light entering the camera.

Reflector: It redirects the light path and reflects it into the viewfinder, allowing the operator to compose the shot accurately.

Pentaprism: Its function is to correct the left–right inversion of the image on the focusing screen, ensuring that the view through the viewfinder exactly matches the orientation of the scene as seen directly.

Viewfinder: The part of a digital camcorder that allows you to monitor the image through an eyepiece.

Shutter: The mechanism a camera uses to control the effective exposure time of the photosensitive film, which comes in two types: mechanical shutters and electronic shutters.

Image sensor: A functional device that utilizes the photoelectric conversion capability of photoelectric components to convert light into usable electrical signals.

What is exposure?

Light enters the eyeball through the pupil and forms an image on the retina—this is essentially an exposure process.

In a digital camera, exposure can be summarized as follows: when you press the shutter button, the shutter opens and remains open for a set period of time. During this interval, light enters the camera through the lens and strikes the image sensor. The photosensitive pixels on the sensor record the incoming light, thereby forming an image.

How should exposure be evaluated?

From the perspective of the overall image, if the subject in a photograph is too dark, it’s called underexposure. If it’s too bright, it’s called overexposure.

However, these descriptions are all based on subjective evaluations. Because individual perceptions vary, exposure analysis for certain borderline images may not be entirely consistent. This is where histograms come into play: they allow us to assess the exposure of an image.

A histogram is an essential tool that uses graphical data to help photographers assess a photo’s exposure and color distribution. The x-axis represents brightness or color values, spanning all possible lightness levels from left to right. The y-axis shows the number of pixels, indicating how frequently a particular brightness or color value appears in the image. Common types of histograms include RGB color histograms and grayscale histograms.

A grayscale histogram arranges the image’s intensity values along a numerical axis from black to white, then counts how many pixels fall at each brightness level, representing the results with bars of varying heights. This makes it easy to quickly determine whether a photo is overexposed, underexposed, or properly exposed.

Grayscale histogram

RGB color histogram

How do you adjust exposure?

 Cameras control exposure through three key settings—aperture, shutter speed, and ISO—making it a biomimetic design. Here, we’ll explain this from the perspective of human visual perception: imagine suddenly turning on a light in a dimly lit room. Our eyes undergo a process in which the pupils rapidly constrict to reduce the amount of incoming light, and the cone cells are activated, shifting vision from scotopic (low-light) to photopic (bright-light) conditions.

During photography, precise exposure control is achieved by adjusting the three key settings; the shutter, also known as the exposure time, plays a crucial role. Shutter control does indeed resemble the human eye’s dynamic regulation of light through the pupil, but whereas the eye’s adjustment is a continuous physiological process, the shutter operates in discrete steps, using mechanical or electronic switches to precisely regulate the duration during which light reaches the sensor—especially when capturing moving subjects.

When shooting moving subjects, setting the shutter speed too slow can result in a blurry image.

Aperture control is analogous to how the human eye adjusts the amount of light entering by constricting or dilating the pupil. In a camera, the aperture size determines the quantity of light that reaches the sensor per unit of time. Photos taken with a large aperture allow more light in per unit time, but they produce a shallow depth of field, rendering only objects within a narrow range around the focal plane sharply focused, while everything else appears blurred. By contrast, photos taken with a small aperture offer a greater depth of field, keeping objects across a wider range on either side of the focal plane in sharp focus, though at the cost of reduced light intake per unit time.

 ISO, also known as light sensitivity, is analogous to the switch between rod and cone cells in the human eye when transitioning between bright and dim lighting conditions. In digital cameras, ISO is adjusted by varying the amplifier gain: the higher the gain, the greater the sensitivity. As ISO increases, the sensor becomes more sensitive to light. We’ve previously examined the sources of sensor noise and learned that sensors inherently produce some photon‑shot noise. Raising ISO to boost overall image brightness amplifies this noise, thereby degrading image quality.

Why is automatic exposure necessary?

The very purpose of automatic exposure is to simplify the shooting process for everyday users. On the other hand, in video applications—such as surveillance, autonomous‑driving perception, and aerial photography—users often need to capture a sequence of continuous frames over an extended period. During this time, factors like scene distance, lighting conditions, and color temperature can all change. By the time manual adjustments to camera settings can be made, the subject of interest may already have disappeared.

Automatic exposure can be divided into three steps: luminance acquisition, luminance analysis, and exposure parameter adjustment.

1, Brightness Acquisition

Specifically, brightness information is extracted. For Bayer‑type image sensors, obtaining the luminance channel requires a color‑space conversion that transforms RGB signals into the YUV color space, thereby yielding the luminance value for each pixel.

2, Brightness Analysis

Exposure metering is at the heart of the ISP’s auto-exposure algorithm. It requires analyzing the overall luminance of the image to determine whether it is overexposed or underexposed, enabling the system to apply the appropriate correction strategy. This is also where algorithmic differences among manufacturers are most pronounced. Furthermore, the depth of analysis varies depending on the image sensor’s bit depth: for example, an 8-bit sensor employs a 255‑level histogram, while a 10-bit sensor uses a 1023‑level histogram.

There are three metering methods: evaluative metering, center-weighted average metering, and spot metering.

   Evaluative metering modeIt measures light across the entire frame, combining all reflected light within the scene to produce an overall exposure assessment. The advantage of this method is that it easily yields a well-balanced image, preventing localized highlights from overexposing and resulting in a more even histogram across the whole frame.

    Center-weighted average meteringThis refers to a metering mode that prioritizes the center while treating other areas as secondary. This ensures that, when shooting portraits, the subject is used as the metering reference, resulting in accurate exposure for the subject under any lighting conditions.

    Spot meteringThis refers to metering at a single point, typically located at the same position as the autofocus point—a very small area. The advantage of this method is that it allows the photographer to achieve accurate exposure for a specific point they deem correct or for the subject itself, without being influenced by surrounding light sources.

 In metering, an image is typically divided into multiple zones. This segmentation allows automatic exposure to be tailored to different types of photographs—such as landscapes or portraits—since each genre requires a distinct exposure balance. During the metering process, depending on each manufacturer’s AE algorithm, the system calculates metrics such as the average luminance, variance, minimum, and maximum values for each zone, thereby assessing the current exposure conditions.

3, Exposure Parameter Adjustment

Once the current image’s exposure has been evaluated, you can adjust the exposure by modifying the three exposure settings: aperture, shutter speed, and ISO. However, in practical applications, different scenes call for different adjustment strategies. Digital cameras offer three exposure‑control modes—semi‑automatic, fully automatic, and manual—with the semi‑automatic mode being one of them.

Aperture Priority Mode: When motion blur is a significant concern, Aperture Priority is typically chosen—first adjusting the aperture, then the shutter speed, and finally the ISO.

Shutter Priority Mode: When a deep depth of field is required, Shutter Priority mode is typically chosen—first adjust the exposure time, then the aperture, and finally the ISO.

Auto mode: Primarily used for quick shooting by beginners or capturing spontaneous moments.

Manual mode: Requires the user to manually adjust the three exposure settings.

The goal of automatic exposure in digital camcorders is to ensure that the grayscale of the captured image approximates middle gray, achieving an ideal visual brightness. Although there are various adjustment methods for configuring camera parameters, automatic exposure typically does not prioritize adjusting the ISO sensitivity, as increasing it would introduce noise.

Different application scenarios require configuring different AecRoutes: For daytime scenes, it is generally necessary to limit the exposure time and prioritize adjusting the gain to prevent motion blur caused by excessively long exposure; for nighttime scenes, the exposure time should typically be adjusted first, followed by the exposure gain, in order to improve the image’s signal-to-noise ratio and enhance nighttime brightness.

Expand

AE module strategy of a major company:

The AE module consists of two main components: the metering and statistics module and the AE control algorithm module.

The metering statistics module provides brightness statistics to the AE control algorithm module. These statistics include histogram data and block‑wise average brightness values. The statistical information primarily comprises the following components:

1, 256-bin weighted histogram statistics based on the raw image.

2, Block-wise R/G/B/Y mean statistics based on the raw image.

3, 32-segment weighted histogram statistics based on the gamma‑preprocessed RGB image.

4, Block-wise R/G/B/Y mean statistics based on the gamma‑preprocessed RGB image.

The AE control algorithm module compares the input luminance statistics with the target brightness, calculates the new exposure settings, and then automatically adjusts the sensor’s exposure time, exposure gain, and lens aperture to produce an image with appropriate brightness.