Camera sensor Architecture in Nvidia orin/thor

The NvMedia Sensor Input Processing Library (SIPL) framework provides a simplified API to capture the output of image sensors connected to NVIDIA® DRIVE AGX platforms. SIPL is implemented in C++ on top of the NvMedia API.

The purpose of SIPL is to abstract the following operations from the application layer:

•Programming the image sensors, EEPROMs, serializers, and deserializers.

•Programming platforms to capture and process the images using hardware image processing pipelines (ISPs).

Architecture

The following diagram illustrates the architecture of the SIPL framework.

A screenshot of a cell phone Description automatically generated

The SIPL Query component is used to query data about which external devices are supported, how they are connected to the platform, and how they should be configured. It returns this information in the form of a PlatformCfg struct.

The SIPL DeviceBlock is a component of the SIPL framework which initializes and controls the external devices attached to the platform. It uses I2C to program the settings specified in the PlatformCfg struct. It initializes the GMSL deserializers, brings up the serializer/deserializer (serdes) link between the camera modules and the deserializers, and begins streaming from the sensors on the camera modules. The application does not need to use this component directly; it is instead used by SIPL Core.

The SIPL Core component orchestrates the entire capture process from initialization, through running, to deinitialization. It uses the SIPL DeviceBlock component to initialize the external devices, and it uses NvMedia ICP and NvMedia ISP to initialize the SoC to process captured images. It also allows the user to register callbacks to handle images both before and after they have gone through the ISP pipeline.