It uses industry-standard MIPI connectivity: D-PHY and MIPI CSI-2 for the FPGA interface and Altera’s FPGA AI tool for smart camera development to develop glass-to-glass edge vision for smart cameras, with real-time 4K AI interference.
The MIPI interface supports up to 2.5Gbps per lane and up to 8x lanes per MIPI interface for seamless data reception from multiple 4K image sensors to the FPGA fabric for further processing.
Each MIPI CSI-2 IP converts pixel data to AXI4-Streaming outputs, for connectivity to other IP cores within Altera’s Video and Vision Processing (VVP) suite of tools.
Altera‘s FPGA AI suite is configurable and can be optimised to run the Ultralytics YOLOv8 nano detection and pose inference models (pictured) for smart camera use.
The hardware includes a multi-sensor input video switch feeding into an image signal processing (ISP) subsystem. The ISP is a video processing pipeline incorporating various VVP IP cores allowing the raw sensor image data to be processed into RGB video data. The backend of the ISP pipeline feeds the AI pipeline which consists of VVP IP cores to buffer, format, and scale the video suitable for AI processing. The backend of the AI pipeline drives the resulting 4Kp30 streaming video output data (complete with AI inference overlay) through an Altera DisplayPort IP.
The software stack is Linux based and runs on the HPS. The software runs compiled YOLOv8 nano models from the microSD card. Altera points out that the end-use must license and download the models directly from Ultralytics.
The software stack consists of an application software binary running on the Linux operating system with various layers of drivers. The backend part of the application software interrogates the hardware, dynamically discovers the IP components and configures them. The AI inference part of the application software schedules inference requests to Altera’s FPGA AI Suite IP, and processes the inference results. The results are rendered as graphics in a frame buffer, which the hardware overlays on the video stream. There are multiple feedback loops, including automatic white balance, auto exposure and adaptive noise reduction algorithms, all performed in real-time. The frontend of the software creates a web-based GUI and runs it over a web server.
The design is compatible with Altera Quartus Prime Pro Edition version 25.1 Linux.
