The PYNQ Machine Learning and Computer Vision Acceleration Platform

Announcements
1

Hi PYNQ Community,

On behalf of Alpha Data, I am excited to share the Machine Learning and Computer Vision Acceleration Platform (MLVAP) example design, originally created for the ADM-XRC-9Z1 to explore the power of PYNQ. The example design is available in the pynq-mlvap repository with guidance for porting to other boards and building it inside a custom Docker container.

The MLVAP example design combines the PYNQ Composable methodology with DPU on PYNQ. The composable pipeline accelerates image pre-processing for the DPU running machine learning models tackling computer vision tasks. The pipeline supports colour channel conversion, 2 types of resize operations, shifting, scaling and 8-bit fixed point quantisation.

MLVAP architecture
MLVAP architecture769×799 81.5 KB

The MLVAP design is tied together by the MlvapOverlay providing a high-level interface to configure and use the design in a few simple steps:

from pynq_mlvap import MlvapOverlay

# Load the MLVAP overlay
ol = MlvapOverlay('mlvap.bit')

# Load the ML model compiled for the DPU
ol.load_model('path/to/model.xmodel')

# Specify the pre-processing parameters and steps
in_h, in_w = ...
out_h, out_w = ...
means = (..., ..., ...)
std_devs = (..., ..., ...)
steps = [...] # e.g. ['bgr2rgb', 'resize', 'preprocess']

# Configure the pipeline
ol.config(steps, in_h, in_w, out_h, out_w, means, std_devs)

# Start the pre-processing IPs
ol.start()

# Run the pre-processing pipeline and ML inference for an input image
output_data = ol.run(input_image) # input_image.shape == (in_h, in_w, 3)

# Stop the pre-processing IPs
ol.stop()

# Post-process output data from the ML model
...

Complete example notebooks are provided for 3 machine learning models:

Model Task Pre-processing
YOLOv3 Object Detection bgr2rgb, letterbox, scaling, quantisation
RefineDet Object Detection resize, shifting, quantisation
Semantic-FPN Semantic Segmentation bgr2rgb, resize, shifting, scaling, quantisation

I hope you enjoy exploring this example design and I certainly look forward to seeing what other designs this may inspire. Maybe a design dedicated to RF signal pre-processing and ML deployment on RFSoC boards?

1 Like