These steps are to use new vitis vision-based functions or custom HLS and insert them into the Composable Pipeline. I hope this helps to unravel how to extend this great tool. Please let me know if you have feedback.
High level overview:
- Create new IP
- Add the new IP to the Composable Pipeline Vivado design
- Build the new bitstream
- Edit the Composable Pipeline Python code to support the new function
For this example, I will create a gammacorrection HLS block and insert it into the pipeline.
Creating a new IP block using OpenCV
The existing Composable Pipeline functions are in PYNQ_Composable_Pipeline/src. NOTE: the name of the directory and the cpp file are expected by the Makefile to be the name of the xfOpenCV kernel function. In this case, it would be gammacorrection.
cd PYNQ_Composable_Pipeline/src
mkdir gammacorrection
cp LUT/LUT.cpp gammacorrection/gammacorrection.cpp
Make the necessary edits:
- include the correct vitis vision hpp file
- change the function name to be the name of the xfOpenCV kernel + _accel
- Make sure that the #defines are correct ( remove unused ones and create any new ones )
- Edit the offsets as necessary. The general pipeline framework expects rows at 0x10 and cols at 0x18. Look in the HWH file to get the offsets if needed
In this example, the gammacorrection.cpp file is:
// Copyright (C) 2021 Xilinx, Inc
//
// SPDX-License-Identifier: BSD-3-Clause
#include "hls_stream.h"
#include "common/xf_common.hpp"
#include "common/xf_infra.hpp"
#include "imgproc/xf_gammacorrection.hpp"
#define DATA_WIDTH 24
#define NPIX XF_NPPC1
/* set the height and width */
#define WIDTH 1920
#define HEIGHT 1080
#define TYPE XF_8UC3
typedef xf::cv::ap_axiu<DATA_WIDTH,1,1,1> interface_t;
typedef hls::stream<interface_t> stream_t;
// https://xilinx.github.io/Vitis_Libraries/vision/2020.2/api-reference.html#custom-convolution
void gammacorrection_accel(stream_t& stream_in,
stream_t& stream_out,
unsigned int rows,
unsigned int cols,
float gaPreparing to unpack .../gir1.2-gtksource-3.0_3.24.11-2_amd64.deb ...
egister both port=stream_in
#pragma HLS INTERFACE axis register both port=stream_out
#pragma HLS INTERFACE s_axilite port=rows offset=0x10
#pragma HLS INTERFACE s_axilite port=cols offset=0x18
#pragma HLS INTERFACE s_axilite port=gammaval offset=0x20
#pragma HLS INTERFACE s_axilite port=return
xf::cv::Mat<TYPE, HEIGHT, WIDTH, NPIX> img_in(rows, cols);
xf::cv::Mat<TYPE, HEIGHT, WIDTH, NPIX> img_out(rows, cols);
#pragma HLS DATAFLOW
// Convert stream in to xf::cv::Mat
xf::cv::AXIvideo2xfMat<DATA_WIDTH, TYPE, HEIGHT, WIDTH, NPIX>(stream_in, img_in);
// Run xfOpenCV kernel:
xf::cv::gammacorrection<TYPE,TYPE,HEIGHT,WIDTH,NPIX>(img_in, img_out, gammaval);
// Convert xf::cv::Mat to stream
xf::cv::xfMat2AXIvideo<DATA_WIDTH, TYPE, HEIGHT, WIDTH, NPIX>(img_out, stream_out);
}
Build the vitis IP
The Makefile builds the HLS in boards/ip/vitis_vision. If the compile is not successful, delete the boards/ip/vitis_vision/.vhlsprj directory and rebuild.
cd ../boards/ZCU104.build
make vision_ip
Now the IP is available to vivado as the composable pipeline project already has this directory in the IP sources.
Modify the Vivado project
Open the project in boards/ZCU104.build/cv_dfx_4_pr/cv_dfx_4_pr.xpr
Navigate to Open Block design Double-click on Composable Choose one of the IP blocks to remove - like rgb2hsv - and remove it.
Add in the New Ip by clicking the â+â icon and searching for gammacorrection Move it into the location of the old IP connect the pins (there is only one way to do this ) Click on the gammacorrection IP and rename it by removing the _0 at the end Navigate to Window->Address Editor and go to the bottom for âUnassigned entryâ and click assign
Now, build it
- Run Synthesis
- Run Implementation
- Run Generate Bitstream
Collect the overlay files and package everything
After editing the design, the overlay files will be out of date. Download the overlay.tcl file (culled from the main tcl file)
# collect the overlay files
vivado -mode batch -source overlay.tcl -notrace
# Make the dictionary
make dict
# Make the ZIP file for transferring to the board
make zip
# copy the file to the board:
scp composable-video-pipeline-zcu104-v1_0_2.zip xilinx@<board>:/tmp
Install the new overlays
Log into the board as the xilinx user
sudo -i
cd ~xilinx/jupyter_notebooks
# Install the overlay directory and file
unzip /tmp/composable-video-pipeline-zcu104-v1_0_2.zip
Note: In the Jupyter Notebooks, reference the overlay path by full path ol = Overlay(â/home/xilinx/jupyter_notebooks/overlay/cv_dfx_4_pr.bitâ)
Edit the Composable Pipeline python
On the board, the pynq_composable files are located at /usr/local/share/pynq-venv/lib/python3.8/site-packages/pynq_composable/
Edit the libs.py to get base functionality. apps.py needs to be edited for the interactive widgets to work (future work)
There are 2 main steps. The first is to install the new accelerator in the VitisVisionIP class to get some common functionaly. The second is to add the new class that calls the functionality.
Step 1:
Edit the bindto array in the VitisVisionIP class to replace the removed accelerator with the new one. In this case, it looks like:
bindto = [
'xilinx.com:hls:dma2video_accel:1.0',
'xilinx.com:hls:video2dma_accel:1.0',
'xilinx.com:hls:rgb2gray_accel:1.0',
'xilinx.com:hls:gaussian_filter_accel:1.0',
'xilinx.com:hls:gray2rgb_accel:1.0',
'xilinx.com:hls:pyrUp_accel:1.0',
'xilinx.com:hls:subtract_accel:1.0',
'xilinx.com:hls:gammacorrection_accel:1.0',
'xilinx.com:hls:rgb2xyz_accel:1.0',
"xilinx.com:hls:absdiff_accel:1.0",
"xilinx.com:hls:add_accel:1.0",
"xilinx.com:hls:bitwise_and_accel:1.0",
"xilinx.com:hls:bitwise_not_accel:1.0",
"xilinx.com:hls:bitwise_or_accel:1.0",
"xilinx.com:hls:bitwise_xor_accel:1.0",
]
Step 2: Add the new functionality. This is where the mapping of python variables and registry maps occurs. The rows and cols happens because it inherits from the VitisVisionIP. Other axilite variables get mapped here. In this case, gammaval needs to be added. Changing the variable in Python doesnât cause it to be written to the register map by default. So a getter/setter is created to actually write the value. In this case, it writes the float to 0x20.
Note: Examining cv_dfx_4_pr.hwh will show the register maps. In this case, under gammacorrection_accel: rows: ADDRESS_OFFSET=16 â 0x10 cols: ADDRESS_OFFSET=24 â 0x18 gammaval: ADDRESS_OFFSET=32 â -x20
class gammacorrection(VitisVisionIP):
"""gammacorrection"""
bindto = ['xilinx.com:hls:gammacorrection_accel:1.0']
def __init__(self, description):
print("gammacorrection: init")
super().__init__(description=description)
self._gammaval = 1.0
def start(self):
super().start()
self.gammaval=1.0
print("gammacorrection: start")
@property
def gammaval(self) -> float:
return self._gammaval
@gammaval.setter
def gammaval(self, gammaval: float):
if not isinstance(gammaval, (float, int)):
raise ValueError("gammaval must a number")
self._gammaval = float(gammaval)
self.write(0x20, _float2int(self.gammaval))
#print("setting gammaval to " + str(self.gammaval))
Test it in the Jupyter Notebook
Open the pynq_composable/custom_pipeline/02_first_custom_pipeline.ipynb notebook. File â Make a Copy and save it as GammaCorrection
Changes:
#Downlaod Composable Overlay:
ol = Overlay("/home/xilinx/jupyter_notebooks/overlay/cv_dfx_4_pr.bit")
#Let us Compose:
gamma = cpipe.gammacorrection_accel
video_pipeline = [cpipe.hdmi_source_in, gamma, cpipe.hdmi_source_out]
cpipe.compose(video_pipeline)
#Play with IP
import time
for i in range(0,1000):
gamma.gammaval = i*0.0025
time.sleep(0.01)
gamma.gammaval=1
Then run the Jupyter Notebook.
TODO:
- insert new functions into the DFX region
- Path for the overlays isnât default
- Make the apps work
