Changing the IP in the Composable Pipeline

These steps are to use new vitis vision-based functions or custom HLS and insert them into the Composable Pipeline. I hope this helps to unravel how to extend this great tool. Please let me know if you have feedback.

High level overview:

  • Create new IP
  • Add the new IP to the Composable Pipeline Vivado design
  • Build the new bitstream
  • Edit the Composable Pipeline Python code to support the new function

For this example, I will create a gammacorrection HLS block and insert it into the pipeline.

Creating a new IP block using OpenCV

The existing Composable Pipeline functions are in PYNQ_Composable_Pipeline/src. NOTE: the name of the directory and the cpp file are expected by the Makefile to be the name of the xfOpenCV kernel function. In this case, it would be gammacorrection.

cd PYNQ_Composable_Pipeline/src
mkdir gammacorrection
cp LUT/LUT.cpp gammacorrection/gammacorrection.cpp

Make the necessary edits:

  • include the correct vitis vision hpp file
  • change the function name to be the name of the xfOpenCV kernel + _accel
  • Make sure that the #defines are correct ( remove unused ones and create any new ones )
  • Edit the offsets as necessary. The general pipeline framework expects rows at 0x10 and cols at 0x18. Look in the HWH file to get the offsets if needed

In this example, the gammacorrection.cpp file is:

// Copyright (C) 2021 Xilinx, Inc
//
// SPDX-License-Identifier: BSD-3-Clause

#include "hls_stream.h"
#include "common/xf_common.hpp"
#include "common/xf_infra.hpp"
#include "imgproc/xf_gammacorrection.hpp"

#define DATA_WIDTH 24
#define NPIX XF_NPPC1

/*  set the height and width  */
#define WIDTH 1920
#define HEIGHT 1080
#define TYPE XF_8UC3


typedef xf::cv::ap_axiu<DATA_WIDTH,1,1,1> interface_t;
typedef hls::stream<interface_t> stream_t;

// https://xilinx.github.io/Vitis_Libraries/vision/2020.2/api-reference.html#custom-convolution
void gammacorrection_accel(stream_t& stream_in,
                    stream_t& stream_out,
                    unsigned int rows,
                    unsigned int cols,
                    float gaPreparing to unpack .../gir1.2-gtksource-3.0_3.24.11-2_amd64.deb ...
egister both port=stream_in
#pragma HLS INTERFACE axis register both port=stream_out
#pragma HLS INTERFACE s_axilite port=rows offset=0x10
#pragma HLS INTERFACE s_axilite port=cols offset=0x18
#pragma HLS INTERFACE s_axilite port=gammaval offset=0x20
#pragma HLS INTERFACE s_axilite port=return

    xf::cv::Mat<TYPE, HEIGHT, WIDTH, NPIX> img_in(rows, cols);
    xf::cv::Mat<TYPE, HEIGHT, WIDTH, NPIX> img_out(rows, cols);

#pragma HLS DATAFLOW

    // Convert stream in to xf::cv::Mat
    xf::cv::AXIvideo2xfMat<DATA_WIDTH, TYPE, HEIGHT, WIDTH, NPIX>(stream_in, img_in);

    // Run xfOpenCV kernel:
    xf::cv::gammacorrection<TYPE,TYPE,HEIGHT,WIDTH,NPIX>(img_in, img_out, gammaval);

    // Convert xf::cv::Mat to stream
    xf::cv::xfMat2AXIvideo<DATA_WIDTH, TYPE, HEIGHT, WIDTH, NPIX>(img_out, stream_out);
}

Build the vitis IP

The Makefile builds the HLS in boards/ip/vitis_vision. If the compile is not successful, delete the boards/ip/vitis_vision/.vhlsprj directory and rebuild.

cd ../boards/ZCU104.build
make vision_ip

Now the IP is available to vivado as the composable pipeline project already has this directory in the IP sources.

Modify the Vivado project

Open the project in boards/ZCU104.build/cv_dfx_4_pr/cv_dfx_4_pr.xpr

Navigate to Open Block design Double-click on Composable Choose one of the IP blocks to remove - like rgb2hsv - and remove it.
Add in the New Ip by clicking the ‘+’ icon and searching for gammacorrection Move it into the location of the old IP connect the pins (there is only one way to do this ) Click on the gammacorrection IP and rename it by removing the _0 at the end Navigate to Window->Address Editor and go to the bottom for “Unassigned entry” and click assign

Now, build it

  • Run Synthesis
  • Run Implementation
  • Run Generate Bitstream

Collect the overlay files and package everything

After editing the design, the overlay files will be out of date. Download the overlay.tcl file (culled from the main tcl file)

# collect the overlay files 
vivado -mode batch -source overlay.tcl -notrace

# Make the dictionary
make dict

# Make the ZIP file for transferring to the board 
make zip 

# copy the file to the board:
scp composable-video-pipeline-zcu104-v1_0_2.zip xilinx@<board>:/tmp

Install the new overlays

Log into the board as the xilinx user

sudo -i 
cd ~xilinx/jupyter_notebooks
# Install the overlay directory and file 
unzip /tmp/composable-video-pipeline-zcu104-v1_0_2.zip 

Note: In the Jupyter Notebooks, reference the overlay path by full path ol = Overlay("/home/xilinx/jupyter_notebooks/overlay/cv_dfx_4_pr.bit")

Edit the Composable Pipeline python

On the board, the pynq_composable files are located at /usr/local/share/pynq-venv/lib/python3.8/site-packages/pynq_composable/

Edit the libs.py to get base functionality. apps.py needs to be edited for the interactive widgets to work (future work)

There are 2 main steps. The first is to install the new accelerator in the VitisVisionIP class to get some common functionaly. The second is to add the new class that calls the functionality.

Step 1:
Edit the bindto array in the VitisVisionIP class to replace the removed accelerator with the new one. In this case, it looks like:

bindto = [
        'xilinx.com:hls:dma2video_accel:1.0',
        'xilinx.com:hls:video2dma_accel:1.0',
        'xilinx.com:hls:rgb2gray_accel:1.0',
        'xilinx.com:hls:gaussian_filter_accel:1.0',
        'xilinx.com:hls:gray2rgb_accel:1.0',
        'xilinx.com:hls:pyrUp_accel:1.0',
        'xilinx.com:hls:subtract_accel:1.0',
        'xilinx.com:hls:gammacorrection_accel:1.0',
        'xilinx.com:hls:rgb2xyz_accel:1.0',
        "xilinx.com:hls:absdiff_accel:1.0",
        "xilinx.com:hls:add_accel:1.0",
        "xilinx.com:hls:bitwise_and_accel:1.0",
        "xilinx.com:hls:bitwise_not_accel:1.0",
        "xilinx.com:hls:bitwise_or_accel:1.0",
        "xilinx.com:hls:bitwise_xor_accel:1.0",
    ]

Step 2: Add the new functionality. This is where the mapping of python variables and registry maps occurs. The rows and cols happens because it inherits from the VitisVisionIP. Other axilite variables get mapped here. In this case, gammaval needs to be added. Changing the variable in Python doesn’t cause it to be written to the register map by default. So a getter/setter is created to actually write the value. In this case, it writes the float to 0x20.

Note: Examining cv_dfx_4_pr.hwh will show the register maps. In this case, under gammacorrection_accel: rows: ADDRESS_OFFSET=16 → 0x10 cols: ADDRESS_OFFSET=24 → 0x18 gammaval: ADDRESS_OFFSET=32 → -x20

class gammacorrection(VitisVisionIP):
    """gammacorrection"""

    bindto = ['xilinx.com:hls:gammacorrection_accel:1.0']

    def __init__(self, description):
        print("gammacorrection: init")
        super().__init__(description=description)
        self._gammaval = 1.0

    def start(self):
        super().start()
        self.gammaval=1.0
        print("gammacorrection: start")

    @property
    def gammaval(self) -> float:
        return self._gammaval

    @gammaval.setter
    def gammaval(self, gammaval: float):
        if not isinstance(gammaval, (float, int)):
            raise ValueError("gammaval must a number")

        self._gammaval = float(gammaval)
        self.write(0x20, _float2int(self.gammaval))
        #print("setting gammaval to " + str(self.gammaval))

Test it in the Jupyter Notebook

Open the pynq_composable/custom_pipeline/02_first_custom_pipeline.ipynb notebook. File → Make a Copy and save it as GammaCorrection

Changes:

#Downlaod Composable Overlay: 
ol = Overlay("/home/xilinx/jupyter_notebooks/overlay/cv_dfx_4_pr.bit")

#Let us Compose:
gamma = cpipe.gammacorrection_accel

video_pipeline = [cpipe.hdmi_source_in, gamma, cpipe.hdmi_source_out]

cpipe.compose(video_pipeline)

#Play with IP
import time
for i in range(0,1000):
    gamma.gammaval = i*0.0025
    time.sleep(0.01)
gamma.gammaval=1

Then run the Jupyter Notebook.

TODO:

  • insert new functions into the DFX region
  • Path for the overlays isn’t default
  • Make the apps work
2 Likes

Hi @jcollier,

Thank you for sharing this :grinning:

One small comment, the bindto of the VitisVisionIP shouldn’t include the vlnv of the derived objects. So, 'xilinx.com:hls:gammacorrection_accel:1.0', should only be part of the gammacorrection class.
Otherwise, there’s a risk of VitisVisionIP being assigned as driver to the gammacorrection IP.

Mario

Thanks much. I will update my instructions. I was unclear how and why I would do that, but it worked.