Vitis Vision Libraries + PYNQ HelloWorld

Willy · July 27, 2022, 12:03pm

Hi all, @marioruiz
Refer to the PYNQ Helloworld for the resize IP example, I try to use the OtsuThreshold and Threshold function in vitis library to replace the opencv:

ret, img_binary = cv2.threshold(img_gray, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

Code in HLS IP:

void binaryimage_accel(stream_t& stream_in, 
		    stream_t& stream_out,
                   int rows, 
                   int cols) {


    #pragma HLS INTERFACE axis register both port=stream_in
    #pragma HLS INTERFACE axis register both port=stream_out

    #pragma HLS INTERFACE s_axilite port=rows offset=0x10      
    #pragma HLS INTERFACE s_axilite port=cols offset=0x18      
    #pragma HLS INTERFACE s_axilite port=return


    xf::cv::Mat<INTYPE, HEIGHT, WIDTH, NPIX> img_in(rows, cols);
    xf::cv::Mat<INTYPE, HEIGHT, WIDTH, NPIX> img_out(rows, cols);


    #pragma HLS DATAFLOW


    uint8_t otsuval;
    short int maxval = 255;

    // Convert stream to xf::cv::Mat
    axis2xfMat<DATA_WIDTH, INTYPE, HEIGHT, WIDTH, NPIX>(stream_in, img_in);

    // OtsuThreshold
    xf::cv::OtsuThreshold<INTYPE, HEIGHT, WIDTH, NPIX>(img_in, otsuval);


    // Thresholding wit the otsu threshould
    xf::cv::Threshold<XF_THRESHOLD_TYPE_BINARY, XF_8UC1, HEIGHT, WIDTH, NPIX>(img_in, img_out, otsuval, maxval);

    // Convert xf::cv::Mat to stream
    xfMat2axis<DATA_WIDTH, INTYPE, HEIGHT, WIDTH, NPIX>(img_out, stream_out);

}

Got build error as below, it seems the Threshold only accept the SRC_T = XF_8UC1.

ERROR: [HLS 207-3339] no matching function for call to ‘Threshold’ (src/binaryimage.cpp:122:5)
INFO: [HLS 207-4378] candidate function not viable: no known conversion from ‘xf::cv::Mat<XF_8UC3, 1080, 1920, XF_NPPC1>’ to ‘xf::cv::Mat<0, 1080, 1920, 1> &’ for 1st argument (/home/willychiang/Desktop/PYNQ_KV260/ip/vitis_lib/vision/L1/include/imgproc/xf_threshold.hpp:106:6)
"
https://github.com/Xilinx/Vitis_Libraries/blob/master/vision/L1/include/imgproc/xf_threshold.hpp

But the stream_in is converting to xf::cv::Mat by axis2xfMat with 24bit DATA_WIDTH (I think it only support 8UC3).
Does have any ways to transfer the 8UC3 Mat to 8UC1 foramt?

Another question about the rgb2gray_accel in Composable_Pipeline, why we need to transfer the gray image back to rgb?

Willy · July 27, 2022, 12:36pm

I suppose the rgb2gray (8UC3 to 8UC1) and gray2rgb (8UC1 to 8UC3 ) can also be used for color channel transfer when the sream_in image is gray image (8UC3).
After applying the rgb2gray and gray2rgb functions, my code seems is compiler pass now…
But I still want to know does my operations is legally and make sense…

Sample code:

#include “hls_stream.h”
#include “common/xf_common.hpp”
#include “common/xf_infra.hpp”
#include “imgproc/xf_otsuthreshold.hpp”
#include “imgproc/xf_threshold.hpp”
#include “imgproc/xf_cvt_color.hpp”
#include “imgproc/xf_duplicateimage.hpp”

#define DATA_WIDTH 24
#define NPIX XF_NPPC1

/* set the height and width */
#define WIDTH 1920
#define HEIGHT 1080

#define INTYPE XF_8UC3
#define TMPTYPE XF_8UC1

typedef ap_axiu<DATA_WIDTH,1,1,1> interface_t;
typedef hls::stream<interface_t> stream_t;

/*

We use the custom axis2xfMat and xfMat2axis and instead default
xf::cv::AXIvideo2xfMat and xf::cv::xfMat2AXIvideo
because the Hello-World uses a regular DMA.
So, we only need last is only asserted for final pixel of the image.
*/

template <int W, int TYPE, int ROWS, int COLS, int NPPC>
void axis2xfMat (hls::stream<ap_axiu<W, 1, 1, 1> >& AXI_video_strm, xf::cv::Mat<TYPE, ROWS, COLS, NPPC>& img) {
ap_axiu<W, 1, 1, 1> axi;

const int m_pix_width = XF_PIXELWIDTH(TYPE, NPPC) * XF_NPIXPERCYCLE(NPPC);

int rows = img.rows;
int cols = img.cols >> XF_BITSHIFT(NPPC);

assert(img.rows <= ROWS);
assert(img.cols <= COLS);

loop_row_axi2mat:
for (int i = 0; i < rows; i++) {
loop_col_zxi2mat:
for (int j = 0; j < cols; j++) {
#pragma HLS loop_flatten off
#pragma HLS pipeline II=1

        AXI_video_strm.read(axi);
        img.write(i*rows + j, axi.data(m_pix_width - 1, 0));
    }
}

}

template <int W, int TYPE, int ROWS, int COLS, int NPPC>
void xfMat2axis(xf::cv::Mat<TYPE, ROWS, COLS, NPPC>& img, hls::stream<ap_axiu<W, 1, 1, 1> >& dst) {
ap_axiu<W, 1, 1, 1> axi;

int rows = img.rows;
int cols = img.cols >> XF_BITSHIFT(NPPC);

assert(img.rows <= ROWS);
assert(img.cols <= COLS);

const int m_pix_width = XF_PIXELWIDTH(TYPE, NPPC) * XF_NPIXPERCYCLE(NPPC);

loop_row_mat2axi:
for (int i = 0; i < rows; i++) {
loop_col_mat2axi:
for (int j = 0; j < cols; j++) {
#pragma HLS loop_flatten off
#pragma HLS pipeline II = 1

        /*Assert last only in the last pixel*/
        if ((j == cols-1) && (i == rows-1)) {
            axi.last = 1;
        } else {
            axi.last = 0;
        }

        axi.data = 0;
        axi.data(m_pix_width - 1, 0) = img.read(i*rows + j);
        axi.keep = -1;
        dst.write(axi);
    }
}

}

void binaryimage_accel(stream_t& stream_in,
stream_t& stream_out,
int rows,
int cols) {

#pragma HLS INTERFACE axis register both port=stream_in
#pragma HLS INTERFACE axis register both port=stream_out

#pragma HLS INTERFACE s_axilite port=rows offset=0x10      
#pragma HLS INTERFACE s_axilite port=cols offset=0x18      
#pragma HLS INTERFACE s_axilite port=return


xf::cv::Mat<INTYPE, HEIGHT, WIDTH, NPIX> img_in(rows, cols);

xf::cv::Mat<TMPTYPE, HEIGHT, WIDTH, NPIX> img_tmp1(rows, cols);
xf::cv::Mat<TMPTYPE, HEIGHT, WIDTH, NPIX> img_tmp2(rows, cols);
xf::cv::Mat<TMPTYPE, HEIGHT, WIDTH, NPIX> img_tmp3(rows, cols);
xf::cv::Mat<TMPTYPE, HEIGHT, WIDTH, NPIX> img_tmp4(rows, cols);

xf::cv::Mat<INTYPE, HEIGHT, WIDTH, NPIX> img_out(rows, cols);


#pragma HLS DATAFLOW


uint8_t otsuval;
short int maxval = 255;

// Convert stream to xf::cv::Mat
axis2xfMat<DATA_WIDTH, INTYPE, HEIGHT, WIDTH, NPIX>(stream_in, img_in);


// Convert original image to grayscale
xf::cv::rgb2gray<INTYPE, TMPTYPE, HEIGHT, WIDTH, NPIX>(img_in, img_tmp1);

xf::cv::duplicateMat<TMPTYPE, HEIGHT, WIDTH, NPIX>(img_tmp1, img_tmp2, img_tmp3);

// OtsuThreshold
xf::cv::OtsuThreshold<TMPTYPE, HEIGHT, WIDTH, NPIX>(img_tmp2, otsuval);

// Thresholding wit the otsu threshould
xf::cv::Threshold<XF_THRESHOLD_TYPE_BINARY, XF_8UC1, HEIGHT, WIDTH, NPIX>(img_tmp3, img_tmp4, otsuval, maxval);


// Convert grayscale image to rgb
xf::cv::gray2rgb <TMPTYPE, INTYPE, HEIGHT, WIDTH, NPIX>(img_tmp4, img_out);

// Convert xf::cv::Mat to stream
xfMat2axis<DATA_WIDTH, INTYPE, HEIGHT, WIDTH, NPIX>(img_out, stream_out);

}

marioruiz · July 27, 2022, 2:38pm

Hi,

But the stream_in is converting to xf::cv::Mat by axis2xfMat with 24bit DATA_WIDTH (I think it only support 8UC3).

These functions are designed only for the resize function. If you want to do this transformation to a different number of channels, you will have to change the code.
It may be easier just to use the streaming version rather than a memory mapped one.

Another question about the rgb2gray_accel in Composable_Pipeline, why we need to transfer the gray image back to rgb?

I do this, to make sure that the output image will be only visible correctly even if this IP is the last one in the pipeline.

Mario

Willy · August 9, 2022, 11:48am

Hi @marioruiz
I still do some experiments for the helloworld project.
I remove the resize IP and add rgb2gray IP, other one threshold IP.

The rgb2gray IP is design to input (8u3c) and output (8u1c).
and put the output as input for the binary threshold IP (8u1c) and output (8u1c).
The result is system will hang on dma1.recvchannel.wait() which used for binary threshold IP.

My question is if we design the output format to 8u1c, should we remove the axis_dwidth_converter (24bits to 32bits), and also the same concept, for the input/output format are 8u1c, the dwidth_converter (32bits to 28bits) and (28bits to 32bits) need to be removed?

marioruiz · August 9, 2022, 12:09pm

The AXI Data width converter is added to match the data width in your IP streaming ports. So, you will need to adapt this.

For instance, for a 8u1c you need to do 32 to 8 and vice versa.

Mario

GGChe · January 22, 2024, 2:03pm

Hi there! I want to use the vision library in my project to compute the standard deviation of some buffer. I cant make it work, but I see that you are using it. How did you make it work? I mean, how did you settle the environment and everything to use the vision imgproc library, xf::cv::Mat etc?

Thanks!

Topic		Replies	Views
Vitis Vision Library on PYNQ Support	5	2563	June 17, 2021
Vitis vision core fails on PYNQ V2.5.1 Support	34	4039	June 23, 2021
Vitis Vision Libraries Support	14	1400	June 23, 2021
Vitis Vision Library Vivado Design Support	4	919	February 14, 2022
Vitis library crop function kria SOM kv260 Support	0	242	September 14, 2023

Vitis Vision Libraries + PYNQ HelloWorld

Related topics