PYNQ: PYTHON PRODUCTIVITY

Vitis vision core fails on PYNQ V2.5.1

I used Vitis vision Github library to replicate the resize stream IP (Hello world example) on PYNQ V2.5.1.

It fails by stalling on the receive side.
On the width converters I use both keep and strb and tuser set to 1.

Any idea what has changed.
For the jupyter notebook I use the existing example with RGB format.

    #include "xf_resize_config.h"


#define NPC_T XF_NPPC1


#if RGB
#define TYPE XF_8UC3
#define CH_TYPE XF_RGB
#else
#define TYPE XF_8UC1
#define CH_TYPE XF_GRAY
#endif

#define INPUT_PTR_WIDTH   24
#define OUTPUT_PTR_WIDTH  24

void resize_accel(hls::stream< xf::cv::ap_axiu<24, 1, 1, 1> > &srcPtr, hls::stream< xf::cv::ap_axiu<24, 1, 1, 1> > &dstPtr, int rows_in, int cols_in, int rows_out,  int cols_out);




void resize_accel(hls::stream< xf::cv::ap_axiu<INPUT_PTR_WIDTH, 1, 1, 1> > &srcPtr, 
				  hls::stream< xf::cv::ap_axiu<OUTPUT_PTR_WIDTH, 1, 1, 1> > &dstPtr, 
				  int rows_in, int cols_in,  int rows_out,  int cols_out) 
{
// clang-format off

    #pragma HLS INTERFACE axis port=srcPtr
    #pragma HLS INTERFACE axis port=dstPtr

    #pragma HLS INTERFACE s_axilite port=rows_in              
    #pragma HLS INTERFACE s_axilite port=cols_in              
    #pragma HLS INTERFACE s_axilite port=rows_out              
    #pragma HLS INTERFACE s_axilite port=cols_out              
    #pragma HLS INTERFACE s_axilite port=return
    // clang-format on

    xf::cv::Mat<TYPE, HEIGHT, WIDTH, NPC_T> in_mat(rows_in, cols_in);

    xf::cv::Mat<TYPE, NEWHEIGHT, NEWWIDTH, NPC_T> out_mat(rows_out, cols_out);

// clang-format off
    #pragma HLS DATAFLOW
    // clang-format on


    xf::cv::AXIvideo2xfMat<INPUT_PTR_WIDTH,TYPE,HEIGHT,WIDTH,NPC_T>(srcPtr, in_mat);

    xf::cv::resize<INTERPOLATION, TYPE, HEIGHT, WIDTH, NEWHEIGHT, NEWWIDTH, NPC_T, MAXDOWNSCALE>(in_mat, out_mat);
    
    xf::cv::xfMat2AXIvideo<OUTPUT_PTR_WIDTH,TYPE,NEWHEIGHT,NEWWIDTH,NPC_T>(out_mat, dstPtr);    

}

Can you post your block design as well? Have you changed anything there? For the axi width converter, I don’t remember configuring it manually. If that IP also exists in the resizer design, it would be better to use the same configuration.

Attached is the Vitis Vision stream IP core and block design.VitisResize_MainDesign.tcl (31.1 KB) xilinx_com_hls_resize_accel_1_0.zip (132.3 KB)

Nothing was changed. The ip was replaced. I noticed that the generated IP’s have extra sideband signals.

I did some comparison but it looks like a lot of things have changed?

Can you make sure you start with the original design and only replace that core?

This is the original design.

Apart from Vivado version and IP core I don’t see any other difference.

I have 2019.1 and 2020.1 installed so I had to downgrade to the 2019.1 version. Not sure if 2020.1 is compatible with PYNQ2.5.1 so I went with 2019.1

The only other change was adding the Vitis IP. There are only 4 IP blocks (DMA, resize and AXIS width converters) so I don’t see how can there be many changes.

Unless I am missing something looks like the issue is with the Vitis Core not the Vivado design.

When you say you changed the core, how did you change it? There might be configurations that are not good for the python code. For example, the pixel byte width. For the side band signals, I would just use the original settings.

I used the attached zip file under L1 Vitis Vision folder and issued vivado_hls -f run.tcl

VitisStreanResizePYNQ.zip (8.5 KB)

The sidebands is the part that is not clear.
Using the HLS code above infers the sideband signals automatically, whereas the resize IP core from the old xfopencV did not have these sidebands .
The designers in that release wrote C++ custom code to convert from the axis stream to xfmat in the old library.

void axis2xfMat (xf::Mat<TYPE, HEIGHT, WIDTH, NPC1> &_src, axis_t *src, int src_rows, int src_cols) {
#pragma HLS inline off

	for (int i=0; i<src_rows; i++) {
		for (int j=0; j<src_cols; j++) {
#pragma HLS pipeline
#pragma HLS loop_flatten off
			_src.data[i*src_cols+j] = src[i*src_cols+j].data;
		}
	}
}

void xfMat2axis (xf::Mat<TYPE, HEIGHT, WIDTH, NPC1> &_dst, axis_t *dst, int dst_rows, int dst_cols) {
#pragma HLS inline off

	for (int i=0; i<dst_rows; i++) {
		for (int j=0; j<dst_cols; j++) {
#pragma HLS pipeline
#pragma HLS loop_flatten off
			ap_uint<1> tmp = 0;
			if ((i==dst_rows-1) && (j== dst_cols-1)) {
				tmp = 1;
			}
			dst[i*dst_cols+j].last = tmp;
			dst[i*dst_cols+j].data = _dst.data[i*dst_cols+j];
		}
	}
}

I tried to use Vitis Vision L1 layer to generate IP cores and I see that Xilinx has released API calls for doing this now:
AXIvideo2xfMat and xfMat2AXIvideo.

But , the catch is that you cannot remove the sidebands as far as I understand. Otherwise the API calls fail.

Attached is the Vitis vision modified project.
I also tested a gray only (X8UC1) version and that also fails.
My guess , the issue is with the side-bands and the way the PYNQ DMA is configured but I may be missing something.

I think it might be related to the Vitis version (but I am not sure). To my best knowledge, Vitis is released from 2019.2 so I can’t tell the impact of back-porting it to 2019.1.

I can create the project using Vivado 2020.1 but I think the sideband issue is the same. How did the PYNQ team create the IP for resize on the current release? I don’t see the sources for the resize IP on the Hello World repo.

Source is HLS and is here:
https://github.com/Xilinx/PYNQ-HelloWorld/tree/master/boards/ip

Cathal

The IP source code is not functional with the current Vitis Vison library. (2020.1)
I tried to compile and I am getting the following errors:

~/Documents/resizePYNQ$ vivado_hls -f script.tcl 

****** Vivado(TM) HLS - High-Level Synthesis from C, C++ and SystemC v2020.1 (64-bit)
  **** SW Build 2902540 on Wed May 27 19:54:35 MDT 2020
  **** IP Build 2902112 on Wed May 27 22:43:36 MDT 2020
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.

source /tools/Xilinx/Vivado/2020.1/scripts/vivado_hls/hls.tcl -notrace
INFO: [HLS 200-10] Running '/tools/Xilinx/Vivado/2020.1/bin/unwrapped/lnx64.o/vivado_hls'
INFO: [HLS 200-10] For user 'iti' on host 'iti-VirtualBox' (Linux_x86_64 version 5.4.0-42-generic) on Wed Oct 07 10:51:48 EDT 2020
INFO: [HLS 200-10] On os Ubuntu 18.04.5 LTS
INFO: [HLS 200-10] In directory '/home/iti/Documents/resizePYNQ'
Sourcing Tcl script 'script.tcl'
INFO: [HLS 200-10] Opening project '/home/iti/Documents/resizePYNQ/resize'.
INFO: [HLS 200-10] Adding design file '../Vitis_Libraries/vision/L2/examples/resize/xf_resize_config.h' to the project
INFO: [HLS 200-10] Adding design file '../Vitis_Libraries/vision/L2/tests/resize/resize_DOWN_BILINEAR_NO_RGB/xf_config_params.h' to the project
INFO: [HLS 200-10] Adding design file 'xf_axis_config.h' to the project
INFO: [HLS 200-10] Adding design file 'xf_resize_accel.cpp' to the project
INFO: [HLS 200-10] Adding test bench file 'xf_resize_tb.cpp' to the project
INFO: [HLS 200-10] Opening solution '/home/iti/Documents/resizePYNQ/resize/solution1'.
INFO: [SYN 201-201] Setting up clock 'default' with a period of 10ns.
INFO: [SYN 201-201] Setting up clock 'default' with an uncertainty of 2.7ns.
INFO: [HLS 200-10] Setting target device to 'xc7z020-clg484-2'
INFO: [SCHED 204-61] Option 'relax_ii_for_timing' is enabled, will increase II to preserve clock frequency constraints.
INFO: [HLS 200-10] Analyzing design file 'xf_resize_accel.cpp' ... 
ERROR: [HLS 200-70] Compilation errors found: In file included from xf_resize_accel.cpp:1:
xf_resize_accel.cpp:42:11: error: type 'hls::stream<DATATYPE, 2>' does not provide a subscript operator
 _src.data[i*src_cols+j] = src[i*src_cols+j].data;
 ~~~~~~~~~^~~~~~~~~~~~~
xf_resize_accel.cpp:62:38: error: type 'hls::stream<DATATYPE, 2>' does not provide a subscript operator
   dst[i*dst_cols+j].data = _dst.data[i*dst_cols+j];
                            ~~~~~~~~~^~~~~~~~~~~~~
In file included from xf_resize_accel.cpp:1:
In file included from xf_resize_accel.cpp:30:
In file included from ./xf_axis_config.h:36:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:367:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed.h:55:
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed_base.h:838:13: warning: shift count is negative [-Wshift-count-negative]
      ret.V <<= (_AP_I - _AP_W);
            ^   ~~~~~~~~~~~~~~~
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int_base.h:359:18: note: in instantiation of member function 'ap_fixed_base<53, 21, true, 5, 3, 0>::to_ap_int_base' requested here
    Base::V = op.to_ap_int_base().V;
                 ^
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:249:9: note: in instantiation of function template specialization 'ap_int_base<17, false>::ap_int_base<53, 21, true, 5, 3, 0>' requested here
      : Base((ap_fixed_base<_AP_W2, _AP_I2, true, _AP_Q2, _AP_O2, _AP_N2>)op) {}
        ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:233:22: note: in instantiation of function template specialization 'ap_uint<17>::ap_uint<53, 21, 5, 3, 0>' requested here
        nextYScale = indexy_pre + scaley;
                     ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:397:13: note: in instantiation of function template specialization 'computeInterpolation<1, 42, 20, 17, 48, 16, 48, 16, 1>' requested here
            computeInterpolation<INTERPOLATION_TYPE, COMP_INDEX_WIDTH, COMP_INDEX_INT, INDEX_INT, SCALE_WIDTH,
            ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize.hpp:63:9: note: in instantiation of function template specialization 'resizeNNBilinear<9, 2160, 3840, 1, 1080, 1920, 1, 2>' requested here
        resizeNNBilinear<TYPE, SRC_ROWS, SRC_COLS, NPC, DST_ROWS, DST_COLS, INTERPOLATION_TYPE, MAX_DOWN_SCALE>(_src,
        ^
xf_resize_accel.cpp:87:2: note: in instantiation of function template specialization 'xf::cv::resize<1, 9, 2160, 3840, 1080, 1920, 1, 2>' requested here
 xf::cv::resize<1, XF_8UC3, 2160, 3840, 1080, 1920, XF_NPPC1, 2>(src_mat, dst_mat);
 ^
In file included from xf_resize_accel.cpp:1:
In file included from xf_resize_accel.cpp:30:
In file included from ./xf_axis_config.h:36:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:367:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed.h:55:
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed_base.h:838:13: warning: shift count is negative [-Wshift-count-negative]
      ret.V <<= (_AP_I - _AP_W);
            ^   ~~~~~~~~~~~~~~~
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int_base.h:359:18: note: in instantiation of member function 'ap_fixed_base<42, 20, true, 5, 3, 0>::to_ap_int_base' requested here
    Base::V = op.to_ap_int_base().V;
                 ^
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:249:9: note: in instantiation of function template specialization 'ap_int_base<17, false>::ap_int_base<42, 20, true, 5, 3, 0>' requested here
      : Base((ap_fixed_base<_AP_W2, _AP_I2, true, _AP_Q2, _AP_O2, _AP_N2>)op) {}
        ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:234:18: note: in instantiation of function template specialization 'ap_uint<17>::ap_uint<42, 20, 5, 3, 0>' requested here
        indexy = (ap_uint<INDEX_INT>)indexy_pre;
                 ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:397:13: note: in instantiation of function template specialization 'computeInterpolation<1, 42, 20, 17, 48, 16, 48, 16, 1>' requested here
            computeInterpolation<INTERPOLATION_TYPE, COMP_INDEX_WIDTH, COMP_INDEX_INT, INDEX_INT, SCALE_WIDTH,
            ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize.hpp:63:9: note: in instantiation of function template specialization 'resizeNNBilinear<9, 2160, 3840, 1, 1080, 1920, 1, 2>' requested here
        resizeNNBilinear<TYPE, SRC_ROWS, SRC_COLS, NPC, DST_ROWS, DST_COLS, INTERPOLATION_TYPE, MAX_DOWN_SCALE>(_src,
        ^
xf_resize_accel.cpp:87:2: note: in instantiation of function template specialization 'xf::cv::resize<1, 9, 2160, 3840, 1080, 1920, 1, 2>' requested here
 xf::cv::resize<1, XF_8UC3, 2160, 3840, 1080, 1920, XF_NPPC1, 2>(src_mat, dst_mat);
 ^
In file included from xf_resize_accel.cpp:1:
In file included from xf_resize_accel.cpp:30:
In file included from ./xf_axis_config.h:36:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:367:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed.h:55:
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed_base.h:838:13: warning: shift count is negative [-Wshift-count-negative]
      ret.V <<= (_AP_I - _AP_W);
            ^   ~~~~~~~~~~~~~~~
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int_base.h:359:18: note: in instantiation of member function 'ap_fixed_base<43, 21, true, 5, 3, 0>::to_ap_int_base' requested here
    Base::V = op.to_ap_int_base().V;
                 ^
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:249:9: note: in instantiation of function template specialization 'ap_int_base<17, false>::ap_int_base<43, 21, true, 5, 3, 0>' requested here
      : Base((ap_fixed_base<_AP_W2, _AP_I2, true, _AP_Q2, _AP_O2, _AP_N2>)op) {}
        ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:237:22: note: in instantiation of function template specialization 'ap_uint<17>::ap_uint<43, 21, 5, 3, 0>' requested here
        nextYScale = indexy_pre + (ap_fixed<COMP_INDEX_WIDTH, COMP_INDEX_INT>)scaley;
                     ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:397:13: note: in instantiation of function template specialization 'computeInterpolation<1, 42, 20, 17, 48, 16, 48, 16, 1>' requested here
            computeInterpolation<INTERPOLATION_TYPE, COMP_INDEX_WIDTH, COMP_INDEX_INT, INDEX_INT, SCALE_WIDTH,
            ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize.hpp:63:9: note: in instantiation of function template specialization 'resizeNNBilinear<9, 2160, 3840, 1, 1080, 1920, 1, 2>' requested here
        resizeNNBilinear<TYPE, SRC_ROWS, SRC_COLS, NPC, DST_ROWS, DST_COLS, INTERPOLATION_TYPE, MAX_DOWN_SCALE>(_src,
        ^
xf_resize_accel.cpp:87:2: note: in instantiation of function template specialization 'xf::cv::resize<1, 9, 2160, 3840, 1080, 1920, 1, 2>' requested here
 xf::cv::resize<1, XF_8UC3, 2160, 3840, 1080, 1920, XF_NPPC1, 2>(src_mat, dst_mat);
 ^
In file included from xf_resize_accel.cpp:1:
In file included from xf_resize_accel.cpp:30:
In file included from ./xf_axis_config.h:36:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:367:
In file included from /tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed.h:55:
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_fixed_base.h:838:13: warning: shift count is negative [-Wshift-count-negative]
      ret.V <<= (_AP_I - _AP_W);
            ^   ~~~~~~~~~~~~~~~
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int_base.h:359:18: note: in instantiation of member function 'ap_fixed_base<32, 22, true, 5, 3, 0>::to_ap_int_base' requested here
    Base::V = op.to_ap_int_base().V;
                 ^
/tools/Xilinx/Vivado/2020.1/common/technology/autopilot/ap_int.h:249:9: note: in instantiation of function template specialization 'ap_int_base<8, false>::ap_int_base<32, 22, true, 5, 3, 0>' requested here
      : Base((ap_fixed_base<_AP_W2, _AP_I2, true, _AP_Q2, _AP_O2, _AP_N2>)op) {}
        ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:57:17: note: in instantiation of function template specialization 'ap_uint<8>::ap_uint<32, 22, 5, 3, 0>' requested here
        pixel = (typename DataType<DEPTH, NPPC>::cname)((ap_fixed<32, 22>)(P1 + P2 + P3 + P4));
                ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:140:13: note: in instantiation of function template specialization 'interpolatePixel<9, 1, 1>' requested here
            interpolatePixel<DEPTH, INTERPOLATION_TYPE, NPPC>(
            ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize_nn_bilinear.hpp:498:21: note: in instantiation of function template specialization 'computeOutputPixel<9, 1, 1, 17, 2, 48, 16>' requested here
                    computeOutputPixel<SRC_TYPE, INTERPOLATION_TYPE, NPPC, INDEX_INT, BUFFER_WORDS, WEIGHT_WIDTH,
                    ^
../Vitis_Libraries/vision/L1/include/imgproc/xf_resize.hpp:63:9: note: in instantiation of function template specialization 'resizeNNBilinear<9, 2160, 3840, 1, 1080, 1920, 1, 2>' requested here
        resizeNNBilinear<TYPE, SRC_ROWS, SRC_COLS, NPC, DST_ROWS, DST_COLS, INTERPOLATION_TYPE, MAX_DOWN_SCALE>(_src,
        ^
xf_resize_accel.cpp:87:2: note: in instantiation of function template specialization 'xf::cv::resize<1, 9, 2160, 3840, 1080, 1920, 1, 2>' requested here
 xf::cv::resize<1, XF_8UC3, 2160, 3840, 1080, 1920, XF_NPPC1, 2>(src_mat, dst_mat);
 ^
4 warnings and 2 errors generated.
Failed during preprocessing.
    while executing
"source script.tcl"
    ("uplevel" body line 1)
    invoked from within
"uplevel \#0 [list source $arg] "

INFO: [Common 17-206] Exiting vivado_hls at Wed Oct  7 10:52:00 2020...

OK, answering my own question ; the Vitis version linked vitis_lib @ 40b98f7 has a couple of bugs on the xf_infra.hpp and xf_axi.hpp files.

I still don’t understand why using IP’s with sidebands inferred by cvMat2AXIvideoxf() does not work?

@rock, @cathalmccabe.

Hi,
a) Is there any explanation why the PYNQ team has not used xf::cv::cvMat2AXIvideoxf and *xf::cv::AXIvideo2cvMatxf() but implemented a custom stream datatype to xfmat for the conversion functions?

Is there any updated version that uses the new Vitis vision release for the xfresize example?

I tried to modify the above functions by removing the keep signals and using the same custom stream datatype. .Now the sidebands do not get generated but the core still fails with the exact error as above.

So that would imply that the issue seems to be related to the stream conversion functions and dma library. Is that correct?

b) Does PYNQ support opening xclbin files with Vitis on ZYNQ devices?