# Added IP before HDMI out IP not working

**URL:** <https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079>\
**Category:** Support\
**Created:** [April 21, 2022, 7:48am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079 "2022-04-21T07:48:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![aripod](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@aripod](https://discuss.pynq.io/u/aripod)\
**Post date:** [April 21, 2022, 7:48am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/1 "2022-04-21T07:48:23Z")

</div>

Hello,

I re-built the base overlay for the pynq-Z1 (image version 2.7) and added an HLS IP between the output of the VDMA (MM2S) and the HDMI\_OUT IP (in\_stream):

 ![Screenshot from 2022-04-21 09-37-37](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/2/29e8b2b69b12f71b33a24c42143bbeb581fbf259.png)

The code for the IP is the following:

```auto
const uint32_t APP_COLS = 1920;
const uint32_t APP_ROWS = 1080;
const uint32_t APP_PIXELS_APP = APP_COLS* APP_ROWS;
const vx_uint32 APP_VEC_NUM = 1;

template <class TYPE, const size_t SIZE> //
struct vx_image_data { //
    TYPE pixel[SIZE];
    uint8 last;
};

typedef vx_image_data<vx_uint32, APP_VEC_NUM> app_u32_image;

void Passthrough(app_u32_image input[APP_PIXELS_APP], app_u32_image output[APP_PIXELS_APP]) {
#pragma HLS interface ap_ctrl_none port=return
#pragma HLS INTERFACE axis port=input
#pragma HLS INTERFACE axis port=output

	// Computation
#pragma HLS DATAFLOW

	app_u32_image tmp;
	for(int i=0; i<APP_PIXELS_APP; i++) {
		tmp = input[i];
		output[i] = tmp;
	}
}

```

And the jupyter notebook is:

```auto
from pynq import Overlay
from pynq.lib.video import *

#overlay = Overlay('base.bit')
overlay = Overlay('/home/xilinx/customOverlays/passthrough.bit')
hdmi_in = overlay.video.hdmi_in
hdmi_out = overlay.video.hdmi_out

hdmi_in.configure(PIXEL_RGBA)
hdmi_out.configure(hdmi_in.mode, PIXEL_RGBA)
hdmi_in.cacheable_frames = False
hdmi_out.cacheable_frames = False
hdmi_in.start()
hdmi_out.start()
mymode = hdmi_in.mode
print("My mode: "+str(mymode))
height = hdmi_in.mode.height
width = hdmi_in.mode.width
bpp = hdmi_in.mode.bits_per_pixel

hdmi_in.tie(hdmi_out)

```

I have an hdmi camera connected to the HDMI IN and a screen connected to the HDMI OUT. I can see the images from the camera when I load the base overlay but when I load the passthrough one, there is no HDMI signal and the screen goes into sleep mode.

The HLS IP takes RGBA (32 bit) pixels, that is why I sent the hdmi {in,out} mode to `PIXEL_RGBA`. Is there an issue with the HLS itself or where the IP is added into the block design?

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\
**Post date:** [April 21, 2022, 10:13am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/2 "2022-04-21T10:13:55Z")

</div>

Hi @aripod,

Using structs and arrays is discouraged in Vitis HLS to works with streams, and (I believe) it should not work. In addition to that, you are not generating the `user` signal, which indicates start of frame.

I would suggest you look at the PYNQ HLS IP and use `hls::stream` for streams [PYNQ/boards/ip/hls/pixel\_unpack at master · Xilinx/PYNQ · GitHub](https://github.com/Xilinx/PYNQ/tree/master/boards/ip/hls/pixel_unpack)

Mario

---

<div class="post-metadata">

**Author:** ![aripod](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@aripod](https://discuss.pynq.io/u/aripod)\
**Post date:** [April 21, 2022, 3:12pm UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/3 "2022-04-21T15:12:47Z")

</div>

Hi @marioruiz,

Thanks for your help. I added the `user` signal but still doesn’t work. I still don’t get a “valid” hdmi signal and the screen goes into sleep mode.  
I am not using Vitis but vivado HLS 2019.2 to generate the IP and then vivado 2020.2 for the overlay.

I added it like:

```auto
template <class TYPE, const size_t SIZE> //
struct vx_image_data { //
    TYPE pixel[SIZE];
    ap_uint<1> last;
    ap_uint<1> user;
};

```

I’ve notice that the hdmi\_in IP, out\_stream has also `tkeep` but in\_stream in hdmi\_out doesn’t. Should the input of my IP also have tkeep or is that optional?

 ![Screenshot from 2022-04-21 17-11-14](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/6/6b2c26541d07edba29013797066125bf98527a4a.png)

What if I don’t do it like that but rather add another DMA and the IPs without `tuser` and `tkeep` (only `tvalid`, `tdata`, `tready` and `tlast`)?

 ![Screenshot from 2022-04-21 17-27-26](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/1/1ad57546255420818c50edac85ee9c92ee1efa58.jpeg)

Here I added HP3 for axi\_dma\_1, all to the 142MHz clk as the video IPs.

How is it with the data types? I tried that and had an issue due to the data type of the PynqBuffer being always uint8 (see [here](https://discuss.pynq.io/t/hdmi-data-type-as-32-bits/3978)).

---

<div class="post-metadata">

**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\
**Post date:** [April 21, 2022, 4:23pm UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/4 "2022-04-21T16:23:14Z")

</div>

Hi,

You should be using Vitis HLS 2020.2 to match the Vivado version as well as using `hls:stream`.

You should also verify that your HLS IP is achieving an II=1. Otherwise, you will not get an output.

There a two part tutorial on DMA here

> [@Tutorial: PYNQ DMA (Part 1: Hardware design)](https://discuss.pynq.io/t/tutorial-pynq-dma-part-1-hardware-design/3133):
>
> PYNQ DMA tutorial (Part 1: Hardware design) This tutorial will show you how to use the Xilinx AXI DMA with PYNQ. It will cover adding the AXI DMA to a new Vivado hardware design and show how the DMA can be controlled from PYNQ. This tutorial is based on the v2.6 PYNQ image and will use Vivado 2020.1. If you are using a different PYNQ version you should be able to follow the same steps in this tutorial, but you should make sure you are using the supported version of Vivado for that PYNQ release. …

> How is it with the data types? I tried that and had an issue due to the data type of the PynqBuffer being always uint8 (see [here](https://discuss.pynq.io/t/hdmi-data-type-as-32-bits/3978)).

You are using `PIXEL_RGB` in that case, so you will get three channels per pixel 8-bit each that’s why you se uint8

Mario

---

<div class="post-metadata">

**Author:** ![aripod](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@aripod](https://discuss.pynq.io/u/aripod)\
**Post date:** [April 21, 2022, 4:30pm UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/5 "2022-04-21T16:30:17Z")

</div>

> [@marioruiz](#):
>
> You are using `PIXEL_RGB` in that case, so you will get three channels per pixel 8-bit each that’s why you se uint8

Hi @marioruiz,

Thanks again for your help.

As you can see in the last image, the DMA has 32 bits for `tdata` and I set `PIXEL_RGBA`. I understand that I would get now four channels with 8 bits which is what I get with the pynqbuffer. However, if I allocate a buffer for the dma as in the tutorials `input_buffer = allocate(shape=(5,), dtype=np.uint32)` and `output_buffer = allocate(shape=(5,), dtype=np.uint32)`, how can I set the pynqbuffer that I get the HDMI frame in to also have `dtype=np.uint32` so I can stream each frame that comes from the VDMA through the DMA? How should I use the inframe which is a pynqbuffer from VDMA to stream the data to DMA instead of using `input_buffer`? As far as I understand, it is a datatype issue, isn’t it? I insists on this because the original IP (with only `tlast` works with dma using pre-stored images, I want now to use it with live-feed from the hdmi).

---

<div class="post-metadata">

**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\
**Post date:** [April 21, 2022, 4:48pm UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/6 "2022-04-21T16:48:50Z")

</div>

Hi,

The buffer you get from the VDMA is always going to have multiple channels per each pixel. So, you want to pack 4x 8-bit into a 1x 32-bit element. In essence, you want to go from 1920x1080x4 unit8 to 1920x1080x1 uint32.

You can try to pack the 4 channels into a single uint32, but it is going to be slow as it involves shifts and multiplications. You can explore numpy `frombuffer` and `tobytes` to reinterpret to pixels.

You can try something like this [Any method to reduce the calculation time on PS (PYNQ-z2) - #5 by marioruiz](https://discuss.pynq.io/t/any-method-to-reduce-the-calculation-time-on-ps-pynq-z2/3915/5)

You will get your VDMA buffer then you will create your DMA buffer

```python
buf_a =pynq.allocate((1920,1080), dtype=np.uint32) 

```

Instead of assigning anything to this buffer you change the physical address to the VDMA buffer

```python
buf_a.physical_address = buf_vdma.physical_address 

```

Hopefully this would do the trick.

The real question is why do you want to use a DMA for video? The VDMA is optimized for Video

Mario

---

<div class="post-metadata">

**Author:** ![aripod](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@aripod](https://discuss.pynq.io/u/aripod)\
**Post date:** [April 22, 2022, 9:33am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/7 "2022-04-22T09:33:48Z")

</div>

Hi @marioruiz ,

Thanks again for your help and the “trick” to use the physical address. I tried with the `frombuffer` and `tobytes` methods but the performances decreased quite a lot as expected.

> [@marioruiz](#):
>
> The real question is why do you want to use a DMA for video? The VDMA is optimized for Video

That’s a fair question. I wanted to do that as I am using a specific image processing library and I didn’t want to modify it. The passthrough following your example worked:

```auto
#include <ap_fixed.h>
#include <ap_int.h>
#include "hls_stream.h"
#include <ap_axi_sdata.h>

typedef ap_axiu<32,1,0,0> wide_pixel;
typedef hls::stream<wide_pixel> wide_stream;

void Passthrough(wide_stream& stream_in, wide_stream& stream_out) {
#pragma HLS INTERFACE ap_ctrl_none port=return
//#pragma HLS INTERFACE s_axilite register port=mode clock=control
#pragma HLS INTERFACE axis depth=32 port=stream_in register
#pragma HLS INTERFACE axis depth=32 port=stream_out register

	wide_pixel in_pixel;
	wide_pixel out_pixel;
	bool last = false;

	while(!last) {
#pragma HLS pipeline II=1
		stream_in.read(in_pixel);
		out_pixel.data = in_pixel.data;
		out_pixel.last = in_pixel.last;
		out_pixel.user = in_pixel.user;
		last = in_pixel.last;
		stream_out.write(out_pixel);
	}
}

```

I will have create a wrapper around the functions of the img. proc. library around that passthrough example.

---

<div class="post-metadata">

**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\
**Post date:** [April 22, 2022, 10:33am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/8 "2022-04-22T10:33:06Z")

</div>

Hi @aripod  
Great 😀

If you are not accessing the array from software, you may want to write your code in a less hacky way.

```python
buf_a = pynq.allocate((1920,1080,4), dtype=np.uint8)
buf_a [:] = buf_vdma[:]

```

Or you can directly reuse `buf_vdma`, as pynq.allocate((1920,1080,4), dtype=np.uint8) is equivalent to pynq.allocate((1920,1080), dtype=np.uint32) in physical memory.

Mario

---

<div class="post-metadata">

**Author:** ![aripod](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@aripod](https://discuss.pynq.io/u/aripod)\
**Post date:** [April 22, 2022, 11:10am UTC](https://discuss.pynq.io/t/added-ip-before-hdmi-out-ip-not-working/4079/9 "2022-04-22T11:10:55Z")

</div>

Hi @marioruiz,

The less hacky way works perfectly. However, I had to change it to `buf_a = pynq.allocate((1080,1920, 4), dtype=np.uint8)`.

I will focus on using hls::stream and have everything on hardware, just using `hdmi_in.tie(hdmi_out)`.

Thanks for the help