I’m using pynq-z2 for image processing project. Basically, I’m using hdmi_in to read input frames and then these frames will be transferred to the designed process ip by dma, then hdmi_out will display the image on the monitor. Everything is fine when I fed a image without using ‘for’ loop to test the ip block, and it generated right result on the monitor. However, when I tried to use the ip inside the for loop, it can only shows the last computed frame, and other output frames are totally black on my monitor. To identify the bug, I insert a print(frame_out) sentence to check wether frame_out is empty. Sadly, it is not empty. That’s so weird!! Any suggestions?
Can you provide more info about your design?
What the IP is and how it is connected, Block diagram, what software calls you are using etc.
Cathal
The ip is a moving object detect core, and it collects two input frames and the ip will detect the moving objects and mark the object contour with red dots.
HLS CODE:
void frame_diff(AXI_STREAM & INPUT_STREAM1, AXI_STREAM & INPUT_STREAM2, AXI_STREAM & OUTPUT_STREAM)
{
#pragma HLS INTERFACE axis port=INPUT_STREAM1
#pragma HLS INTERFACE axis port=INPUT_STREAM2
#pragma HLS INTERFACE axis port=OUTPUT_STREAM
#pragma HLS interface ap_ctrl_none port=return
RGB_IMAGE img_01(MAX_HEIGHT, MAX_WIDTH);
RGB_IMAGE img_02(MAX_HEIGHT, MAX_WIDTH);
RGB_IMAGE img_03(MAX_HEIGHT, MAX_WIDTH);
RGB_IMAGE img_c1(MAX_HEIGHT, MAX_WIDTH);
RGB_IMAGE img_c2(MAX_HEIGHT, MAX_WIDTH);
GRAY_IMAGE img_1(MAX_HEIGHT, MAX_WIDTH);
GRAY_IMAGE img_2(MAX_HEIGHT, MAX_WIDTH);
GRAY_IMAGE img_3(MAX_HEIGHT, MAX_WIDTH);
GRAY_IMAGE img_4(MAX_HEIGHT, MAX_WIDTH);
hls::Scalar<1,unsigned char> _s1;
hls::Scalar<3,unsigned char> _s2;
hls::Scalar<3,unsigned char> _d;
#pragma HLS STREAM variable=img_c2 depth=8000
#pragma HLS dataflow
hls::AXIvideo2Mat_DMA(INPUT_STREAM1, img_01);
hls::AXIvideo2Mat_DMA(INPUT_STREAM2, img_02);
hls::Duplicate(img_01,img_c1,img_c2);
hls::CvtColor<HLS_BGR2GRAY>(img_c1,img_1);
hls::CvtColor<HLS_BGR2GRAY>(img_02,img_2);
hls::AbsDiff(img_1,img_2,img_3);
hls::Threshold(img_3,img_4,80,255,HLS_THRESH_BINARY);
loop_height: for(int i= 0; i < 1080; i++) {
loop_width: for (int j= 0; j < 1920; j++) {
#pragma HLS LOOP_FLATTEN OFF
#pragma HLS PIPELINE
img_4 >> _s1 ;
img_c2 >> _s2 ;
if(_s1.val[0]==255)
{
_d.val[0] = 0;
_d.val[1] = 0;
_d.val[2] = 255;
}
else
{
_d.val[0] = _s2.val[0];
_d.val[1] = _s2.val[1];
_d.val[2] = _s2.val[2];
}
img_03 << _d;
// img_03.write(_d);
}
}
hls::Mat2AXIvideo_DMA(img_03, OUTPUT_STREAM);
VIVADO BLOCK DESIGN:
8UC3.pdf (146.7 KB)
Jupyter Code:
class DMA():
def pre_process(src1,src2):
dma_0.sendchannel.transfer(src1)
dma_1.sendchannel.transfer(src2)
dma_0.recvchannel.transfer(frame_out)
dma_0.sendchannel.wait()
dma_1.sendchannel.wait()
dma_0.recvchannel.wait()
return frame_out
from pynq import Xlnk
xlnk = Xlnk()
init = 0
numframes = 100
frame_out = xlnk.cma_array(shape=(1080,1920,3), dtype=np.uint8)
start = time.time()
i=0
while i<numframes:
frame_curr = hdmi_in.readframe()
# Capture frame-by-frame
if init==0:
frame_prev = frame_curr
init=1
DMA.pre_process(frame_prev,frame_curr)
print(frame_out)
frame_prev = frame_curr
end_time = time.time()
fps=1.0/(end_time-start_time)
start_time = end_time
cv2.putText(frame_out,‘FPS:’+ str(fps), (10, 10), cv2.FONT_HERSHEY_PLAIN, 1.0, (255, 255, 255), 2)
hdmi_out.writeframe(frame_out)
i+=1
print(numframes/(time.time()-start))
I designed another ip for 720p video, and it works fine with the code above. Perhaps the performance for 1080p is not so good.
Hi @Tony_Ke0, I am facing a similar problem with the Pynq-Z2. I only have stability at 720p.
Digging around the video overlay I saw that the Pixel clock is 142MHz, less than the 148.5MHz required for the 1080p @60Hz.
@cathalmccabe, could you please let me know if that would be fine to put a clocking wizzard to convert pixel clock to 148.5MHz or the 142MHz clock was propositally chosen due to device limitations?
Thank you
The 142 MHz clock is not the pixel clock. The pixel clock is separate.
If this clock is slower than the pixel clock, you need a bigger FIFO (in the AXI stream to video IP), so increasing this clock might make it harder to close timing, but you should/may be able to reduce buffer. I wouldn’t do this. I think this was set to this freq as the minimum to process 1080p data without the sync and blanking periods, but open to correction on this.
The pixel clock is generated from the Dynamic clock generator. For the speedgrade used in the Zynq chip for this board (-1), 1080p is outside the spec for the IO. The spec is for worse case which is why you might see 1080p works for some people on the forums, and not the others. i.e. there will be variation between chips, and most boards are running at room temperature, not at corner case temperatures. I find some equipment is more sensitive. E.g. One monitor may work, another may not with the same board. According to the spec, the board can only meet 720p.
1080p may be OK for playing around with, or for hobbyists - if it works with your equipment, but you wouldn’t use this Zynq chip for a commercial application @1080p. Perhaps think about this as overclocking.
Cathal
1 Like
Hi @cathalmccabe, thank you very much indeed for this information.
So the limiting factor is the IO pins? It is not very clear for me. I have a Pynq-Z2 and a Zybo Z7-20. They both have the same chip (xc7z020-clg400). The zybo meets 1080p without problem (if using the HDMI demo + some custom IPs: Zybo Z7 HDMI Input/Output Demo - Digilent Reference).
Please, let me ask another question. I tried to rebuild the base overlay (from pynq to zybo) using only the video and led gpios, buttons and switches (4 switches on zybo). I am ysing Pynq v2.5 and the gpio is working fine, but the hdmi signal is not detected.
constraints.txt (2.9 KB)
I am enclosing the diagram block and xdc constraints.
I also got this critical warning:
[Timing 38-249] Generated clock base_i/video/hdmi_out/frontend/rgb2dvi_0/U0/SerialClk has no logical paths from master clock axi_dynclk_0_PXL_CLK_O.
Resolution: Review the path between the master clock and the generated clock with the schematic viewer and correct the -source option. If it is correct and the master clock does not have a timing path to the generated clock, define the generated clock as a primary clock by using create_clock.
Thank you very much for your support.
PS: this is not intended for commercial purpose, is for a project in university.
base.pdf (233.6 KB)
Yes, IO are a limiting factor.
They both have this chip: XC7Z020-1CLG400C
-1 indicates the speed grade. This is the slowest speed grade.
I may be wrong, but I don’t think your Zybo is meeting timing - you are effectively overclocking it.
If you search Digilent forums, you can see others posting issues about 1080p. A few are not answered, a few mentioned this issue.
If you look inside the RGB2DVI IP, you can see there is an option to set for 1080p or 720p. I would say it is set to 720p for your Zybo, and if you set to 1080p you will probably get a timing failure.
As mentioned, it may work on some boards, and not others.
For the constraints, I think the path you are referring to is between the dynamic clock generator and the RGB2DVI block. Both of these IP are from Digilent. The pdf for the rgb2dvi says the serial clock is 5x the pixel clock, and I can see there is a constraint for this. I think Vivado is saying it can’t find any logical paths between these clocks.
Maybe check all your wires are connected properly and that they aren’t being optimized away for some reason. You could also check in RGB2DVI if the option to generate the serial clock internally is set. If it is set I guess the serial clock would be ignored, and this warning may be triggered?
If you don’t see any problems, I would say you could ignore this warning. I’m not familiar enough with the workings of the RGB2DVI to know for certain.
Do you only see the HDMI signal not detected with your own design and do you see this for all resolutions, or do you see this with the PYNQ base design too?
Cathal
1 Like
Thank you. I have been using the zybo with 1080p in different devices (two monitors and one frame grabber) without problems. It may have something wrong with my design when I rebuild the base overlay from pynq to zybo. Even at 720p it is not working. After hdmi_in.start() and hdmi_out.start() the computer blinks but no video signal is detected.
Also, when I try to run hdmi_in.readframe(), this error is shown: RuntimeError: DMA channel not started.
Sorry for not replying this before. I missed this. I cannot not see any signal with this particular design on the zybo for any resolutions. I can see this working at 720 on the pynq-z2 and I can see my own designs on the zybo at 1080p.
Hi @cathalmccabe, I am trying to rebuild the video overlay as follows:
1.HDMI project demo (Zybo Z7 HDMI Input/Output Demo - Digilent Reference) works in standalone mode.
2. I export hwf and bit and when I load the overlay this is the output of
from pynq import Overlay
from pynq.lib.video import
base = Overlay(“/home/xilinx/pynq/overlays/video/video.bit”)
base?
Type: Overlay
String form: <pynq.overlay.Overlay object at 0xb3ab60b0>
File: /usr/local/lib/python3.6/dist-packages/pynq/overlay.py
Docstring:
Default documentation for overlay /home/xilinx/pynq/overlays/video2/video.bit. The following
attributes are available on this overlay:
IP Blocks
video/hdmi_in/axi_gpio_video : pynq.lib.axigpio.AxiGPIO
video/axi_vdma_0 : pynq.lib.video.dma.AxiVDMA
video/hdmi_in/v_tc_in : pynq.overlay.DefaultIP
video/hdmi_out/v_tc_out : pynq.overlay.DefaultIP
video/hdmi_out/axi_dynclk_0 : pynq.overlay.DefaultIP
Hierarchies
video : pynq.overlay.DefaultHierarchy
video/hdmi_in : pynq.overlay.DefaultHierarchy
video/hdmi_out : pynq.overlay.DefaultHierarchy
Interrupts
None
GPIO Outputs
None
Memories
processing_system7_0 : Memory
However, if I try to load the IP blocks, I have a problem as follows:
base.video.hdmi_in.axi_gpio_video?
Object base.video.hdmi_in.axi_gpio_video not found.
I can only load the axi_dynclk.
Could you please help? I don’t understand the issue.
I am enclosing a zip folder with the bitstream and hwh.
Thank you.
video.zip (437.8 KB)
The PYNQ video subsystem is described here:
https://pynq.readthedocs.io/en/v2.5.1/pynq_libraries/video.html
The PYNQ Python Video class corresponds to this design.
The ZYBO video demo design is different, and wouldn’t work in this way. It doesn’t include all the IP the PYNQ video system has, so you will get the errors you saw.
Cathal
1 Like
Hi @cathalmccabe, thank you.
I did the following tries:
- Rebuild video overlay from Pynq-Z2 overlay and tested on PYNQ-Z2. I adjusted RGB2DVI and DVI2RGB with PixelClk >= 120MHz and resolution 1080p. I also changed the xdc constraint line
create_clock -period 8.334 -waveform {0.000 4.167} [get_ports hdmi_in_clk_p]
to
create_clock -period 6.734 -waveform {0.000 4.167} [get_ports hdmi_in_clk_p]
Now PYNQ-Z2 can stream at 1080p. To check the cloks of PS, I did
from pynq.ps import Clocks
ps_clk = [Clocks.fclk0_mhz, Clocks.fclk1_mhz, Clocks.fclk2_mhz, Clocks.fclk3_mhz]
and the result was [100.0, 142.857143, 200.0, 100.0].
- I retargeted the video overlay to the zybo with the correct constraints like it was before and hdmi_in_hpd is recognized, computer blinks, but no video signal is displayed. When I checked the ps_clk the results now are different and seems to be the issue: [100.0, 150.0, 214.285714, 166.666667]
I don’t have a 200MHz to fed the video pipeline. Do you know what is causing to change this clock?
Thank you
OK, good you got this to work. Just to let you know, I expect your design is not meeting timing somewhere and you get a warning, but it may be “good enough” for what you need.
The clocks you mention are set in the PS.
They will be read from the .tcl or .hwh each time the overlay is loaded, and updated.
You can also be modified these after the design has been loaded.
In your PS settings, it looks like you have:
100, 150, 214…, 166…
Z2 has:
100, 142, 200, 100
You can change the PS settings in your design and rebuild. You could also just try change the clocks after you load your current design.
You can see how to set clocks here:
You can also modify the Tcl/HWH. Not great practice, as you should probably rebuild your design, but might be OK for testing.
Cathal
Hi @cathalmccabe,
on Vivado, my FCLKs are 100, 142, 200 and 100MHz (all IO PLL). I tried to change the clock, but I get this warning and the clocks does not change:
/usr/local/lib/python3.6/dist-packages/pynq/ps.py:312: UserWarning: Setting frequency to the closet possible value 150.0MHz.
round(freq_high / q0, 5)))
/usr/local/lib/python3.6/dist-packages/pynq/ps.py:312: UserWarning: Setting frequency to the closet possible value 214.28571MHz.
round(freq_high / q0, 5)))
In that way, you can probably use a clock wizard in your block design. Have that clock wizard connected to the input PS clock 0, having 100MHz. Configuring the output you will be able to have a 200MHz clock. And you only have to set PS clock 0 to 100MHz - no need to change other PS clocks, since clock wizard does that for you.
1 Like
Hi @rock. Thank you. I tried to put a clocking wizard outputting 200MHz from FCLK0 (100 MHz) and it did not work.
Actually my PS clocks right after boot are 150, 214.28, 300 and 150MHz. It is really odd that these values are approximately [100, 142.86, 200, 100]x1.5.
I don’t really understand what is going on.
Only when I force clocks to [100, 142.86, 200, 100] is that I could get [100.0, 150.0, 214.285714, 166.666667].
I checked that Zybo crystall oscillator is 33.3333MHz, whereas PYNQ-Z2 is 50MHz (1.5 times higher).
I am a bit frustrated for now, once I can’t understand what is going on.
Thank you for the support anyway.
You can adjust this line for your zybo board. Going forward, the best way is to allow users to change it.
The clocks after you boot up the board is calculated using the default value 50MHz (but in reality it is 33MHz for you). That is the reason why you see a 1.5x factor there.
Why doesn’t the clock wizard work for you? If you can generate a 100MHz clock then 200MHz should be possible.
1 Like
Hi @rock, thank you!.
After changing DEFAULT_SRC_CLK_MHZ to 33.333333, the values of PS clocks are [99.999999, 142.857141, 199.999998, 99.999999] MHz.
I am still getting no signal at HDMI display. I just retargeted to Zybo with the proper constraints.constraints.txt (2.9 KB).
FCLK0 is set to 100 MHz and clocking wizard to 200MHz and block design can be seen in this figure.
I am not sure if it is relevant, but hdmi_in_hpd and hdmi_out_hpd return 1 and 0, respectivelly.
PS: the same design was retargeted from PYNQ-Z2 (working at 1080p).
If you can already generate the 200MHz clock, you then don’t need the clock wizard. Just connect clk_200M to FCLK_CLK2?
Anything else that we need to know to help you? For now I have no idea.
1 Like