MIPI + Resizing HLS IP Integration Issue

Support
1

Config: PYNQ 3.0.1, Kria KV260, Ubuntu 22.01

Hi,

I have successfully integrated the resizer IP and MIPI blocks as separate hierarchies. I can read an image and then send it off to be resized via allocate. Now, I am trying to merge the resizer into the MIPI pipeline to get a NxN image out down from a 720x1280 image. I would ideally like to just call readframe() and see a resized image.

So far, I have inserted the resize_accel HLS IP into the MIPI hierarchy block and added its PYNQ IP driver. PYNQ finds the custom driver, sets the registers, and calls the MIPI start function. When I call mipi.readframe(), however, the design hangs.

image

Did I miss a step in trying to integrate a resizing IP in the capture pipeline? Would I need to modify the PYNQ/C source code for this to work? Maybe there’s another way to resize to an NxN image in the PL I may have missed. I suspect it may be an AXI-interface issue. I also would like to know how the pixel pack cpp files get called during runtime? Maybe this would help shine some light on how Vitis HLS IPs are configured during runtime.

This is my block diagram:

image
image1273×584 57.6 KB

PYNQ IPs:
image

Thanks!

References

[1] Access Acceleration kernel without DMA - #2 by haipnh
[2] Tutorial: AXI Master interfaces with HLS IP

Update:
After some digging, the pixel_pack.cpp file uses the axi streams for dataflow designs. How would I modify the xf_resize cpp file to use axis instead of m_axis?

1 Like
2

Hi,
With which board are you working? Did you locate the python file containing the readframe function?

1 Like
3

Hi,

I am working on the Kria KV260 and yes, I was able to find the pcam5c.py and pcam_mipi.c files. I suspect I may have to add another function to initialize the resizer hls ip in the C file but I have little experience on this front. The readframe functions pulls images out of the vdma from my understanding.

Regards,

1 Like
4

Hi @juanppalacios,

The MIPI is a hierarchy and there’s a hierarchy driver associated to IP Pcam5C. This hierarchy driver does not include the resize IP.

The __init__ method of the driver is not executed until you instantiate the IP. So, before reading, instantiate the resize.

Alternatively, you can create a new hierarchy driver to configure the resize IP. You can derive from the Pcam5C class.

So, you will have to:

  • Redefine checkhierarchy to check for the resize IP
  • Instantiate your Resizer driver in the __init__ method

Something like this:

from pynq.lib.video import Pcam5C

class MyPcam5C(Pcam5C):
    @staticmethod
    def checkhierarchy(description):
        return (
                   #your code goes here
        )

    def __init__(self, description):
        super().__init__(description)
        self.resize = # resize IP configuration

You can learn more about DefaultHierarchy drivers here.

https://pynq.readthedocs.io/en/latest/overlay_design_methodology/python_overlay_api.html#creating-hierarchy-drivers

I suggest you move the value assignments of your driver (Resizer) outside of the __init__ method to a different function.

Mario

1 Like
5

Hi,

Thanks for the response. I was able to tidy up the code and create a derived hierarchy. PYNQ now calls our custom hierarchy driver RPcam5C. The readframe call does not return a frame however.

image
image1845×878 205 KB

I’ve set the start and auto-start bits in my control register but it seems no frames are ever written to the VDMA. Going back to my block diagram, I feel like I may have missed something here. The src and dst ports in the xf_resize_accel_stream.cpp file set them as axi stream ports so I connected them to the csi rx ss and axis subset converter blocks, respectively.

void resize_accel(stream_t& src, stream_t& dst,
                  int src_rows, int src_cols,
                  int dst_rows, int dst_cols) {

    #pragma HLS INTERFACE axis register both port=src
    #pragma HLS INTERFACE axis register both port=dst
...

Might need a second pair of eyes on the actual diagram. I am considering moving the resizer accel after the VDMA.

image
image1446×829 82.3 KB

Again, thanks for the help/pointers.

Regards

1 Like
6

Hi @juanppalacios,

I see the issue. All these IP are configured by the shared library .so, so the image resolution does not match after you add the resize IP, this will lead to the read not to finish.

I would suggest you move the resize to the end of the pipeline so, you minimize number of changes in the other IP configuration.

Mario

2 Likes
7

Hi all,
as pointed by @marioruiz I connected a hand-made hls resizer at the end of the MIPI pipeline (before the pixel_pack IP) and it worked fine. I did not have to modify the mipi hiherarchy, I just used the PYNQ default driver to configure the resizer.
Best regards

2 Likes