AUP-ZU3 Pcam 5C read frame stuck

Support
1

Board: AUP-ZU3 (8GB)

PYNQ Version: 3.1

Issue:

readframe() hangs indefinitely when trying to capture frames from a Pcam 5C (OV5640) camera using the base overlay’s MIPI pipeline. The call never returns and no frames are captured.

Steps to Reproduce:

Start with a clean PYNQ image for the ZU3 board.

Boot the board and connect to the Jupyter notebook through your web browser.

In the notebook, navigate to /base/video/mipi_to_displayport.ipynb

Run Cells 1-4. The 4th cell runs indefinitely without returning anything.

frame = mipi.readframe()
PIL.Image.fromarray(frame[:,:,[2,1,0]])

Debugging Steps Taken:

The issue appears to be similar to the one presented here, but unlike this issue, I was able to see camera responding whether I ran mipi = base.mipi or not.

Output from running i2cdetect -y 3:

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --

From this, I gather that the camera is powered on and there is some communication between it and the board, but I haven’t been able to stream any frames.

I have also been looking through the post found here. However, the fixes appear to be specific to the KV260 board.

What are some steps I can take to more concretely find the source of the issue? I haven’t found similar issued reported specific to the AUP-ZU3 board, has anyone else seen this type of issue?

2

Hi @Preston-Walker,

Welcome to the PYNQ community.

We have found that those cameras are very fragile. The I2C has a different path than the data.
So, if possible can you try a different ribbon cable or camera?

The MIPI subsystem in the Kria is the same as in the AUP-ZU3, so you can try the fix proposed in the other blog.

Mario

3

Thanks for the quick response!

The camera is brand new, so I would hope it isn’t already broken. I tried swapping the ribbon cable and didn’t see any differences. I don’t have another camera available, have you found them to be so fragile that I should consider a replacement even with this one coming right out of the box?

As far as the solution from the Kria blog, I see this reply as the “solution“ with two jupyter notebooks and the bit/hwh files. I assume that the bit and hwh files are specific to the board and will need to be rebuilt, while the ipynb files should be fine on the AUP-ZU3. Does that sound correct?

Preston

4

The camera is brand new, so I would hope it isn’t already broken. I tried swapping the ribbon cable and didn’t see any differences. I don’t have another camera available, have you found them to be so fragile that I should consider a replacement even with this one coming right out of the box?

I haven’t found them very reliable, or may be an signal integrity issue. Never got to the bottom of it.

As far as the solution from the Kria blog, I see this reply as the “solution“ with two jupyter notebooks and the bit/hwh files. I assume that the bit and hwh files are specific to the board and will need to be rebuilt, while the ipynb files should be fine on the AUP-ZU3. Does that sound correct?

Correct.

5

Thank you for the response Mario! Sorry for the delay on my end, there were some changes in our lab, and I have only been helping from a distance while my colleague,@Victor-Santana , continues this project. Victor was having issues with his account, so I will post this message on his behalf:

Hi Mario,

Jumping into the thread here, my name is Victor Santana, and I am an undergraduate research assistant working alongside Preston on this AUP-ZU3 and Pcam 5C project.

Following up on your last message to Preston regarding rebuilding the Vivado files, our team has been working hard to implement the KV260 workaround, but we are still hitting a wall. We wanted to share a detailed update on the debugging steps we’ve taken to see if your team might have any further insights.

As a quick refresher on our environment:

  • Board: RealDigital AUP-ZU3 (8GB DDR4 variant)
  • PYNQ Version: 3.1
  • Tool Version: Vivado/Vitis 2024.1 (also tested on custom images built with PetaLinux)

The Issue: Even with brand new ribbon cables, the I2C control communication connects perfectly, but the MIPI video data path hangs indefinitely. The Python execution gets completely stuck at the frame = mipi.readframe() command. It never throws a timeout error, no frames are captured, and the MIPI CSI-2 Rx Subsystem registers indicate 0 packets received.

Debug steps we have taken so far: To rule out as many user errors and hardware variables as possible, we have tried the following:

  • Physical Hardware Checks: As Preston mentioned, we initially suspected the fragile 15-pin FFC ribbon cable. We have now tested this setup using brand new cables across two completely different AUP-ZU3 boards and two different cameras. The exact same hang happens on all hardware combinations, so we are confident it is not an isolated microscopic cable fracture.
  • I2C Verification: We ran !i2cdetect -y 3 on all setups, and it correctly returns 0x3c. From what we understand, this confirms that the OV5640 sensor is receiving power and its digital core is responding perfectly to the I2C control plane.
  • Custom Vivado Overlays: Per your advice on the KV260 fix, we tried bypassing the prebuilt image by building our own custom PYNQ images using the instructions found in the Xilinx/AUP-ZU3 GitHub repo. We patched the pipelines to the very best of our abilities, generated the new bitstream, uploaded the .bit and .hwh files directly to our Jupyter environment, and ran the base mipi_to_displayport.ipynb notebook. Unfortunately, it still hangs.
  • Manual GPIO Power-Up: We tried manually toggling the camera’s power-up pin (CAM0_PWUP) using PYNQ MMIO (at offset 0x08 for GPIO Channel 2) before running the standard Python initialization. We read that this can help ensure the sensor’s analog core is fully awake before sending the massive array of I2C configuration commands.
  • Hardware Probing (ILA): To definitively check if any video data is making it into the FPGA, we embedded a Vivado Integrated Logic Analyzer (ILA) onto the video_out_tvalid pin of the mipi_csi2_rx_subsyst block. When we run the Python script, the ILA stays stuck on “Waiting for Trigger.” We even tested the ILA on a board push-button to verify our trigger logic was correct. This proves that the MIPI Receiver is truly outputting zero digital packets into the fabric.

Given that the hardware is new and the ILA confirms the packets are dropping before they enter the Vivado pipeline, are there any specific MIPI D-PHY analog timing parameters (like HS_SETTLE_NS) that we need to manually adjust when building the AUP-ZU3 bitstream?

We really appreciate your patience and help in guiding us through this! We are happy to provide our custom .bit / .hwh files, Jupyter Notebooks, or any other logs that might help clarify things.

6

This is indeed very strange. Would you have other hardware to try the camera?

I am trying to work in a full Python driver that may make the debugging easier.
Can you let me know what is the PCAM5C revision?

7

We have two cameras and two AUP-ZU3 boards. We have tried all permutations of those and gotten the same results. We don’t have any other mipi-compatible boards to try. Our camera revision is Rev C.0.

8

You can try the experimental pure Python support. To do this, please follow these steps

On the board, from JupyterLab open a new terminal.

cd /tmp && git clone https://github.com/mariodruiz/PYNQ.git -b pcam-python
cd /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/lib
mv video bk.video
cp -r /tmp/PYNQ/pynq/lib/video .

This will replace the local video drivers with the experimental ones. After this, try to re-run the notebook and report, you will see that the class has a .diagnostics() method that you can run if things don’t work well.

One thing I noticed from your report is that you are using bus 3 for the I2C. Can you verify this by checking this variable?

Please, report your findings.

9

Hi Mario,

Thank you for the experimental branch. Because our board didn’t have native internet routing through our Linux host, we temporarily used a USB-C to Ethernet adapter to plug the board directly into our router. This worked perfectly to clone your pcam-python branch and swap out the backend drivers.

Because the standard readframe() call hangs the kernel indefinitely, we had to force-interrupt the kernel and run your .diagnostics() method post-hang.

image
image1466×927 88.3 KB

Here are the results:

{'packet_count': 0,

 'stream_line_buffer_full': False,

 'short_packet_fifo_not_empty': False,

 'short_packet_fifo_full': False,

 'active_lanes': 2,

 'maximum_lanes': 2}

And for your I2C question, your driver successfully found it automatically: Driver found the camera on I2C Bus: 3

Because I2C is communicating perfectly and the MIPI RX subsystem is receiving exactly 0 packets, what can you tell us about these results?

Thanks again for your help.

Best regards,

10

Hi @Preston-Walker,

Thanks for running the diagnostics. packet_count: 0 with the short packet FIFO also empty tells me nothing is reaching the CSI-2 controller at all, so the issue is upstream at the D-PHY, not the VDMA. I have pushed some

updates to the branch that read the D-PHY status directly, which should tell us more. Please, use the steps I described before to get the latest changes.

A few things to try

  1. Are you using the stock base overlay? If not, pelase use it

  2. Try the sensor test pattern. Before readframe, enable the built-in color bars:

    mipi.test_pattern(True)

    This is generated inside the sensor and streamed over MIPI on its own, so it isolates the sensor imaging config from the MIPI path. If the bars stream, the transport is fine; if not, the problem is in the D-PHY/MIPI.

  3. Dump the diagnostics again. Whether or not the test pattern works, please paste the full output of:

    mipi.diagnostics()

    The new fields (dphy_cl_init_done, dphy_cl_stop_state, dphy_dl0_pkt_count, …) will show whether the D-PHY is initializing and whether the clock lane ever leaves the stop state.

Once I see this I can tell you the next step. Depending on what the D-PHY reports, we may need to check the continuous-clock setting on the sensor or sweep the HS_SETTLE timing, but let’s see the numbers first.

Mario

11

Hi Mario,

We followed your instructions carefully to test the boards.

First, we noticed there are two different prebuilt images online for the 8G AUP-ZU3 board. One is from RealDigital, and the other is from AMD PYNQ. We compared the two files using the diff command and vbindiff, and we saw that they are different (they have different hex values).

Because they are different, we decided to run your suggested test on both images.

1. RealDigital Image Test

(Downloaded from: AUP-ZU3 Board | Real Digital )

- We ran the mipi_to_displayport.ipynb notebook.

- We ran mipi.test_pattern(True) right before the readframe step.

- When the program got stuck, we interrupted the kernel and ran mipi.diagnostics().

Here are the results:

{‘packet_count’: 0,

‘stream_line_buffer_full’: False,

‘short_packet_fifo_not_empty’: False,

‘short_packet_fifo_full’: False,

‘active_lanes’: 2,

‘maximum_lanes’: 2,

‘dphy_enabled’: True,

‘dphy_cl_init_done’: False,

‘dphy_cl_stop_state’: False,

‘dphy_cl_mode’: 0,

‘dphy_dl0_init_done’: False,

‘dphy_dl0_stop_state’: False,

‘dphy_dl0_pkt_count’: 0,

‘dphy_dl1_init_done’: False,

‘dphy_dl1_stop_state’: False,

‘dphy_dl1_pkt_count’: 0}

2. AMD PYNQ Image Test

(Downloaded from: PYNQ supported boards and PYNQ pre-built images | PYNQ )

- We followed the exact same steps as above.

Here are the results:

AMD PYNQ Image

{‘packet_count’: 0,

‘stream_line_buffer_full’: False,

‘short_packet_fifo_not_empty’: False,

‘short_packet_fifo_full’: False,

‘active_lanes’: 2,

‘maximum_lanes’: 2,

‘dphy_enabled’: True,

‘dphy_cl_init_done’: False,

‘dphy_cl_stop_state’: False,

‘dphy_cl_mode’: 0,

‘dphy_dl0_init_done’: False,

‘dphy_dl0_stop_state’: False,

‘dphy_dl0_pkt_count’: 0,

‘dphy_dl1_init_done’: False,

‘dphy_dl1_stop_state’: False,

‘dphy_dl1_pkt_count’: 0}

Hardware Notes & Camera Setup:

- We used two different 32GB MicroSD cards so each image had its own card to boot from.

- I have attached a picture of our camera setup so you can see how everything is plugged in.

image
image1775×2367 815 KB

Did we follow your steps correctly? Please take a look at the attached picture and let us know what you think.

Thank you again for your patience and for all your help. We look forward to hearing about the next steps.

12

This is really useful, thanks for testing both images. Good to know they behave the same.

The important new field is dphy_cl_init_done: False (and the same on both data lanes). This means the D-PHY link itself is not coming up. The clock lane is also not in stop state dphy_cl_stop_state: False, so the receiver is not even detecting the LP-11 idle from the sensor’s clock lane.

That points to one of two things: either the sensor is not actually driving its MIPI clock lane, or that clock lane is not reaching the receiver. Since I2C works but the clock lane does not, this is exactly the fragile part
I mentioned earlier.

Can you check that the sensor really is streaming? After the setup cells (with the pipeline started), please run and paste the output:

hex(mipi._sensor.read_reg(0x3008))   # expect 0x02 (streaming, not 0x42 standby)
hex(mipi._sensor.read_reg(0x300e))   # MIPI control, expect 0x45 (2-lane MIPI)
hex(mipi._sensor.read_reg(0x4800))   # bit5 should be 0 -> continuous clock
hex(mipi._sensor.read_reg(0x300a)), hex(mipi._sensor.read_reg(0x300b))  # ID 0x56, 0x40

If register 0x3008 does not read back as 0x02, the sensor never actually started and we chase that. If it reads 0x02 but the D-PHY still shows init_done False, then the sensor thinks it is streaming but the clock lane is not getting
through — which is either the sensor MIPI output or the physical clock-lane connection.

13

Hi Mario,

Thank you so much for the help.

This makes total sense, and we appreciate you explaining in detail what this means. Since both PYNQ images behave the same, we decided to run this latest test only on the image available on AMD’s PYNQ website. Here are the values we got:

-– STREAMING CHECK —

0x2

0x45

0x14

0x56

0x40

Here is a picture of our notebook:

image
image1565×932 97.9 KB

We ran that block of code right after this:

videomode = VideoMode(1280, 720, 24)

mipi.configure(videomode)

mipi.start()

Since you mentioned the check must be run immediately after the pipeline starts, we believe this is what you meant. Please let us know your thoughts.

Thanks again.

14

Great, it confirms the sensor side is completely healthy, but the D-PHY still sees nothing on the lanes.

One quick thing to try — start the sensor before re-initialising the D-PHY, so the D-PHY runs its lane init while the sensor is already driving the lines. Do not call readframe yet — just check the diagnostics first, otherwise it will hang again.

import time
mipi._sensor.start()
time.sleep(0.1)
mipi.mipi_csi2_rx_subsyst.configure(active_lanes=2)
time.sleep(0.1)
mipi.diagnostics()

Look at dphy_cl_init_done in the output:

  • If it is now True, the reorder worked — go ahead and try frame = mipi.readframe() and let me know if you get an image.
  • If it is still False, do not run readframe (it will just hang). The clock lane is not arriving cleanly at the receiver.
15

Hi Mario,

We ran that block of code right before this block of code:

videomode = VideoMode(1280, 720, 24)

mipi.configure(videomode)

Mipi.start()

Here are the results:

{
  'packet_count': 0,
  'stream_line_buffer_full': False,
  'short_packet_fifo_not_empty': False,
  'short_packet_fifo_full': False,
  'active_lanes': 2,
  'maximum_lanes': 2,
  'dphy_enabled': True,
  'dphy_cl_init_done': False,
  'dphy_cl_stop_state': False,
  'dphy_cl_mode': 0,
  'dphy_dl0_init_done': False,
  'dphy_dl0_stop_state': False,
  'dphy_dl0_pkt_count': 0,
  'dphy_dl1_init_done': False,
  'dphy_dl1_stop_state': False,
  'dphy_dl1_pkt_count': 0
}

Here is an image of our work:

image
image1565×932 111 KB

Please let us know if we ran this code correctly.

Thanks again for the help.

16

Hi @Preston-Walker,

This issue is a bit cryptic as I am not able to reproduce locally.

Can I ask you to share high quality pictures of the board (both sides) and the PCAM5C in case I am missing something?

17

Mario, here are the requested images. Please let me know if you need anything else.

AUP-ZU3_&_Pcam-5C_Images.zip (5.8 MB)

18

Hi @Preston-Walker,

I have the same batch of board and camera as you, however, I cannot reproduce locally.

When you have time, could you pull the latest changes in my branch and try again?
I increased the delays in order to give the oscillator extra time to settle.

Best,
Mario