The result is always the same: a hard cap around 200 Mbps.
Background: Before switching to PYNQ, I spent a significant amount of time debugging this on a standard PetaLinux build. I tried various configurations but could never break past this ~250 Mbps bottleneck. I documented those attempts in my thread on the AMD Forums here.
My Question to the PYNQ Team: Is this low PS Ethernet throughput a known issue or limitation with the current ZynqMP PYNQ images?
Have you observed this ~200 Mbps cap in your own internal testing?
Is it possible that I am the only one facing this, or is there a standard configuration step I am missing to unlock full Gigabit speeds?
We ran into this issue before; I believe the cause was the LPD domain running faster than the FPD domain. It looked like the BSP clocking config was copied from the 208 but because of the different DRAM clock TOPSW_MAINs the FPD clock domain was clocked significantly slower. When I fixed this inversion I was able to saturate a Gb link
After searching through the AMD forums I found the this post which reports a very similar issue and thankfully also a fix:
The primary issue was related to a little paid attention to clock setting in Vivado. The “TIMESTAMP” clock in the clock output sections of Vivado influences the CPU scheduler in Linux. When changing it from 33.333 MHz to 100 MHz, most of my bandwidth problems were solved.
The 33.333 MHz TIMESTAMP clock settings allegedly responsible for the slowness is consistent with the settings in HW design used to generate the RFSoC4x2 BSP. When searching github for the corresponding TCL setting name, one finds many projects with the 100 MHz timestamp setting, such as the one from strath-dsr:
I recently got access to the linked slack thread, and the main solution stated there was to change the source of the TOPSW_MAIN clock source from DPLL to APLL which results in a favorable actual frequency due to the division ratios (changes from ~400 MHz to 533 MHz). Here is a before (left) after (right) comparison of my changes:
I tried changing a few of these clocks and wasn’t able to get any better performance from the 1GbE. I tried TOPSW_MAIN_CTRL->VPLL then APLL, and then tried changing TIMESTAMP_REF_CTRL. I saw no difference between that and the stock v3.1.3 image.
board->host ~200-250 Mbps
host->board ~900 Mbps with around 50% UDP loss (i.e. 450 Mbps actual throughput)
I had a FSBL build with this bootloader config and was able to get ~linerate both ways; I am off work for the next week or so but I can try and replicate when I’m back
After a bit more trial and error, I managed to get this issue fixed.
I changed the following clock source settings in Vivado:
Setting name
Old source
New source
TOPSW_MAIN_CTRL
DPLL
VPLL
ADMA_REF_CTRL
RPLL
IOPLL
IOU_SWITCH_CTRL
RPLL
IOPLL
LPD_SWITCH_CTRL
RPLL
IOPLL
TIMESTAMP_REF_CTRL
PSS_REF_CLK
IOPLL
With these new settings, I created a new BOOT.BIN and saw the following speedup:
Direction
Before
After
Board to host
~220 Mbps
~950 Mbps, 0% loss
Host to board
~430 Mbps, ~50% packet loss
~950 Mbps, 0% loss
This was tested using iperf3, with the board as the server (i.e. iperf3 -s) and the host running the following command: iperf3 -c <board-ip> -u -b 1000M -t 10.
I aim to get these new settings into the next RFSoC4x2 image.
Hi @joshgoldsmith , thanks for tracking this down. Would you be able to share your working BOOT.BIN, or the patched XSA it was built from? I’m hitting the same ~200 Mbit/s TX cap on an RFSoC4x2 running PYNQ 3.1.1. I tried rebuilding BOOT.BIN with the corrected clocks myself and the board wouldn’t boot with it (recovered from backup, no harm done), so I’d rather validate with a known-good image. Happy to test it on my board and report results back here. Thank you.
Agree that it’s really great that this is fixed. Any idea when the next image will be released? If it’s still a ways out, I too would appreciate a copy if you’re willing to send it (my last build failed to boot as well).
Not sure if it’s related, but any idea of these settings would be related to slow USB performance? I was able to modify the device tree to fix USB 3.0, but using a USB SSD I’m only getting about 30 MB/s write and 105 MB/s read (confirmed with lsusb that it’s connected at 5000M). The same drive running the same benchmark on a PC connected at 5000M gets 353 MB/s write, 446 MB/s read.
I don’t want to sidetrack this thread, so I’d be glad to start a new one if this is unrelated - just figured I’d check if it’s plausible that the same clock would affect both Ethernet and USB.
@Ameen you can see the changes I made to the design in my PR. I’m not able to share any binaries with you at the moment but, to answer your question @DogP, we will be releasing a new version of PYNQ in the coming weeks, and all these changes will be in the new images.
@DogP please create a new forum post or github issue about the USB 3 speeds you are seeing
We could also reproduce the solution. We rebuilt the BSP with the settings above, and built a new image from that. With the new image we could saturate the GbE link with iperf3.
Attached you will find the corresponding BOOT.BIN, which you can just copy to your boot folder for a quick fix. The corresponding pynq/image version is 3.0.1.
Thanks to everyone for the quick replies and collaboration.
Thanks for posting an updated BOOT.BIN - seems to work for me, and also looks to fix the USB speed issue I was seeing.
One question though… does your OLED display still work? Mine doesn’t show anything, and dmesg shows these messages when (I think) the board tries to communicate with the display, and possibly also other SPI devices.
spidev spi1.0: SPI transfer timed out
spi_master spi1: failed to transfer one message from queue
Not sure if it’s just me, something with your specific BOOT.BIN build, or a problem related to the actual fix.
