How To Allocate PL DDR4 on RFSoC4x2 in PYNQ 3.0.1

[JennySmith888]

Hello,

I am trying to allocate the PL DDR4 on the RFSoC 4x2 in PYNQ 3.0.1. With a fresh SD-card, pynq.allocate targeting the overlay ddr4 returns a buffer outside the ddr address range. This was solved for the ZCU111 here. I am trying to do the same fix except this time I used @njachimi’s ddr4.dtbo from the RFSoC-MTS repo.

I think the issue is that in my custom overlay, I kept the default DDR4 base address of 0x500000000

In the 4x2 MTS overlay, I noticed the base address has been edited to 100000000 (with size 8G?) and so the .dtbo file is not right for my address space.

I would prefer to just edit the .dts and keep my address space as the default but I am wondering if I will have to change the address space instead? Is there a reason the default was changed? Why is it 8G when the DDR4 is only 4G?

Thanks,
Jenny

[njachimi]

Hi Jenny,
The RFSoC4x2 PS and PL DDR4 memory are both 8GB and same components. The RFSoC-MTS repo has the dts and associated makefile so you can modify the address range in a similar way. Note that the address and range are a pair of 32-bit hexadecimal values so when you modify them, partition the 64-bit address. Also, device tree overlays (dtbo) are finnicky with XRT. You need to stop XRT and then restart it before putting in a device tree overlay otherwise, XRT will quietly allocate you a buffer located in the PS memory space instead of the needed PL-DRAM region.
Also note when editing your Vivado overlay, you don’t need to use the entire 4GB if that is better for you. The address map will jump around due to the addressing needs.

Regards,

Nathan

[JennySmith888]

Thanks for your response @njachimi. I’m a little confused about a two things. First:

The RFSoC4x2 PS and PL DDR4 memory are both 8GB and same components.

They are both 4GB per the product page. Do you mean together the PS and PL DDR4 make up one 8 GB memory? I think in your MTS project the DDR4 range should be 4G not 8G and instead of reg = <0x10 0x00 0x02 0x00>; in your .dts it should be reg = <0x10 0x00 0x01 0x00>;. With your project, I am able to erroneously allocate 8 GB of PL DDR4. If I do two separate 4 GB buffers, they are the same as in if I write to one buffer, the changes are reflected in the other. Maybe there is a reason for this I’m not seeing.

The second thing is that in my project with a different DDR4 base address I have not been able to allocate the memory on the 4x2.

Details:
In my project the PL DDR4 is at 0x500000000 and so the register specification for the reserved memory node should read <0x05 0x00 0x01 0x00>. I recompiled the .dtbo with this new specification and added it to the device tree using the insert_dtbo.py. I appened python3 /usr/local/share/pynq-venv/pynq-dts/insert_dtbo.py to the pynq_venv.sh startup script which worked for me on the ZCU111. I think what I did should have worked but allocate is returning this error.

buff1 = pynq.allocate(int(1.5*1024**3), dtype=np.uint8, target=ol.capture.ddr4_0)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 buff1 = pynq.allocate(int(1.5*1024**3),'u8',ol.capture.ddr4_0)

File /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/buffer.py:158, in allocate(shape, dtype, target, **kwargs)
    156 if target is None:
    157     target = Device.active_device
--> 158 return target.allocate(shape, dtype, **kwargs)

File /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/pl_server/xrt_device.py:156, in XrtMemory.allocate(self, shape, dtype, **kwargs)
    145 def allocate(self, shape, dtype, **kwargs):
    146     """Create a new  buffer in the memory bank
    147 
    148     Parameters
   (...)
    154 
    155     """
--> 156     buf = _xrt_allocate(shape, dtype, self.device, self.idx, **kwargs)
    157     buf.memory = self
    158     return buf

File /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/pl_server/xrt_device.py:104, in _xrt_allocate(shape, dtype, device, memidx, cacheable, pointer, cache)
    102     bo, buf, device_address = pointer
    103 else:
--> 104     bo = device.allocate_bo(size, memidx, cacheable)
    105     buf = device.map_bo(bo)
    106     device_address = device.get_device_address(bo)

File /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/pl_server/xrt_device.py:410, in XrtDevice.allocate_bo(self, size, idx, cacheable)
    408 bo = xrt.xclAllocBO(self.handle, size, xrt.xclBOKind.XCL_BO_DEVICE_RAM, idx)
    409 if bo >= 0x80000000:
--> 410     raise RuntimeError("Allocate failed: " + str(bo))
    411 return bo

RuntimeError: Allocate failed: 4294967295

I checked the device tree using dtc -I fs /sys/firmware/devicetree/base and sure enough I see my memory node in there:

reserved-memory {
		ranges;

		reserved {
			reg = <0x05 0x00 0x01 0x00>;
		};
	};

The address returned in the allocate error message is 0xffffffff . I’m not totally sure what to make of that. Please let me know if you spot any issues with my approach.

P.S.
I did try to just change my DDR4 base address to 0x1000000000 to match yours but the address editor was giving me errors related to “memory aperture”. I think it’s not compatible with the rest of the IP I have in my design on the same network?

[cathalmccabe]

Hi Jenny,
DRAM is 4GB PS + 4GB PL as you see on product page.

I think the returned value you see 0xffffffff just means “error” (or -1).

As a quick check, could you try allocate a much smaller buffer on PL DRAM to see if it works at all?

Cathal

[JennySmith888]

Hi Cathal,

Yes I can allocate 120 MiB but no more and it’s not at the right address.

Here’s a test:

n = int(120*1024**2)
try:
    x=pynq.allocate(n,dtype=np.uint8,target=ol.capture.ddr4_0)
    x.freebuffer()
    print(f'Success @ {n/1024**2} MiB')
except Exception as e:
    print(f'Fail @ {n/1024**2} MiB: {e}')
try:
    x=pynq.allocate(n+1,dtype=np.uint8,target=ol.capture.ddr4_0)
    x.freebuffer()
    print(f'Success @ {n/1024**2} MiB + 1B')
except Exception as e:
    print(f'Fail @ {n/1024**2} MiB + 1B: {e}')

The result:

Success @ 120.0 MiB
Fail @ 120.0 MiB + 1B: Allocate failed: 4294967295

When I allocate the 120 MiB or smaller buffer, the device address is 0x78400000 which is in the PS RAM I believe.

[Idos]

Hey Jenny,

I have the same error right now. I did not find a solution yet.
I am trying to allocate less(63MiB), and preferably from the PS DDR.

@cathalmccabe you are right it returns -1 from XRT

I am following the answers here.

thanks

[marioruiz]

Hi @JennySmith888,

Do you see any error message from zocl in dmesg when you try to allocate PL memory?

Mario

[Idos]

Hey Mario @marioruiz

I am not trying to allocate the PL-DDR, however, it seems to fall back on the contiguous memory allocator, nevertheless, xrt.xclAllocBO returns -1, which means it is trying to allocate buffers without success, It derives that the problem is also with the CMA .

I would try and update this dtbo, or even create a new one. The second question which raises is why the CMA fails. am I in the right direction?

Thanks, everyone.

[JennySmith888]

@marioruiz I’m seeing the same thing as @Idos using dmesg.

[344223.163654] fpga_manager fpga0: writing gen3_top.bin to Xilinx ZynqMP FPGA Manager
[344223.906592] [drm] skip kind 29(AIE_RESOURCES) return code: -22
[344223.906622] [drm] skip kind 8(IP_LAYOUT) return code: -22
[344223.906628] [drm] skip kind 9(DEBUG_IP_LAYOUT) return code: -22
[344223.906632] [drm] skip kind 25(AIE_METADATA) return code: -22
[344223.906637] [drm] skip kind 7(CONNECTIVITY) return code: -22
[344223.906641] [drm] found kind 6(MEM_TOPOLOGY)
[344223.906680] [drm] Memory 0 is not reserved in device tree. Will allocate memory from CMA
[344223.906715] [drm] zocl_xclbin_read_axlf fe53eb73-5250-f8e3-235e-bf097b655c1f ret: 0
[344223.916146] [drm] bitstream fe53eb73-5250-f8e3-235e-bf097b655c1f locked, ref=1
[344223.916189] zocl-drm axi:zyxclmm_drm:  ffff00080081fc10 kds_add_context: Client pid(36272) add context Domain(0) CU(0xffffffff) shared(true)
[344223.916253] zocl-drm axi:zyxclmm_drm:  ffff00080081fc10 kds_del_context: Client pid(36272) del context Domain(0) CU(0xffffffff)
[344223.916273] [drm] bitstream fe53eb73-5250-f8e3-235e-bf097b655c1f unlocked, ref=0
[344229.946488] [drm] Memory allocated from CMA region whereas requested for reserved memory region
[344229.946504] [drm:zocl_create_bo [zocl]] *ERROR* Fail to allocate BO: size 125833216

I’m assuming Memory 0 is supposed to be the PL DDR4. I’m not sure why it says it’s not reserved in the device tree as I recompiled the .dtbo with the correct address/size and I see what appears to be the correct reserved memory in the device tree using dtc -I fs /sys/firmware/devicetree/base

reserved-memory {
		ranges;

		reserved {
			reg = <0x05 0x00 0x01 0x00>;
		};
	};

Here’s my full .dts file:

/dts-v1/;
/plugin/;
/ {
        /*reserved memory*/
        fragment@4 {
                target-path = "/";
                overlay4: __overlay__ {
                        reserved-memory {
                                ranges;
                                reserved {
                                        reg = <0x05 0x00 0x01 0x00>;
                                };
                        };
                };
        };
};

[JennySmith888]

Ok we found a solution which was to just rebuild the image using an edited boards/RFSoC4x2/petalinux_bsp/meta-user/recipes-bsp/device-tree/files/system-user.dtsi where we added the proper PL DDR4 reserved memory node for our address assignments from the get-go. I bet following the exact procedure in the MTS Repo would have worked too, I just wanted it to be more automatic.