# PYNQ3.0.1 allocate ddr4 returns buffer outside of address range **URL:** \ **Category:** Support\ **Created:** [November 14, 2022, 10:30pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918 "2022-11-14T22:30:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![JennySmith888](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/jennysmith888/32/1966_2.png) [@JennySmith888](https://discuss.pynq.io/u/JennySmith888)\ **Post date:** [November 14, 2022, 10:30pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/1 "2022-11-14T22:30:53Z")
Hello, I just upgraded my ZCU111 from PYNQ2.7 to PYNQ3.0.1. I’m using the same bitstream + hwh built with Vivado 2022.1 and I’m noticing some weird behavior with `pynq.allocate()`. I am attempting to allocate a buffer targeting the PL DDR4: ```auto >> buffer = pynq.allocate((n, 2), dtype='i2', target=ol.capture.ddr4_0) >> print(hex(buffer.device_address)) 0x78800000 >>print(hex(ol.capture.ddr4_0.base_address)) 0x500000000 ``` `pynq.allocate()` is returning a buffer below the base address of the PL DDR4? This causes AXI decoding errors when I try to capture data… In PYNQ2.7 everything seems to behave: the first time I call `pynq.allocate()` it returns a buffer with the same base address as the DDR4 and my capture works just fine. It looks like PYNQ3.0.1 correctly identifies the MIG and its base address. ```auto >> ol.capture.ddr4_0? Type: EmbeddedXrtMemory String form: File: /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/pl_server/embedded_device.py Docstring: Class docstring: Class representing a memory bank in a card Memory banks can be both external DDR banks and internal buffers. XrtMemory instances for the same bank are interchangeable and can be compared and used as dictionary keys. ``` I’m not sure what the root cause is. Test `.bit` and `.hwh` attached. [test.zip](https://discuss.pynq.io/uploads/short-url/eKvaYXdVZEupf7f2vVwvNAmHtZx.zip) (6.1 MB) ---
**Author:** ![skalade](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/skalade/32/686_2.png) [@skalade](https://discuss.pynq.io/u/skalade)\ **Post date:** [November 14, 2022, 11:12pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/2 "2022-11-14T23:12:56Z")
Hi there, This is a recent issue due to latest XRT changes [in the way allocation works](https://github.com/Xilinx/XRT/blob/202210.2.13.479/src/runtime_src/core/edge/drm/zocl/zocl_bo.c#L1187) – it expects memory to show up in the device tree. If you look at dmesg output while your overlay loads you might see some `Memory 0 is not reserved in device tree. Will allocate memory from CMA` messages. We don’t quite have a clean, automated fix for this use case in the current release. A workaround you could try applying is to add a device tree overlay with a [reserved-memory node](https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841683/Linux+Reserved+Memory) for your ddr4 memory. There’s an example of inserting a dtbo for pynq in the [kria-pynq repo](https://github.com/Xilinx/Kria-PYNQ/tree/main/dts), you’d have to do something similar with your reserved-memory dtbo. Another workaround is to just use MMIO to write to that memory. Apologies for not having a cleaner solution. Hope this helps. Thanks Shawn ---
**Author:** ![JennySmith888](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/jennysmith888/32/1966_2.png) [@JennySmith888](https://discuss.pynq.io/u/JennySmith888)\ **Post date:** [November 15, 2022, 12:36am UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/3 "2022-11-15T00:36:32Z")
Hi Shawn, Thanks for the quick reply. That is an unfortunate change in XRT. I’d prefer not to use MMIO because in my experience it’s very sensitive to memory access alignment and I’d rather not have to tip toe around that. XRT allocation worked well in PYNQv2.7 and is a core part of our system so I may just revert my board and hope it’s fixed in the next release. I’d be willing to try to develop a reserved-memory node solution but I may need a few more pointers. If I’m understanding your links right, I have to create a `reserve_ddr4.dts` file which will look something like ```auto reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; reserved: buffer@0 { no-map; reg = <0x0 0x70000000 0x0 0x10000000>; }; }; reserved-driver@0 { compatible = "xlnx,reserved-memory"; memory-region = <&reserved>; }; ``` I’m not really clear on what the `.dts` should actually look like. The link you provided also mentioned a `.dtsi` and a device driver. Are those relevent? Is there a tool I can use to generate the `.dts` or do you have any other resources you can point me to on how to construct that file for a single 4 GiB DDR4 with base address `0x500000000`? Assuming I get that right, I would then want to run a script like this: ```auto import os import warnings """ Insert DeviceTree Fragment using pynq If the segment is not inserted already import pynq and create the a DeviceTreeSegment with the full path to the \'pynq.dtbo\' file. Then insert the fragment """ path = os.path.dirname( __file__ ) dtbo = 'reserve_ddr4.dtbo' sysfs_dir = '/sys/kernel/config/device-tree/overlays/' + \ os.path.splitext(dtbo)[0] if not os.path.exists(sysfs_dir): try: from pynq import DeviceTreeSegment except UserWarning: pass if path != '': path = path + '/' db = DeviceTreeSegment(path + dtbo) db.insert() ``` whenever the pynq virtual environment is sourced? Please let me know if there are any major steps I missed and if possible include some more resources on the `.dts` file generation. Thanks for your help with this! Best, Jenny ---
**Author:** ![skalade](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/skalade/32/686_2.png) [@skalade](https://discuss.pynq.io/u/skalade)\ **Post date:** [November 15, 2022, 11:11am UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/4 "2022-11-15T11:11:35Z")
Hi Jenny, The simplest way to do is to just copy what’s done in the Kria-PYNQ repo [Kria-PYNQ/pynq.dts at main · Xilinx/Kria-PYNQ · GitHub](https://github.com/Xilinx/Kria-PYNQ/blob/main/dts/pynq.dts) and replace those fragments with your own reserved-memory one. You might not need the driver stuff in there, could add something like this in pynq.dts: ```auto fragment@4 { target-path = "/"; overlay4: __overlay__ { reserved-memory { ranges; reserved { reg = <0x0 0x70000000 0x0 0x10000000>; }; }; }; }; ``` And just delete the other fragments. You can re-use the makefile from kria-pynq as well to compile the dtbo (will probably need to run `apt-get install device-tree-compiler` first). Thanks Shawn ---
**Author:** ![JennySmith888](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/jennysmith888/32/1966_2.png) [@JennySmith888](https://discuss.pynq.io/u/JennySmith888)\ **Post date:** [November 15, 2022, 7:19pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/5 "2022-11-15T19:19:35Z")
Thank you for clarifying. Here’s my latest attempt: I cloned the [Kria repo](https://github.com/Xilinx/Kria-PYNQ/blob/main/dts/pynq.dts) to my board and edited the `pynq.dts` as you suggested: ```auto /* * Add zocl fragment to the device tree */ /dts-v1/; /plugin/; / { /*afi*/ /*TODO is afi necessary?*/ fragment@4 { target-path = "/"; overlay4: __overlay__ { reserved-memory { ranges; reserved { reg = <0x0 0x70000000 0x0 0x10000000>; }; }; }; }; }; ``` I installed `device-tree-compiler` and used the Kria `Makefile` to generate `pynq.dtbo`. I then tried the `insert_dtbo.py` script in a jupyter notebook ```auto import sys import os path = '/home/xilinx/Kria-PYNQ/dts/' dtbo = 'pynq.dtbo' sysfs_dir = '/sys/kernel/config/device-tree/overlays/' + \ os.path.splitext(dtbo)[0] if not os.path.exists(sysfs_dir): try: from pynq import DeviceTreeSegment except UserWarning: pass if path != '': path = path + '/' db = DeviceTreeSegment(path + dtbo) db.insert() ``` But it errored out with ```auto --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Input In [6], in () 13 path = path + '/' 15 db = DeviceTreeSegment(path + dtbo) ---> 16 db.insert() File /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq/devicetree.py:115, in DeviceTreeSegment.insert(self) 113 read_back = f.read(1024*1024) 114 if read_back != dtbo_data: --> 115 raise RuntimeError('Device tree {} cannot be applied'.format( 116 self.dtbo_name)) 118 if not self.is_dtbo_applied(): 119 raise RuntimeError('Device tree {} cannot be applied.'.format( 120 self.dtbo_name)) RuntimeError: Device tree pynq cannot be applied ``` I also tried ```auto dtbo_path = '/home/xilinx/Kria-PYNQ/dts/pynq.dtbo' d_test = pynq.devicetree.DeviceTreeSegment(dtbo_path) d_test.insert() ``` But got the same error. Interestingly enough `d_test.is_dtbo_applied()` returned `True` so I went ahead and tried to download my overlay with the ddr4 and the board crashed. I probably don’t have too much more debugging bandwidth for this now but if I made a simple error please let me know and I can give it another shot when I have time. Thanks! Jenny ---
**Author:** ![skalade](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/skalade/32/686_2.png) [@skalade](https://discuss.pynq.io/u/skalade)\ **Post date:** [November 16, 2022, 6:30pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/6 "2022-11-16T18:30:12Z")
Hi Jenny, Sorry I didn’t mean to use that reg value exactly as is, it should be adapted to your ddr base address and amount of memory you’re allocating. As an example this was the snippet I just used to get allocate on the rfsoc4x2: ```auto /dts-v1/; /plugin/; / { /*reserved memory*/ fragment@4 { target-path = "/"; overlay4: __overlay__ { reserved-memory { ranges; reserved { reg = <0x10 0x00 0x02 0x00>; }; }; }; }; }; ``` ![image](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/f/fc2ab74da917e011a37d0719f535a5a1b0baba84.png) DDR reg addressing is 64-bit so the reg values have to be split into 2 (upper 32-bit and lower 32-bit) base address (from vivado address editor) 0x1000000000 =\> 0x10 0x00 range (from vivado address editor) (8GB) =\> hex(8589934592) =\> 0x200000000 =\> 0x02 0x00 I think your reg should look something like reg = \<0x05 0x00 0x01 0x00\> If you’ve tried loading a device tree overlay before (and haven’t rebooted), you can remove it with ```auto rmdir /sys/kernel/config/device-tree/overlays/pynq ``` though after lots of overlay experimentation I found a reboot is sometimes necessary. Thanks Shawn ---
**Author:** ![JennySmith888](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/jennysmith888/32/1966_2.png) [@JennySmith888](https://discuss.pynq.io/u/JennySmith888)\ **Post date:** [November 16, 2022, 10:59pm UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/7 "2022-11-16T22:59:31Z")
I had a feeling I had to change the register values but wasn’t sure about the specification. Thanks for walking me through that 🙂. I updated the values to `<0x05 0x00 0x01 0x00>` as suggested and it’s working! I added it to `pynq-venv.sh` following what’s done in the Kria repo and now it’s all working as normal. Thanks so much for your help with this! I will repost step by step instructions below: In an ssh terminal to the board: 1. activate the pynq virtual environment `source /etc/profile.d/pynq_venv.sh` 2. install the device-tree-compiler `sudo apt-get install device-tree-compiler` 3. download the [Kria-PYNQ repo dts folder](https://github.com/Xilinx/Kria-PYNQ/tree/main/dts) 4. edit `pynq.dts` replacing the entire contents of the file with ```auto /dts-v1/; /plugin/; / { /*reserved memory*/ fragment@4 { target-path = "/"; overlay4: __overlay__ { reserved-memory { ranges; reserved { reg = <0x10 0x00 0x02 0x00>; }; }; }; }; }; ``` The `reg` field should be updated like so: The first value is the upper 32-bits of the PL DDR4 base address. For example if the base address is `0x500000000` this value should be `0x500000000 >> 32 = 0x5` represented as a 32-bit number so `0x05` . The second value is the lower 32-bits of the PL DDR4 base address ex: `0x500000000 & 2^32-1 = 0x0` represented as a 32-bit number so `0x00` . The third and fourth values are the upper and lower 32 bits of the allowed memory range. For example if the PL DDR4 base address is at `0x500000000` and it is 4 GiB, the high address is `0x5ffffffff` . The allowed range is `0x5ffffffff-0x500000000+1 = 0x100000000` . The high bits are `0x01` and the low bits are `0x00` so for this example the reg field would read: reg = `<0x05 0x00 0x01 0x00>` . 5. in the `dts` folder run `make` to compile the `pynq.dts` into `pynq.dtbo` 6. Make a folder in the `pynq-venv` for the dts products `mkdir -p /usr/local/share/pynq-venv/pynq-dts/` 7. copy the insert\_dtbo and .dtbo to the folder `cp insert_dtbo.py pynq.dtbo /usr/local/share/pynq-venv/pynq-dts/` 8. append the insert script to the pynq virtual environment startup script `echo "python3 /usr/local/share/pynq-venv/pynq-dts/insert_dtbo.py" >> /etc/profile.d/pynq_venv.sh` I sort of guessed at the permissions I made `insert_dtbo.py`, `pynq.dtbo`, and `pynq-dts/` all belong to root and executable using `sudo chown root: ` and `sudo chmod 755 `. Power cycled and now I am able to allocate the PL DDR4 as normal: ```auto import pynq from pynq import Overlay ol = Overlay('design_with_mig.bit', ignore_version=True) ol.download() buffer = pynq.allocate((2**19, 2), dtype='i2', target=ol.capture.ddr4_0) hex(buffer.device_address) >>'0x500000000' hex(ol.capture.ddr4_0.base_address) >>'0x500000000' ``` ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [November 17, 2022, 8:55am UTC](https://discuss.pynq.io/t/pynq3-0-1-allocate-ddr4-returns-buffer-outside-of-address-range/4918/8 "2022-11-17T08:55:00Z")