MMIO AXI Lite access through an interconnect in IP Subsystem

baileyji · April 28, 2020, 4:41pm

Hi Folks,

pynq doesn’t seem to be properly interpreting the address map of my overlay even though I’ve provided it bd and hwh files.

My design is from a block design (A) in Project A.

This design uses a User IP block I created and exported as a user IP from a block design in Project B

When I instantiate the overlay I find that tab completion is broken and the address space in the IP dict doesn’t reflect that I’d set.

Tab completion on the overlay, ol, object proffers “ol.gen3_reschan_0” and “ol.gen3_reschan_0//S_AXI_CTRL”, the latter of whihc is not valid python. I think the latter should be ol.gen3_reschan_0.S_AXI_CTRL.

ol.hierarchy_dict[‘gen3_reschan_0’] shows

{‘device’: <pynq.pl_server.device.XlnkDevice at 0x7f67029a20>,
‘driver’: ,
‘fullpath’: ‘gen3_reschan_0’,
‘gpio’: {},
‘hierarchies’: {},
‘interrupts’: {},
‘ip’: {‘S_AXI_CTRL’: {‘addr_range’: 16384,
‘device’: <pynq.pl_server.device.XlnkDevice at 0x7f67029a20>,
‘driver’: pynq.overlay.DefaultIP,
‘fullpath’: ‘gen3_reschan_0/S_AXI_CTRL’,
‘gpio’: {},
‘interrupts’: {},
‘mem_id’: ‘S_AXI_CTRL’,
‘parameters’: {‘C_BASEADDR’: ‘0xA0034000’,
‘C_HIGHADDR’: ‘0xA0037FFF’,
‘Component_Name’: ‘test_channelizers_gen3_reschan_0_1’,
‘EDK_IPTYPE’: ‘PERIPHERAL’},
‘phys_addr’: 2684567552,
‘registers’: {‘Memory_phase0_V’: {‘access’: ‘read-write’,
‘address_offset’: 8192,
‘description’: ‘Memory phase0_V’,
‘fields’: {},
‘size’: 4096},
‘Memory_resmap_V’: {‘access’: ‘read-write’,
‘address_offset’: 4096,
‘description’: ‘Memory resmap_V’,
‘fields’: {},
‘size’: 4096},
‘Memory_toneinc_V’: {‘access’: ‘read-write’,
‘address_offset’: 4096,
‘description’: ‘Memory toneinc_V’,
‘fields’: {},
‘size’: 4096},
‘align_V’: {‘access’: ‘read-only’,
‘address_offset’: 8192,
‘description’: ‘Data signal of align_V’,
‘fields’: {‘RESERVED’: {‘access’: ‘read-only’,
‘bit_offset’: 9,
‘bit_width’: 23,
‘description’: ‘Data signal of align_V’},
‘align_V’: {‘access’: ‘read-only’,
‘bit_offset’: 0,
‘bit_width’: 9,
‘description’: ‘Data signal of align_V’}},
‘size’: 32}},
‘state’: None,
‘type’: ‘MazinLab:mkidgen3:gen3_reschan:1.12’}},
‘memories’: {}}

I would have expected the following addresses based on what I set in A’s address map and the offsets in the HLS block generated driver .h files.

Memory_resmap_V: 0xA0020000 + 0x1000 (From the bin_to_res_0 HLS block)
align_V: 0xA0020000 + 0x2000 (From the bin_to_res_0 HLS block)
Memory_phase0_V: 0xA0034000 + 0x1000 (From the resonator_dds_0 HLS block)
Memory_toneinc_V: 0xA0034000 + 0x2000 (From the resonator_dds_0 HLS block)

I’ve tried using mmio.read/write on 0xA0020000+0x1000 and I just get an invalid address space error.

rock · April 28, 2020, 5:45pm

Ok, I think this is a rare use case, so currently the hwh parsing won’t handle this properly maybe. It looks like there are 2 register spaces mapped to the same IP block, so your phys_addr shows ‭2,684,567,552‬, (0x‭A003 4000‬). It only exposes Reg 1 for your user IP. Can you show what code you are using to access 0xA0020000+0x1000? Also, does register_map work?

baileyji · April 28, 2020, 6:39pm

So first I should clarify I mislabled my expectations, I think it should be:

Memory_resmap_V: 0xA0020000 + 0x1000 (bin_to_res_0)
align_V: 0xA0020000 + 0x2000 (bin_to_res_0)
Memory_toneinc_V: 0xA0034000 + 0x1000 (resonator_dds_0)
Memory_phase0_V: 0xA0034000 + 0x2000 (resonator_dds_0)

So Pynq isn’t flipping addresses as my post implied, just skipping one offset and not discovering both cores fully.

@rock Yes, I think you are correct. This block has two AXI lite HLS slaves and I’d expect them to both be accessible. If they got mapped to one base address at 0xA002 0000 then I’d have guessed I could just use a larger offset for the higher regs. I tried (0xA0020000-0xA0034000) + 0x1000 to get at where I thought resmap should be but “ValueError: Offset cannot be negative.”

As for accessing:
print(ol.gen3_reschan_0.S_AXI_CTRL.read(0x1000)) → 0
print(ol.gen3_reschan_0.S_AXI_CTRL.read(0x2000)) → 0
print(ol.gen3_reschan_0.S_AXI_CTRL.register_map) → RegisterMap { align_V = Register(align_V=0, RESERVED=0) }

This last one I’m surprised about as that register should be located at
0xA0020000 + 0x2000. I’m not sure if any of these results are correct and I’m not yet able to get the system online (in part because I need to load the resmap!).

As for why I’m trying to do things this way, see my second comment in this thread over on the xilinx forums. Perhaps there is a better way.

rock · April 28, 2020, 6:55pm

I think you can try
ol.gen3_reschan_0.register_map

as shown in

github.com

Xilinx/PYNQ/blob/master/boards/ZCU104/base/notebooks/board/register_map_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to the Register Map\n",
    "\n",
    "It is possible to access the register map for IP in the overlay directly. This functionality requires that the `.hwh` file is distributed along with the bitstream. This can be generated from Vivado.\n",
    "\n",
    "In this example we are going us the GPIO blocks controlling the buttons and LEDs to demonstrate how to explore and interact with the register map.\n",
    "\n",
    "First we need to import the base overlay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [

This file has been truncated. show original

baileyji · April 28, 2020, 7:16pm

This doesn’t work:

print(ol.gen3_reschan_0.register_map)

AttributeError Traceback (most recent call last)
in ()
----> 1 print(ol.gen3_reschan_0.register_map)

/usr/local/lib/python3.6/dist-packages/pynq/overlay.py in getattr(self, key)
756 else:
757 raise AttributeError(
→ 758 “Could not find IP or hierarchy {} in overlay”.format(key))
759
760 def _keys(self):

AttributeError: Could not find IP or hierarchy register_map in overlay

Even if it did, since the addresses for everything except align_V are memory spaces the resonator map will raise

/usr/local/lib/python3.6/dist-packages/pynq/registers.py:422: UserWarning: Unsupported register size 4096 for register Memory_resmap_V

For bin_to_res

// control
// 0x0000 : reserved
// 0x0004 : reserved
// 0x0008 : reserved
// 0x000c : reserved
// 0x2000 : Data signal of align_V
//          bit 8~0 - align_V[8:0] (Read)
//          others  - reserved
// 0x2004 : // 0x1000 ~
// 0x1fff : Memory 'resmap_V' (256 * 96b)
//          Word 4n   : bit [31:0] - resmap_V[n][31: 0]
//          Word 4n+1 : bit [31:0] - resmap_V[n][63:32]
//          Word 4n+2 : bit [31:0] - resmap_V[n][95:64]
//          Word 4n+3 : bit [31:0] - reserved

For resonator_dds

// control
// 0x1000 ~
// 0x1fff : Memory 'toneinc_V' (256 * 128b)
//          Word 4n   : bit [31:0] - toneinc_V[n][31: 0]
//          Word 4n+1 : bit [31:0] - toneinc_V[n][63:32]
//          Word 4n+2 : bit [31:0] - toneinc_V[n][95:64]
//          Word 4n+3 : bit [31:0] - toneinc_V[n][127:96]
// 0x2000 ~
// 0x2fff : Memory 'phase0_V' (256 * 128b)
//          Word 4n   : bit [31:0] - phase0_V[n][31: 0]
//          Word 4n+1 : bit [31:0] - phase0_V[n][63:32]
//          Word 4n+2 : bit [31:0] - phase0_V[n][95:64]
//          Word 4n+3 : bit [31:0] - phase0_V[n][127:96]

baileyji · April 28, 2020, 7:18pm

The IP modules I wrote that are capable of successfully controlling the individual HLS IP blocks are here:

https://github.com/MazinLab/MKIDGen3/blob/develop/mkidgen3/pynq.py

They just don’t work not that the blocks are buried inside the module.

rock · April 28, 2020, 8:08pm

I think there are 2 approaches that I would do: 1. flatten the gen3_reschan_0 IP so the HLS IPs can directly connect to the top-level AXI interconnect. That should work. 2. If you want to keep the hierarchy, make gen3_reschan_0 a hierarchical block instead of an IP. It will be similar as the 1st approach.

Another thing that I noticed, is the 2rd address segment is A0034000 - 00A0037FFF? Should that be A0024000-A0027FFF? The user IP may just have contiguous addresses but your top-level design has 2 separate address segments. If you change it back, maybe you will be able to see additional things using ol.gen3_reschan_0.S_AXI_CTRL.register_map

baileyji · April 28, 2020, 10:38pm

I redid it with offsets adjusted in the top level so that reg0 is at 0xA002 0000 and reg1 second at 0xA002 4000. The User IP address table is unchanged from the screen-shot in my original post.
ol.gen3_reschan_0.resmap still raises the same error. ol.gen3_reschan_0.S_AXI_CTRL.register_map looks like below, so by all accounts it is still thinking the spaces overlap.

print(ol.gen3_reschan_0.S_AXI_CTRL.register_map)

RegisterMap {
  align_V = Register(align_V=0, RESERVED=0)
}

/usr/local/lib/python3.6/dist-packages/pynq/registers.py:422: UserWarning: Unsupported register size 4096 for register Memory_resmap_V
  v[2], k
/usr/local/lib/python3.6/dist-packages/pynq/registers.py:422: UserWarning: Unsupported register size 4096 for register Memory_toneinc_V
  v[2], k
/usr/local/lib/python3.6/dist-packages/pynq/registers.py:422: UserWarning: Unsupported register size 4096 for register Memory_phase0_V
  v[2], k

It is insisting the register space is 16K though, so if I try to manually read based on what I’d expect the addresses to be I get IndexError: index 5120 is out of bounds for axis 0 with size 4096.

Regarding approach 1) I could investigate breaking out the AXI Lite ports of the HLS cores individually for this core. For others though that may make a really large number of core connections.

For approach 2) For other reasons I’m not sure I can break this up into a hierarchy and not use IP subsystems (Vivado can’t instantiate block diagrams in a top level DB and the RTL wrappers don’t help) and if you’ve got 4-16 copies of a hierarchy you must keep them in sync manually. Plus git revisions become a all jumbled together for the various subsystems.

Any other ideas? The only other thing I can think of is increasing the size of Reg0 and trying to force an overlap? Maybe I just need to get my hands dirty and contribute to some part of pynq directly.

The only other thing I can think to point out is that Vivado itself didn’t want to automatically complete address assignment for Reg1… I had to right click it and select assign address, which it did and validated just fine.

rock · April 28, 2020, 11:15pm

There is also a reason why you don’t want to wrap all the HLS IPs and AXI interconnect block into a single IP. If any IP gets updated, your user IP needs to be manually upgraded. So if you have an IP over another user IP over another HLS IP, this has to be done for each layer. Putting IPs on top level will help make the upgrading much easier.

Hierarchical blocks are much easier to use than IP. Since you can pop it up and directly view its internal organization at the top level. For this reason all of our base and logictools overlays follow that design methodology.

That said, I am not denying the problem you see. One thing to confirm: can you double click your user IP to see if there is anything to configure? When you package this user IP using IP packager, have you checked the address section?

Also, as a baseline, the hacky way to make this work, is to directly use MMIO class to access the known addresses. Something like:

reg0 = MMIO(0xA0020000, 16384)
reg0.read(0x2000)

If you still run into issues, this might mean there is something wrong with the bitstream, instead of Python software.

baileyji · April 29, 2020, 3:32am

I also wrote the user IP here and there is nothing to configure (as I expect).

So I broke out the two HLS IP axi lite ports and got rid of the imbedded interconnect. Still no dice.

The overlay now has both gen3_reschan_dual_0/s_axi_bin2res and gen3_reschan_dual_0/s_axi_dds … but both report the same base address and offsets!

Seems to me there is some sort of bug lurking here. Hierarchies in a single block design may be my only choice…

ol.ip_dict['gen3_reschan_dual_0/s_axi_bin2res'] 

{'addr_range': 16384,
 'device': <pynq.pl_server.device.XlnkDevice at 0x7f900aea20>,
 'driver': pynq.overlay.DefaultIP,
 'fullpath': 'gen3_reschan_dual_0/s_axi_bin2res',
 'gpio': {},
 'interrupts': {},
 'mem_id': 's_axi_bin2res',
 'parameters': {'C_BASEADDR': '0xA0024000',
  'C_HIGHADDR': '0xA0027FFF',
  'Component_Name': 'test_channelizers_gen3_reschan_dual_0_0',
  'EDK_IPTYPE': 'PERIPHERAL'},
 'phys_addr': 2684502016,
 'registers': {'Memory_phase0_V': {'access': 'read-write',
   'address_offset': 8192,
   'description': 'Memory phase0_V',
   'fields': {},
   'size': 4096},
  'Memory_resmap_V': {'access': 'read-write',
   'address_offset': 4096,
   'description': 'Memory resmap_V',
   'fields': {},
   'size': 4096},
  'Memory_toneinc_V': {'access': 'read-write',
   'address_offset': 4096,
   'description': 'Memory toneinc_V',
   'fields': {},
   'size': 4096},
  'align_V': {'access': 'read-only',
   'address_offset': 8192,
   'description': 'Data signal of align_V',
   'fields': {'RESERVED': {'access': 'read-only',
     'bit_offset': 9,
     'bit_width': 23,
     'description': 'Data signal of align_V'},
    'align_V': {'access': 'read-only',
     'bit_offset': 0,
     'bit_width': 9,
     'description': 'Data signal of align_V'}},
   'size': 32}},
 'state': None,
 'type': 'MazinLab:mkidgen3:gen3_reschan_dual:1.0'}

and

ol.ip_dict['gen3_reschan_dual_0/s_axi_dds']

{'addr_range': 16384,
 'device': <pynq.pl_server.device.XlnkDevice at 0x7f900aea20>,
 'driver': pynq.overlay.DefaultIP,
 'fullpath': 'gen3_reschan_dual_0/s_axi_dds',
 'gpio': {},
 'interrupts': {},
 'mem_id': 's_axi_dds',
 'parameters': {'C_BASEADDR': '0xA0024000',
  'C_HIGHADDR': '0xA0027FFF',
  'Component_Name': 'test_channelizers_gen3_reschan_dual_0_0',
  'EDK_IPTYPE': 'PERIPHERAL'},
 'phys_addr': 2684485632,
 'registers': {'Memory_phase0_V': {'access': 'read-write',
   'address_offset': 8192,
   'description': 'Memory phase0_V',
   'fields': {},
   'size': 4096},
  'Memory_resmap_V': {'access': 'read-write',
   'address_offset': 4096,
   'description': 'Memory resmap_V',
   'fields': {},
   'size': 4096},
  'Memory_toneinc_V': {'access': 'read-write',
   'address_offset': 4096,
   'description': 'Memory toneinc_V',
   'fields': {},
   'size': 4096},
  'align_V': {'access': 'read-only',
   'address_offset': 8192,
   'description': 'Data signal of align_V',
   'fields': {'RESERVED': {'access': 'read-only',
     'bit_offset': 9,
     'bit_width': 23,
     'description': 'Data signal of align_V'},
    'align_V': {'access': 'read-only',
     'bit_offset': 0,
     'bit_width': 9,
     'description': 'Data signal of align_V'}},
   'size': 32}},
 'state': None,
 'type': 'MazinLab:mkidgen3:gen3_reschan_dual:1.0'}

rock · April 29, 2020, 6:56am

I don’t think there is a bug. Check phys_addr, they are: 2684502016 (A0024000) and 2684485632 (‭A002 0000‬). The parameters section we parsed is from your IP settings - I think they are just showing what you have when you wrap up you user IP.

And phys_addr is the true address we are using.

baileyji · April 29, 2020, 3:02pm

I missed that. I’ll see if I can actually get things online.

I tried looking through the hwh file and it does seem very tricky to properly tease out which registers should go with which interface. While they are grouped into two sections to figure out the address they go with you’d need to to the section and then tease it out by interface. While I could imagine doing this for this case I suspect some cases are even more involved. Since some parsing is being done to distinguish the interfaces I do still think that something is wrong: the HWH file (which I’ve attached) doesn’t list phase0 under the dds interface, for instance.

test_channelizers…hwh.txt (572.0 KB)

rock · April 29, 2020, 4:18pm

Hi, I checked the hwh file. Which part is missing? The only phase0 that I can find in that file is:

<REGISTER NAME="Memory_phase0_V">
              <PROPERTY NAME="DESCRIPTION" VALUE="Memory phase0_V"/>
              <PROPERTY NAME="ADDRESS_OFFSET" VALUE="8192"/>
              <PROPERTY NAME="SIZE" VALUE="4096"/>
              <PROPERTY NAME="ACCESS" VALUE="read-write"/>
              <PROPERTY NAME="IS_ENABLED" VALUE="true"/>
              <PROPERTY NAME="RESET_VALUE" VALUE="0"/>
            </REGISTER>

Does this register show up in the register list?

baileyji · April 29, 2020, 4:33pm

None of the memories show up in the register list as they are all of size 4096 which results in a warning that the register_map doesn’t support them (so I’m guessing pynq culls them internally). I wrote a driver to handle these so that isn’t really a problem.

The issue I’m referring to is that as you point out phase0 is in the hwh file once for the dds interface but shows up in the ip_dict twice: once for ol.ip_dict[‘gen3_reschan_dual_0/s_axi_dds’] and once for ol.ip_dict[‘gen3_reschan_dual_0/s_axi_bin2res’]. Basically everything belonging to either interface is listed under both interfaces.

For a core that had 16 interfaces (even if the phys_addr) are correct) that would be a huge amount of duplication and make figuring out what is what a real nightmare.

Topic		Replies	Views
PYNQ custom overlay using m_axi Interface not working Support	3	906	May 14, 2021
How to address axilite interface in pynq v3.0 Support	19	674	October 31, 2022
AXI Timer MMIO Support	2	661	April 11, 2021
IP can't load to PYNQ Z1, but could be found at ip_dict Support	3	518	July 29, 2020
My ip of tutorial of hls_m_axi_example doesn't work Support	1	150	December 18, 2023

MMIO AXI Lite access through an interconnect in IP Subsystem

Related Topics