ZCU111 pynq 2.5: problem with MTS Tile1 sync and mixer

Hello,

We are using a ZCU111 and pynq 2.5 (Vivado 2019). We have a design with 5xDAC, (4 in tile 0 and last one in tile 1). First of all, have to say we are newbies at pynq and my apologies if something done here is bad practice. Any advice will be sincerely apreciated.

Our Main parameters are :

AXI Stream Clk: 122.88MHz

Interpolation mode: x8

Samples per cycle: 4

Mixer: Fine, IQ->Real

Nyquist Zone: 1

Sampling Rate: 1966.08Msps

Sysref: 8.192MHz (less than 10MHz, 1966.08/16/15 and 122.88/15)

image

with such parameters we are trying to use MTS with the event source EVNT_SRC_SYSREF at the mixer. Nevertheless, we only achieve synce among Dac00, Dac01, Dac02 and Dac03 (tile 0), but not for the dac at tile 1.

what we are doing at pynq is the following:

-First of all we have a function RunMTS() that uses xrfdc.c structs and pointers written at xrfdc/init.py as recommended at pg269.pdf for the rfdc:

def RunMTS(self):

    self._DAC_Sync_Config = _ffi.new("XRFdc_MultiConverter_Sync_Config*") 
    self._ADC_Sync_Config = _ffi.new("XRFdc_MultiConverter_Sync_Config*")
    self.entero = _ffi.new("int*")


    _lib.XRFdc_MultiConverter_Init(self._DAC_Sync_Config,self.entero,self.entero)

    self._DAC_Sync_Config.Tiles = 0x3
    self._DAC_Sync_Config.SysRef_Enable = 0

    self.status1 = _lib.XRFdc_MultiConverter_Sync(self._instance,_lib.XRFDC_DAC_TILE,self._DAC_Sync_Config)

-In the same way, Secondly we have functions sysrefEnable() and sysrefDisable() at xrfdc/init.py:

def sysrefDisable(self):

    self.enable[0] = 0
    self.status3 = _lib.XRFdc_MTS_Sysref_Config(self._instance, self._DAC_Sync_Config, self._ADC_Sync_Config, self.enable)   

def sysrefEnable(self):

    self.enable[0] = 1
    self.status3 = _lib.XRFdc_MTS_Sysref_Config(self._instance, self._DAC_Sync_Config, self._ADC_Sync_Config, self.enable)

With all this mentioned, when loading our Overlay via jupyter notebook, and declaring our DACs, we have a function for configuring the mixer, following advices from xilinx resetting NCO pahses:

def configDacsMTS(freq):

rf.sysrefDisable()
time.sleep(1)


dac00.NyquistZone = 1
dac00.MixerSettings = {
    'CoarseMixFreq':  xrfdc.COARSE_MIX_BYPASS,
    'EventSource':    xrfdc.EVNT_SRC_SYSREF,
    'FineMixerScale': xrfdc.MIXER_SCALE_1P0,
    'Freq':           freq,
    'MixerMode':      xrfdc.MIXER_MODE_C2R,
    'MixerType':      xrfdc.MIXER_TYPE_FINE,
    'PhaseOffset':    0.0
}
#dac00.UpdateEvent(xrfdc.EVENT_MIXER)
dac_tile0.SetupFIFO(True)

dac01.NyquistZone = 1
dac01.MixerSettings = {
    'CoarseMixFreq':  xrfdc.COARSE_MIX_BYPASS,
    'EventSource':    xrfdc.EVNT_SRC_SYSREF, 
    'FineMixerScale': xrfdc.MIXER_SCALE_1P0,
    'Freq':           freq,
    'MixerMode':      xrfdc.MIXER_MODE_C2R,
    'MixerType':      xrfdc.MIXER_TYPE_FINE,
    'PhaseOffset':    0.0
}
#dac01.UpdateEvent(xrfdc.EVENT_MIXER)
dac_tile0.SetupFIFO(True)

dac02.NyquistZone = 1
dac02.MixerSettings = {
    'CoarseMixFreq':  xrfdc.COARSE_MIX_BYPASS,
    'EventSource':    xrfdc.EVNT_SRC_SYSREF,
    'FineMixerScale': xrfdc.MIXER_SCALE_1P0,
    'Freq':           freq,
    'MixerMode':      xrfdc.MIXER_MODE_C2R,
    'MixerType':      xrfdc.MIXER_TYPE_FINE,
    'PhaseOffset':    0.0
}
#dac01.UpdateEvent(xrfdc.EVENT_MIXER)
dac_tile0.SetupFIFO(True)

dac03.NyquistZone = 1
dac03.MixerSettings = {
    'CoarseMixFreq':  xrfdc.COARSE_MIX_BYPASS,
    'EventSource':    xrfdc.EVNT_SRC_SYSREF, 
    'FineMixerScale': xrfdc.MIXER_SCALE_1P0,
    'Freq':           freq,
    'MixerMode':      xrfdc.MIXER_MODE_C2R,
    'MixerType':      xrfdc.MIXER_TYPE_FINE,
    'PhaseOffset':    0.0
}
#dac01.UpdateEvent(xrfdc.EVENT_MIXER)
dac_tile0.SetupFIFO(True)

dac10.NyquistZone = 1
dac10.MixerSettings = {
    'CoarseMixFreq':  xrfdc.COARSE_MIX_BYPASS,
    'EventSource':    xrfdc.EVNT_SRC_SYSREF,
    'FineMixerScale': xrfdc.MIXER_SCALE_1P0,
    'Freq':           freq,
    'MixerMode':      xrfdc.MIXER_MODE_C2R,
    'MixerType':      xrfdc.MIXER_TYPE_FINE,
    'PhaseOffset':    0.0
}
#dac12.UpdateEvent(xrfdc.EVENT_MIXER)
dac_tile1.SetupFIFO(True)

#Reset every DAC NCO_Phase
dac00.ResetNCOPhase()
dac01.ResetNCOPhase()
dac02.ResetNCOPhase()
dac03.ResetNCOPhase()

dac10.ResetNCOPhase()

time.sleep(0.2)

rf.sysrefEnable()

time.sleep(0.2)

rf.sysrefDisable()

With this code, only sync is achieved on 4 DACs at Tile 0, but we get a delay at Tile 1’s DAC, and so huge (more than 20ns). Furthermore, when using configDacsMTS(), Dacs from Tile 0 get the new frequency and sync, but Tile 1 does not work, it becomes noisy and unstable. When using diferent Event Source (EVNT_SRC_IMMEDIATE, or EVNT_SRC_TILE) mixer behaves well, but of course without sync.

Please, could anyone tell us we could be our mistake?? we are a little bit lost at this point…
Thanks a lots in advantage, any additional info you need from design (screenshots, more parameters) will be given if needed

PD:

additional info we got from DAC_Sync_Config.Latency and DAC_Sync_Config.Offset is we are having such a huge latency, and an offset at one tile:

Latency: 172624
Offset: 0
Latency: 171720
Offset: 31

2 Likes

This question seems to be related to the setup of the RFSoC DACs rather than related to PYNQ.
You may be better posting on the Xilinx forum: https://forums.xilinx.com/

Cathal

Were you ever able to get this working?

Hi Thomas.

We did make this working. The main advise I can give you is to print Offset and Latency from DAC_Sync_Config pointer and check if Latency is too big

If it is such as big as my post before, 90% of times on our designs problem was related to a sysref constraints issue. By checking Vivado constraints properly issue was solved.

Regards

3 Likes

Hi! Is your code published anywhere? I am trying to synchronize DACs to an external signal but am having a hard time achieving synchronization. I tried emulating your code and creating all the structs and things in the c functions but am still not succeeding. Thank you so much!

Hi thomas,

the code is the one published above, i mean, just loading firmware, and using configDacsMTS() for a new freq at mixer…

If you include print latency at xrfdc/init.py (I think it´s at /usr/local/lib/python3.6/xrfdc) and check about offset and latency and there are no low numbers there check your firmware design for constraints issues. 90% of the times it was some vivado constraints issue.

Talking about external source, for the moment we have not tried yet… Sorry
Regards
Fran

1 Like

Hey Fran!

Did your method for MTS work also for the ADCs? I am trying a very similar approach for a custom board with the same chip of the ZCU111, I can achieve synchronization for the 8 DACs, but I am obtaining a very unstable behavior for the ADCs, and I am wondering if I am doing something wrong with the MTS sequence in my xrfdc/init.py. Thanks!

Best regards

Hello faber,

did not test for ADC design sorry…
regards

problem solved, I didn’t tick the option to enable MTS for all tiles in my HW design. Now MTS works for ADCs and DACs.

Best regards

Hi @fjtoralza
Is there any other setup required to get this to work? The reason I ask is that structures like XRFdc_MultiConverter_Sync_Config aren’t in the xilinx embeddedsw repo that gets pulled in by default. Did you checkout a different version of the embedded sw repo by any chance?

Thanks.

the file xrfdc_mts_example.c I think it´s present in a lot of embeddedsw versions…

Be careful about the mask por the tiles. We had a lot of problems by selecting badly that mask, make some test until it matches what you are doing in your design.

2 Likes

Hi @faber1416
Anything else you had to do to get the ADCs to work? My DACs report that MTS sync has completed but my ADCs return the DTC invalid error code. MTS enabled for all ADCs and DACs in the hardware design and sysref toggling as expected. Anything else I may have missed?

Hi @adi8v,

I get the same issue sometimes, XRFDC_MultiConverter_Sync returning the DTC invalid error code. It seems to happen from time to time, but checking the behavior of the ADCs I can see that synchronization happens even in presence of the error code. Not sure if it is just a bug of the driver or something more serious. As I got the ADCs working as I needed in my system, I haven’t investigated further. If you find out more about this, it would be awesome if you could share your findings!

Best regards

1 Like

That’s interesting. If ignore the error an go ahead and check the latencies/offsets I get the following:

###############################################
=== Multi-Tile Sync Report ===
DAC0: Latency(T1) =720, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
DAC1: Latency(T1) =720, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
DAC2: Latency(T1) =720, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =4
DAC3: Latency(T1) =720, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
ADC0: Latency(T1) =120, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
ADC1: Latency(T1) =120, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
ADC2: Latency(T1) =120, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =2
ADC3: Latency(T1) =120, Adjusted Delay Offset(T<cdata ‘unsigned int *’ owning 4 bytes>) =0
###############################################

Any idea if they look reasonable?

I’ve got all 16 DACs looped back to all 16 ADCs using the N79 loopback card that comes with the ZCU216 development platform. I have a sine wave being generated in the FPGA and passed to all 16 DACs then I can trigger a simultaneous capture from all 16 ADCs. Unfortunately the resultant captured data isn’t synchronised as I hoped. I thought this was due to the ADC synchronisation failing but it could be down to the N79 loopback card possibly not having matched tracks I guess.

2 Likes

As far as I understand, the converter latency is matching across tiles, so MTS seems to be working fine. I get the same behavior in my design. I am using a custom board so I am not sure about the N79 card. If you do further testing on this, please let me know if you can eventually solve your issue.

Best

1 Like

Hi @faber1416

Just to update you, I ran a bare metal software application on the same hardware and both the DAC and ADC MTS complete successfully.

1 Like

Hi, @adi8v and thank you for the update.

So I guess it is a problem with the Pynq wrapper returning a wrong error code when synchronizing the ADCs. MTS actually takes place, as the wrapper is using the same libraries of the bare-metal application. What do you think?

Regards

Hi @faber1416 , yes that’s my theory at the moment. I’m doing some loopback captures, correlating the captured ADC data and I “should” see a fixed latency across reboot cycles if all is well. Will let you know how that goes.

1 Like

Hi @faber1416

Update. So i’m not sure this is working as I expected. Generating a 10MHz tone in FPGA bram. Outputting the same tone synchronously to each DAC via an AXI-Stream broadcaster. The DAC mixes to 4.7GHz. Using the N79 loopback card, the 16 ADCs undersample the 4.7GHz signal and DDC to baseband where I can synchronously capture the resultant 10MHz waveforms.

This is what I get:
image

I thought the difference in the captures was due to the loopback card not having matched traces but i’m not sure that would account for that much of a difference if MTS was working.

1 Like

Hey @adi8v, did you eventually solve your problem? I have been using my systems with MTS over the last months and everything has been working as expected.

Regards

2 Likes