PYNQ: PYTHON PRODUCTIVITY

Multi-Tile Synchronization on the RFSoC2x2

Hello,

I am interested in using Multi-Tile Synchronization (MTS) (also referred to as Multi-Converter Synchronization) on the RFSoC2x2. In this post, I will detail some findings from my investigation into MTS on the 2x2 and pose some questions. It’s my hope this post can serve as a starting point for anyone else interested in MTS and yield some ideas for alternate approaches.

I’ll explain briefly why I want to use MTS:
Our application requires two DACs and two ADCs (all converters on the 2x2) all running at 4 GSPS. One DAC/ADC will be producing/sampling an I waveform and the other will be producing/sampling a Q waveform (there are external IQ Mixers). Nominally, the I and Q waveforms will be 90 degrees out of phase with respect to one another. Without Multi-Converter Synchronization, I am concerned the two ADCs/DACs may have different clock divider phase, FIFO latency, clock skew, and data skew. I am concerned these potential sources of latency uncertainty could cause problems like the I and Q waveforms becoming more or less than 90 degrees out of phase over time which would degrade system performance. I am still working on trying to quantify how much this latency uncertainty will impact system performance. It’s possible it could be calibrated out. It could be a very small effect but certainly having the option to synchronize converter clocking across tiles would be preferred.

Now my investigation into MTS:
When placing the RFDC and enabling all converters, Vivado automatically generates the two clocks for the two ADC tiles, the two clocks for the two DAC tiles, and the sysref clock.
image
When enabling MTS across the two DAC tiles and across the two ADC tiles, the RFDC IP will ask for user_sysref_adc and user_sysref_dac clocks. The image below shows an example of how to connect the clocks for MTS across two DAC tiles on the ZCU111. Of the aforementioned MTS clocks, only the user_sysref_dac clock is shown and it is labeled PL SYSREF. I redacted the unused channels on the 2x2 and colored the important clocks as well as noted the package pins where possible.


The two DAC tile clocks (DAC1_CLK_p/n and DAC0_CLK_p/n above) and the sysref clock (SYSREF_p/n above) are routed automatically in Vivado to the proper package pins but it is up to the user to specify the proper PL clock and PL SYSREF. The RFDC documentation states the following regarding these clocks

The PL SYSREF and PL Clock inputs must be differential signals that obey the clock and banking rules for the device. Dedicated clock inputs must be used for these inputs. The AXI4-Stream clocks for the Zynq UltraScale+ RF Data Converter core must be generated from the PL Clock input and not from the clock outputs from the core itself.

I began looking for clocks meeting the above criteria in the board schematic. I found that in addition to supplying the clock and sync sources for the two LMX2594 chips (which create the two DAC tile clocks and two ADC tile clocks, respectively) and the sysref clock, the LMK04832 outputs two clocks to user bank 65 and user bank 66.


I began investigating whether these two clocks could work as a PL Clock or PL SYSREF source. In our design, the ADCs and DACs are all running at 4 GSPS which means the two s_axis lines into the DACs have a required axis clock of 256 MHz and the two m_axis lines out of the ADCs have a required axis clock of 512 MHz. I went about trying to bring in the 122.88 MHz LMK clock on user bank 66 and generate the two axis clocks in the spirit of the following diagram:

I configured a clocking wizard as shown below, requesting a 256 MHz clock and 512 MHz clock and taking care to impose phase alignment on the clocks as required by MTS. (More on the requested 8 MHz clock at the end of this post).



One thing to note is that this method is predicted to yield relatively high Pk-to-Pk Jitter and Phase Error. Prior to using MTS, we had produced these 256 MHz and 512 MHz axis clocks from the 409.600 MHz PLLs internal to the RFDC tiles which results in one-half to one-third the phase error and jitter. Large jitter is especially undesirable in our design because it makes it more challenging to achieve timing closure (a top consideration for the 512 MHz clock domain). However, as noted above MTS does not allow sourcing clocks for the PL Clock or PL SYSREF from within the RFDC.

I then began to investigate the LMK04832 chip to see if it was possible to program the output clock lmk_clk2 to a different value which yielded lower jitter (I did not end up finding any values which yielded lower jitter). I was also trying to ensure that the lmk_clk2 output which I was considering using for the PL SYSREF was synchronous with the sysref clock (lmk_clk11) at the package pin inputs which is a requirement of MTS. I discovered there is a Texas Instruments eval software for the LMK chip which allows you to pull in the register values from the LMK04832_122.88.txt file located in /usr/local/lib/python3.6/dist-packages/xrfclk/ on a RFSoC2x2 board with a PYNQv2.6 image. The software shows the following output clocks for the LMK04832 when programmed with the default config file on the RFSoC2x2 PYNQv2.6 image (note the frequency units are off by a factor of 10):



These same clocks are shown in the LMK represented in the RFSoC2x2 schematic:

Looking at the three previous images, it is important to note that the sysref clock (highlighted in yellow in all images) is set to 122.88 MHz. MTS requires this sysref clock is the GCD of the DAC sample rate and the ADC sample rate (both 4.096 GSPS in our application) divided by sixteen and MTS also requires the sysref clock is less than 10 MHz. These constraints are on pages 113 and 114 in PG269. In our application the documentation would suggest we need to set this LMK CLK_out11 which will serve as the MTS SYSREF clock to 8 MHz.

However, looking back at the eval software, it would appear changing this value requires either

  1. Changing the CLK_out11 associated clock divider which would also change the value of CLK_out10 which is used to program the LMX2594 chip which supplies the DAC tile clocks. This method may involve also reprogramming the LMX chip to ensure it supplies the correct DAC tile clocks with a different reference frequency.
  2. Changing the Clock Mode Select from device clock which is the VCO shown here: To the LMK’s SYSREF circuit shown here (note this is different from the MTS sysref): Comparing this with the LMK wiring shown on the RFSoC2x2 board schematic (two images previous) which shows a fixed 122.88 MHz VCXO and 12.2880 MHz TCXO, it would appear our only option for changing the LMK CLKout11 to the desired MTS sysref_clk value of 8 MHz would be to use an external clock source and route it in through the J7 SMA clock input.

Our final thought was to abandon the LMK CLKout11 which Vivado defaults the sysref_clk to when you place the RFDC IP and instead ask the Clock Wizard to also generate an 8 MHz clock from the 122.88 MHz clock sourced from the LMK CLKout2 which is routed to user bank 66 on the RFSoC2x2 (this explains the 8 MHz clock shown in the Clock Wizard configuration images above). This strategy has the added benefit of ensuring the clocks generated from the wizard which will serve as the MTS PL Clock, PL SYSREF, and SYSREF are all in phase and synchronous with the input clock and therefore each other which is a requirement of MTS.

Please feel free to comment thoughts on which of the strategies for generating the required MTS 8 MHz sysref_clk is most viable. I would also be interested to know if anyone else is considering using MTS on the RFSoC2x2? Any other thoughts welcome as well.

4 Likes

Hi Jenny,
Can I try to clarify your question please?
Are you asking if it is better to generate the 8MHz clock via the clock wizard from the 122.88 MHz LMK output OR use a external clock through SMA?
You tried using the LMK and you find this is relatively high jitter vs what you saw using the internal PLLs?
Edit: I should also add that you may be better asking questions like this on the Xilinx forums.

Cathal

1 Like

Part of the issue is that for our broader reasons with the instrument system (~5 of these boards) we really would prefer to only use onboard sources for everything other than a PPS and 10MHz rubidium reference clock (which isn’t viable for the sysref). Jenny has a better grasp of the specifics here though so I won’t to step on her toes.

2 Likes

Thank you for looking at this Cathal. I hear what you are saying about posting on the Xilinx forum but I will still clarify here for any interested PYNQ users:

We want to use Multi-Tile Sync with 4.096 GSPS ADCs and DACs. This arrangement requires several clocks (listed below). It doesn’t appear possible to get these clocks from the RFSoC2x2. We think we would have to also feed in a clock using the external SMA which we would rather not do.

CLOCKS REQUIRED FOR MTS WITH OUR SAMPLE RATES:

  • 8 MHz (analog sysref, LMK4832 out2 or SMA )
  • 8 MHz (user sysref dac & adc, LMK4832 out1 or 11 or SMA )
  • 256 MHz (dac axis, MMCM)
  • 512 MHz (adc axis, MMCM)
  • 409.XX MHz (or other allowable Freq. by the internal DAC PLL, DAC clock, LMX2594)
  • 409.XX MHz (or other allowable Freq. by the internal ADC PLL, ADC clock, LMX2594)
  • <200MHz (Axi control clock)

All of the first four need to be synchronous. The RFDC product guide seems to indicate all of these clocks must come into the part via differential clock pins and explicitly that the DAC and ADC AXIS clocks may not come from the RFDC core. The phase noise and jitter on the 512 MHz clock heavily impacts timing as it drives >50% of our design.

The LMK chip is to configured to provide 15.36 MHz and 122.88 MHz user signals and doesn’t seem to be able to provide nice powers of two (probably hence the LMX chips?). So the question really boils down to: “How does one get these clocks?” As we described, the MMCM can create 8, 256, and 512 MHz clocks all synchronous with the LMK out_1 or out_11 but with large jitter and phase error. Moreover, the LMK chip doesn’t seem to be able to generate a solid 8 MHz for the analog sysref and the MMCM can not (as far as we can tell) be used for this.

2 Likes

Hi, Just very briefly looking this over and trying not to infatuate to much about this deep subject of MTS :slight_smile:

As you’ve found out, an infinite range of frequencies is not achievable. There are constraints, push this lever and you pull another. Again, I haven’t read it all enough to understand your exact problem, so sorry for a bit of a philosophical answer and perhaps I totally missed the point.

I wonder out loud: what if you embrace the 122.8MHz and other fundamental frequencies that the PLLs do support? What if you then go away from using spot on 4096MHz sample rates and 256/512MHz, and an exact 8MHz for the Sysref clock?

Philosophically if a high quality clock is paramount then you should optimize for that and change the other system parameters. And then just go with the flow in the digital domain. Perhaps, then you do give up a little speed in the PL or have to calculate some timing or freqs with fractional numbers. On the latter, no clock is perfect anyways, even if you call it 8.000000000MHz it isn’t; exact isn’t needed just good enough. If you really need to get back to an integer frequency in the digital domain then dual clock FIFOs can do the job, just pull out data a little faster than it goes in and you’re good to go.

Kind regards

2 Likes

Sorry for delayed replied. I don’t have much to add.
Just to address one of your comments; LMX is the clock synthesizer. LMK is the clock conditioner. This is why you don’t see the flexibility in the LMK for clock frequencies.
If you need a higher quality clock I think external may be your only option.

Cathal

1 Like