Hello,
I am interested in using Multi-Tile Synchronization (MTS) (also referred to as Multi-Converter Synchronization) on the RFSoC2x2. In this post, I will detail some findings from my investigation into MTS on the 2x2 and pose some questions. It’s my hope this post can serve as a starting point for anyone else interested in MTS and yield some ideas for alternate approaches.
I’ll explain briefly why I want to use MTS:
Our application requires two DACs and two ADCs (all converters on the 2x2) all running at 4 GSPS. One DAC/ADC will be producing/sampling an I waveform and the other will be producing/sampling a Q waveform (there are external IQ Mixers). Nominally, the I and Q waveforms will be 90 degrees out of phase with respect to one another. Without Multi-Converter Synchronization, I am concerned the two ADCs/DACs may have different clock divider phase, FIFO latency, clock skew, and data skew. I am concerned these potential sources of latency uncertainty could cause problems like the I and Q waveforms becoming more or less than 90 degrees out of phase over time which would degrade system performance. I am still working on trying to quantify how much this latency uncertainty will impact system performance. It’s possible it could be calibrated out. It could be a very small effect but certainly having the option to synchronize converter clocking across tiles would be preferred.
Now my investigation into MTS:
When placing the RFDC and enabling all converters, Vivado automatically generates the two clocks for the two ADC tiles, the two clocks for the two DAC tiles, and the sysref clock.
When enabling MTS across the two DAC tiles and across the two ADC tiles, the RFDC IP will ask for user_sysref_adc and user_sysref_dac clocks. The image below shows an example of how to connect the clocks for MTS across two DAC tiles on the ZCU111. Of the aforementioned MTS clocks, only the user_sysref_dac clock is shown and it is labeled PL SYSREF. I redacted the unused channels on the 2x2 and colored the important clocks as well as noted the package pins where possible.
The two DAC tile clocks (DAC1_CLK_p/n and DAC0_CLK_p/n above) and the sysref clock (SYSREF_p/n above) are routed automatically in Vivado to the proper package pins but it is up to the user to specify the proper PL clock and PL SYSREF. The RFDC documentation states the following regarding these clocks
The PL SYSREF and PL Clock inputs must be differential signals that obey the clock and banking rules for the device. Dedicated clock inputs must be used for these inputs. The AXI4-Stream clocks for the Zynq UltraScale+ RF Data Converter core must be generated from the PL Clock input and not from the clock outputs from the core itself.
I began looking for clocks meeting the above criteria in the board schematic. I found that in addition to supplying the clock and sync sources for the two LMX2594 chips (which create the two DAC tile clocks and two ADC tile clocks, respectively) and the sysref clock, the LMK04832 outputs two clocks to user bank 65 and user bank 66.
I began investigating whether these two clocks could work as a PL Clock or PL SYSREF source. In our design, the ADCs and DACs are all running at 4 GSPS which means the two
s_axis lines into the DACs have a required axis clock of 256 MHz and the two m_axis lines out of the ADCs have a required axis clock of 512 MHz. I went about trying to bring in the 122.88 MHz LMK clock on user bank 66 and generate the two axis clocks in the spirit of the following diagram:
I configured a clocking wizard as shown below, requesting a 256 MHz clock and 512 MHz clock and taking care to impose phase alignment on the clocks as required by MTS. (More on the requested 8 MHz clock at the end of this post).
One thing to note is that this method is predicted to yield relatively high Pk-to-Pk Jitter and Phase Error. Prior to using MTS, we had produced these 256 MHz and 512 MHz axis clocks from the 409.600 MHz PLLs internal to the RFDC tiles which results in one-half to one-third the phase error and jitter. Large jitter is especially undesirable in our design because it makes it more challenging to achieve timing closure (a top consideration for the 512 MHz clock domain). However, as noted above MTS does not allow sourcing clocks for the PL Clock or PL SYSREF from within the RFDC.
I then began to investigate the LMK04832 chip to see if it was possible to program the output clock lmk_clk2 to a different value which yielded lower jitter (I did not end up finding any values which yielded lower jitter). I was also trying to ensure that the lmk_clk2 output which I was considering using for the PL SYSREF was synchronous with the sysref clock (lmk_clk11) at the package pin inputs which is a requirement of MTS. I discovered there is a Texas Instruments eval software for the LMK chip which allows you to pull in the register values from the LMK04832_122.88.txt file located in /usr/local/lib/python3.6/dist-packages/xrfclk/ on a RFSoC2x2 board with a PYNQv2.6 image. The software shows the following output clocks for the LMK04832 when programmed with the default config file on the RFSoC2x2 PYNQv2.6 image (note the frequency units are off by a factor of 10):
These same clocks are shown in the LMK represented in the RFSoC2x2 schematic:
Looking at the three previous images, it is important to note that the sysref clock (highlighted in yellow in all images) is set to 122.88 MHz. MTS requires this sysref clock is the GCD of the DAC sample rate and the ADC sample rate (both 4.096 GSPS in our application) divided by sixteen and MTS also requires the sysref clock is less than 10 MHz. These constraints are on pages 113 and 114 in PG269. In our application the documentation would suggest we need to set this LMK
CLK_out11 which will serve as the MTS SYSREF clock to 8 MHz.
However, looking back at the eval software, it would appear changing this value requires either
- Changing the
CLK_out11associated clock divider which would also change the value ofCLK_out10which is used to program the LMX2594 chip which supplies the DAC tile clocks. This method may involve also reprogramming the LMX chip to ensure it supplies the correct DAC tile clocks with a different reference frequency. - Changing the Clock Mode Select from
device clockwhich is the VCO shown here: To the LMK’s SYSREF circuit shown here (note this is different from the MTS sysref): Comparing this with the LMK wiring shown on the RFSoC2x2 board schematic (two images previous) which shows a fixed 122.88 MHz VCXO and 12.2880 MHz TCXO, it would appear our only option for changing the LMKCLKout11to the desired MTSsysref_clkvalue of 8 MHz would be to use an external clock source and route it in through the J7 SMA clock input.
Our final thought was to abandon the LMK CLKout11 which Vivado defaults the sysref_clk to when you place the RFDC IP and instead ask the Clock Wizard to also generate an 8 MHz clock from the 122.88 MHz clock sourced from the LMK CLKout2 which is routed to user bank 66 on the RFSoC2x2 (this explains the 8 MHz clock shown in the Clock Wizard configuration images above). This strategy has the added benefit of ensuring the clocks generated from the wizard which will serve as the MTS PL Clock, PL SYSREF, and SYSREF are all in phase and synchronous with the input clock and therefore each other which is a requirement of MTS.
Please feel free to comment thoughts on which of the strategies for generating the required MTS 8 MHz sysref_clk is most viable. I would also be interested to know if anyone else is considering using MTS on the RFSoC2x2? Any other thoughts welcome as well.
