100 Gigabit Ethernet for RFSoC-PYNQ Overlays
Modern FPGA designs increasingly require high-speed offload solutions to manage large data volumes. 100 Gigabit Ethernet (100 GbE) is capable of routing 100 gigabits per second, leading it to overtake slower 10 Gigabit and 40 Gigabit networks as the preferred solution for high-speed networking. Previous networking schemes have relied on traditional SFP28 connectors which are only capable of 25 Gb/sec and therefore require four connectors and cables to achieve 100 GbE. The newer QSFP connector has all four 25 Gb/sec lanes routed through one connector. This reduced form factor allows 100 GbE + QSFP solutions to provide higher-density networks while simplifying cabling and reducing power consumption. Despite the obvious utility, simple and reusable 100 GbE examples have eluded open-source code… until now.
This blog presents a simple Overlay design for the ZCU111 complete with a PYNQ test notebook demonstrating how to work with the Xilinx Integrated 100G Ethernet Subsystem (CMAC IP Core). This work addresses some of the common pitfalls of CMAC work including choosing the correct transceivers, setting the appropriate clock, and managing the data transfer. The design also features the newer QSFP connector and demonstrates how one can design and test a QSFP-fed 100 GbE Network on an RFSoC. The QSFP connector is made compatible with the ZCU111 by the Aldec QSFP to FMC daughter card which can be used to convert the ZCU111 FMC+ connector into two QSFP ports, each of which can do 100 GbE. If you are interested in how to do this please read below. For a more hands-on approach, check out the project source here.
How to 100 GbE
What is the CMAC?
Xilinx UltraScale+ devices come with the option to instantiate the UltraScale+ Integrated 100G Ethernet Subsystem (CMAC). The CMAC is a hardened IP inside of the FPGA designed to efficiently carry out the standard ethernet protocol. The ZCU111 part has two of these hardened circuits. Either of the CMACs can be chosen in the CMAC IP configuration GUI. This blog focuses on CAUI-4 mode which means 100 GbE will be achieved through four GTY transceivers, each supplying 25 GbE. This group of four transceivers is known as a quad. Each CMAC has different options for quads.
How to Pick the Right Transceivers?
Which CMAC and which quad you should use depends on how you want to route the signals out of the board. The board schematic will tell you which FPGA package pins are connected to which connectors, and the Vivado I/O Planner view will tell you which transceiver sites are connected to those pins. Once you identify the correct transceiver sites for your application, you can select them in the CMAC IP configuration.
Example: Target the SFP28 Cage:
-
The board schematic tells us the four SFP28 connectors on the board are connected to FPGA package pins Y35, Y36, AA38, …, R39.
-
Any of the aforementioned pins will lead us to the correct group of GT sites. Let’s look at the first one, Y35 in the I/O Planner.
-
If you click the pin, the “Package Pin Properties” window (left) will tell you the site is X0Y4. This sets the CMAC and GT Group Selection in the IP configuration (right).
Which Clock?
The Gigabit transceivers require a specific 156.25 MHz reference clock. This must be specified as a top-level constraint in the Vivado project. The schematic gives several clock options but I recommend USER_MGT_SI570 because it comes online at board power on and is already programmed to the correct frequency. The ZCU111 schematic tells us this clock is on bank 129. The SFP transceivers are connected to bank 128 which is ok because you are allowed to use the clock on one bank to drive the adjacent bank. Note you cannot route the clock any further, i.e., you cannot use a clock on bank 128 to drive bank 130.
Aldec Daughter Card Connected to ZCU111 FMC+ to Create QSFP
If you want to give the QSFP a connector a try, you can use the Aldec QSFP to FMC daughter card to convert the ZCU111 FMC+ connector to two QSFP ports. The process for choosing the transceivers is the same as above; however, you need to choose the transceiver quad connected to the right pins on the FMC+ connector to reach the QSFP1 or QSFP2 port on the Aldec board. This can be determined once you have the schematics for the Aldec board which are included with board purchase. The repository contains three Vivado projects which will build three bitstreams, allowing you to switch between Overlays and experiment with sending 100 GbE out of the 4 native SFP28s, or either of the two QSFPs on the Aldec daughter card.
External Loopback Modules
Passive external loopback modules internally route the transmit and receive channels to one another, allowing full external loopback testing of the CMAC, transceivers, and hardware setup. For this test, I used 4 FS “generic” 25 GbE SFP28 passive loopback modules (Product: 109377) and 2 Amphenol 100 GbE QSFP passive loopback modules (Product: SF-100GLB0W00-0DB). If you don’t have loopback modules, you can still experiment with the CMAC by enabling “Near-End PMA Loopback” in the notebook which routes the transmit path to the receive path internally as opposed to sending them off package via the transceivers. Functionality can also be probed in Vivado Hardware Manager by Integrated Logic Analyzers which exist on the CMAC transmit and receive paths.
Signal Integrity
If you are especially lucky, you may witness a special phenomenon known fondly as “signal integrity issues”. 100 GbE is a high-speed, high-frequency business and the connectors, transceivers, etc. are sensitive to any damage, dust, or poor connections. If you find yourself seeing mysterious bit-flips, the IBERT tool can be placed similar to an Integrated Logic Analyzer and can be used to explore transceiver health and connection integrity. Below you can see an example of a weak link:
The first row shows site X0Y4 as a green, 25 GbE link as expected; however, the “Errors” column shows it is accumulating bit errors and the overall BER is higher than the other four transceivers in the quad.
It’s also possible within IBERT to scan the usable area in the channel and create an eye diagram.
The eye diagram shows a lot of red and relatively little blue which means there is not much usable area in the channel and confirms the signal integrity problem indicated by the bit errors. Ideally, the eye should be more open, as opposed to squinting. Thankfully, the CMAC has some easy-to-enable options which can boost signal integrity. If needed, more options can be selected to expose individual transceiver properties. These properties can be customized to try to compensate for the issue and can be explored in detail in IBERT. I found enabling the decision-feedback equalizer (DFE) in the CMAC configuration fixed the problems I was seeing on site X0Y4. Enabling DFE helped open the eye and eliminated the bit errors.
PYNQ Verification
PYNQ makes it very easy to experiment with the CMAC through a simple Python driver and test Jupyter Notebook. The driver will be included in the PYNQ v2.7 release. The Overlay and notebook are designed to be as simple as possible, allowing users to generate ethernet packets in Python and route them through the CMAC over standard DMA.
Conclusion
Overall, 100 GbE is an exciting solution for expanding networks. I hope this blog was useful to anyone working with the Xilinx 100G Integrated Ethernet Subsystem and interesting to those wanting to learn more. The goal of this blog and associated repository is to increase 100 GbE utility and usability. I’m looking forward to hopefully seeing more amazing offload designs— especially on RFSoCs!
Acknowledgment
Special thank you to Aldec for loaning the FMC to QSFP daughter card and enabling this work.