Setup Nic on host PC for 100 Gigabit Ethernet for RFSoC

Hello Jenny,

Thanks again for your work. It really helps me a lot!
I’m working on your system, but I would like to know if you have any idea how to communicate with a PC equipped with an SFP card?

By this I mean that the PC normally has an IP address and can communicate with another PC by different means (sockets for example).

But in this case, the ZCU111 doesn’t seem to use the TCP/IP protocol, and sends raw data packets instead.

Maybe there is a method to bypass the TCP/IP part of the SFP card on the PC and send data packets directly to the ZCU111?

I don’t know if it’s very clear?

Thanks in advance

Damien

Hi Damien,

Unfortunately, I’m not sure how to help with this. You might need to mess with the settings of your SFP card or the CMAC settings? I would recommend reaching out to the vendor of the SFP card and getting more information about the expected data format. It’s possible your card is expecting the CMAC to be implementing Auto-Negotiation, etc. which is not available on the free version of the CMAC IP. It’s also possible your card is expecting to see a signature related to a specific vendor. It would be good to check and make sure you can demonstrate external loopback using a passive module on both the ZCU111 SFP and the PC SFP. Hopefully the SFP card vendor can provide more support.

For ZCU111 (and RFSoC 4x2) there is no TCP/IP stack. If you want this, you would need to build it into the hardware design yourself. You can use UDP instead.
With a direct connection from your board to your NIC, you need to assign a static IP address to the NIC and the board.

  • Check/set the Link Mode to 100G
  • Set the FEC to RS (RS-FEC)
  • Set the FEC speed to 100G
    You may need to check other settings on your card.

You can also increase the MTU - this lets you transfer larger frames. You will need to check the driver for your NIC for details on how to do this.

From Python, you can use the socket library on the board and on your host PC to transfer data.

E.g.

Cathal

Hello Jenny,

Thanks for your feedback. The SFP card is a FS card: Intel E810-XXVAM2-Based Ethernet Network Interface Card, 25G Dual-Port SFP28, PCIe 4.0 x 8. It is usually used for classical networking (with TCP/IP layer).

It’s a good idea to test loopback modules on this card, I’ll try that.

Damien

Hello Cathal,

Thanks for your feedback and for creating a new topic.

I have some questions about your proposals:

1- “You can use UDP instead” → Do you have more info on this ?

2- "you need to assign a static IP address to the board. → Also, I can’t find anything in the UltraScale+ Integrated 100G Ethernet Subsystem IP configuration

3- “Check/set the Link Mode to 100G
Set the FEC to RS (RS-FEC)
Set the FEC speed to 100G
You may need to check other settings on your card.”
→ I’m not sure what to do. In the IP configuration, I have to check the option “Enable RS-FEC Transcode Bypass”? If yes, then the IP doesn’t seem to be usable at all, because most of the connectors will disappear.
I would like to have more information about this.

4- Concerning Jenny’s example, it seems that all 4 SFP28 modules are working at the same time: you can’t do a transfer with 1, 2 or 3 SFPs, only 4. This is not really a problem, but on the PC side, which “protocol” should I use to link the 4 ports together? I’m used to using LACP for example on more classical networks, but in this case I’m not sure.

Damien

Sorry for any confusion. This design (and the base Overlay) only includes the Ethernet layer.
You would need to add this or similar to have UDP layer:

You can try sending raw packets, but you need to construct the packets yourself.

Cathal

Hello Cathal,

Thanks for your feedback :slight_smile:
I will try all of these. After a first reading, it seems clearer to me.

Concerning point 4, I would like a small precision:

4- Concerning Jenny’s example, it seems that all 4 SFP28 modules are working at the same time: you can’t do a transfer with 1, 2 or 3 SFPs, only 4. This is not really a problem, but on the PC side, which “protocol” should I use to link the 4 ports together? I’m used to using LACP for example on more classical networks, but in this case I’m not sure.

Maybe it is necessary to use a QSFP28 pcie card in the PC and a QSFP28 to 4 * SFP28 cable to connect to the ZCU111? There is this reference at FS : #50494

Thanks in advance

Damien