Cache coherency in pynq image using pynq-z2 board (Ethernet Bring-Up)

Support
1

We have developed a Custom IP for interfacing the Zynq 7020 device with the PS. The default pynq image does not configure ethernet and we have to configure it manually. We did the initial testing with barebones C programs on Vitis, but later needed to port to Linux for easy multi-threaded programming and ease of TCP/IP sockets.

We are facing a problem with cache coherency between the PL and the PS. The DMA is connected to the ACP port.

We made a buffer to receive data from the DMA and send it over TCP sockets. Here, the data sent from the buffer is always zero (the initialized value, checked with Wireshark). However, using a different program I made to peek into the buffer memory reveals the correct values in buffer.

Code snippet:

#define BASE_ADDR_BUFFER        0x0e000000
#define BUFFER_MAX 2048
 
int main(int argc, char const *argv[])
{
    // Handle all Memory Maps
    int dev_mem = open("/dev/mem", O_RDWR | O_SYNC);
    printf("dev_mem open");
    vaddr_dma = mmap(NULL, 0x10000, PROT_READ | PROT_WRITE, MAP_SHARED, dev_mem, 0x40400000);
    vaddr_det = mmap(NULL, 0x10000, PROT_READ | PROT_WRITE, MAP_SHARED, dev_mem, 0x43c00000);
    vaddr_buffer = mmap(NULL, BUFFER_MAX, PROT_READ | PROT_WRITE, MAP_SHARED, dev_mem, BASE_ADDR_BUFFER);
    memset(vaddr_buffer, 0, BUFFER_MAX);
 
    // Handle all the socket init stuff
   
    // Initialize everything
 
    num_sent = 0;
    while(1){
        while(dma_isBusy(vaddr_dma));
        irqStatus = dma_intrGetIrq(vaddr_dma);
        dma_intrAckIrq(vaddr_dma, irqStatus);
        if (irqStatus & DMA_IRQ_ERROR_MASK)
        {
            printf("DMA Internal Error. Reseting...\n");
            dma_resetWait(vaddr_dma);
            printf("DMA Reset done\n");
        }
        send(sock_fd, buffer, BUFFER_MAX, 0);
        printf("Sent #%u\n",num_sent);
        num_sent++;
    }
}

Peek Buffer code which shows correct values:

#define BASE_ADDR_BUFFER        0x0e000000
#define BUFFER_MAX 2048
 
unsigned int *vaddr;
char *data_ptr;
 
int main(int argc, char const *argv[]){
    int dev_mem = open("/dev/mem", O_RDWR | O_SYNC);
    vaddr = mmap(NULL, BUFFER_MAX, PROT_READ | PROT_WRITE, MAP_SHARED, dev_mem, BASE_ADDR_BUFFER);
    data_ptr = vaddr;
 
    for (int i = 0; i < BUFFER_MAX; ++i)
    {
    	if (i%32 == 0){
    		printf("%04d: ", i);
    	}
    	printf("%02X", data_ptr[i]);
    	if (i%8 == 7){
    		printf(" ");
    	}
    	if (i%32 == 31){
    		printf("\n");
    	}
    }
    munmap(vaddr, BUFFER_MAX);
    close(dev_mem);
    return 0;
}

The DMA connected to the ACP port writes directly to the APU cache memory and therefore any read operation to the location from the processor itself (test2 code) will see the expected values. The GEN controller have an internal DMA engine for data movement, so although I don’t really know whether the send function in Linux will make use of it. In such case the DMA within the GEM will access to DDR memory directly so will not fetch the values written in the APU cache in the design.

How to ensure that the cache is flushed prior to the ethernet controller starting to use it.

1 Like
2

Can you connect the DMA to the HP port instead of the ACP port? This would be the easiest solution.
If you can’t do this you need to flush the cache in software before the memory will be accessed.

Cathal

1 Like
3

Thanks.

I arrived at a working dataflow where the data was accessed after flushing by the cache in software.

I have not tried connecting the DMA to the HP port.

1 Like
4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.