Hello All
I am trying to implement image filter2d on Pynq-Z1 using Vitis Vision Library and Vitis HLS/Vivado. I am now stuck on how to feed filter parameters on my design. I do not know how to connect the filter correctly. I set the filter as HLS INTERFACE axis port in Vitis HLS.
Below is attached my design.
Any help is very appreciated.
Pynq 2.7
Board PYNQ-Z1
Vitis HLS 2023.1
Vivado 2023.1
Hi Kamal,
As you have set the filter parameter interface as axi stream port, you have to send the filter data using axi stream protocol. For that you can use another DMA, and connect the output mm2s port to the filter port. But I think it is better to have a filter port using axi4-lite interface rather than using it as a stream port because neither it is much of an overhead for the register space nor it is changing continuously.
Bests,
Mizan
Hi mizan,
Thank you for your reply.
I have already tried to use another DMA. However, I do not know how to connect the other pins. Look at my design below.
I have modified my design to be able to send filter parameters as Axi stream. Look at my design below, when I try to feed filter parameters from Python jupyter notebook it throws the error AttributeError: ‘numpy.ndarray’ object has no attribute ‘physical_address’
Any help is very appreciated
# In[1]:
from PIL import Image
import numpy as np
import cv2
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')
from pynq import allocate, Overlay
# In[2]:
filter2d_design = Overlay("filter2d.bit")
# In[3]:
for i in filter2d_design.ip_dict:
print(i)
# In[4]:
dma_0 = filter2d_design.axi_dma_0
dma_1 = filter2d_design.axi_dma_1
filter2d = filter2d_design.Filter2d_accel_0
# In[5]:
image_path = "/home/xilinx/jupyter_notebooks/images/Emir_Abdelkader_4K_Noisy.jpg"
original_image = Image.open(image_path)
# In[6]:
canvas = plt.gcf()
size = canvas.get_size_inches()
canvas.set_size_inches(size*2)
width, height = original_image.size
print("Image size: {}x{} pixels.".format(width, height))
plt.figure(figsize=(12, 10));
_ = plt.imshow(original_image)
# In[7]:
in_buffer = allocate(shape=(height, width, 3),
dtype=np.uint8, cacheable=1)
out_buffer = allocate(shape=(height, width, 3),
dtype=np.uint8, cacheable=1)
# In[8]:
in_buffer[:] = np.array(original_image)
filter = np.array([[0.0,1.0,0.0],[1.0,-4.0,1.0],[0.0,1.0,0.0]],np.float32)
# In[9]:
def run_kernel():
dma_0.sendchannel.transfer(in_buffer)
dma_0.recvchannel.transfer(out_buffer)
dma_1.sendchannel.transfer(filter)
filter2d.write(0x00,0x81) # start
dma_0.sendchannel.wait()
dma_0.recvchannel.wait()
dma1.sendchannel.wait()
# In[10]:
#bilateral.register_map.sigma_color = 200
#filter2d.register_map.filter = 3
filter2d.register_map.rows = height
filter2d.register_map.cols = width
# In[11]:
run_kernel()
filter2d_image = Image.fromarray(out_buffer)
# In[12]:
print("Image size: {}x{} pixels.".format(width, height))
plt.figure(figsize=(12, 10));
_ = plt.imshow(filter2d_image,cmap='gray', vmin=0, vmax=255) #,cmap='gray', vmin=0, vmax=255
Hi @Kamal,
A couple of things here. Even though it is unimportant at this point as the code flow is a little bit unclear to me, you have not stated where the errors come in. You have set filter2d as a stream, so I assume you are storing it somewhere, or you have completely ignored just sending it one time. Also as far as I remember there is no float implementation of the filter attributes in filter2d function in vision library (I might be wrong). The transferring process of the data is also ambiguous as filter is sent at last. At last, as I previously suggested it is better to implement the filter, size as an Axi4-lite interface rather than stream. If you are implementing a stream you should be careful how the process is running inside your IP. Another suggestion would be to use vdma instead of DMA when you are using images instead of data, it is easier to handle.
Mizan
Hi @Kamal,
The error is very clear: AttributeError: ‘numpy.ndarray’ object has no attribute ‘physical_address’ you cannot pass an ndarray to the DMA. You need to pynq.allocate to be able to move this data.
Mario
Thank you all for your help.
I succeeded in sending filter parameters as integers. However, I am unable to send float numbers.
The Vitis Vision library states that the filter parameters should be either 16-bit or 16-bit fixed point equivalent. I have several questions:
I tried the following, for ‘pynq allocate’, I defined the data type as float
filter_param = allocate(shape=(3,3), dtype=np.float32)
Then, I tried to affect the NumPy array as
filter_param [:] =np.ones((3,3),np.float32)/9.0
However, I do not get the same results as software implementation.
Thank you for your replies.
I am not exactly sure, why in documentation they have written it
The input filter of any size, provided the dimensions should be an odd number. The filter co-efficients either a 16-bit value or a 16-bit fixed point equivalent value.
But in actual implementation of the code in the library, i could not find any implementation which is taking filter as float.
static void xFFilter2Dkernel(xf::cv::Mat<SRC_T, ROWS, COLS, NPC, XFCVDEPTH_IN_1>& _src_mat,
xf::cv::Mat<DST_T, ROWS, COLS, NPC, XFCVDEPTH_OUT_1>& _dst_mat,
short int _filter_kernel[K_HEIGHT][K_WIDTH],
unsigned char shift,
uint16_t rows,
uint16_t cols)
template <int DEPTH_SRC, int DEPTH_DST, int F_HEIGHT, int F_WIDTH, int PLANES>
void xFApplyFilter2D(XF_PTNAME(DEPTH_SRC) _kernel_pixel[F_HEIGHT][F_WIDTH],
short int _kernel_filter[F_HEIGHT][F_WIDTH],
XF_PTNAME(DEPTH_DST) & out,
unsigned char shift)
Maybe something inside the function is doing something which is unclear to me.
Hi mizan
Thank you for your reply,
Please take a look at this implementation on GitHub. They define filter parameters as a float.
https://github.com/Xilinx/PYNQ-ComputerVision/blob/master/boards/Pynq-Z1/notebooks/filter2d.ipynb
Hi @Kamal,
The reference you provided is already almost obsolete, they are not maintaining it, though the vision library code for convolution back then was almost the same. I believe they have done something with their top layer which converts the float to a 16-bit signed number, that’s probably doing the desired processing. You can check the below code, how or where it is converted.
Mizan
Hello
I have added a function that converts float values to fixed-point representation but without success. The Jupyter Python code freezes when I run the kernel.
This is the code
const float min_fixed = -1.0;
const float max_fixed = 1.0;
const int bits = 16;
// Function to convert a 3x3 float matrix to a 3x3 fixed-point matrix
void convertFloatToFixedPoint(const float* floatArray, short int* filter, int arraySize) {
for (int i = 0; i < arraySize; ++i) {
float floatValue = floatArray[i];
// Assuming float values in the range [-1.0, 1.0]
short int fixedValue = static_cast<short int>(floatValue * 32767.0);
filter[i] = fixedValue;
}
}
In the main function, I used the following code to read filter values as float and convert them to fixed point representation.
int arraySize = 9;
short int filter[arraySize];
convertFloatToFixedPoint(filter_val, filter, arraySize);
The main function was declared as: void Filter2d_accel(stream_t& img_in, float* filter_val, unsigned char shift, stream_t& img_out, int rows, int cols)
Any help is very appreciated