Kria base overlay, mipi cam5c, how to change the ColorSpaceConversion?

Support
1

Hi all,
I am working on the Kria base overlay (mipi_to_display notebook). I am trying to store a gray image via VDMA (8bits/pixel, Y channel).

Using the following code:

videomode = VideoMode(1280, 720, 8)
mipi.configure(videomode)  
frame = mipi.readframe()
frame.shape  # (720, 1280)

I get a gray image but, looking at the pynq code I see that pixel data is just written with the B channel. The reason is that v_proc_sys IP is configure to bypass the BGR pixels coming from the demosaic/gamma pipeline.

My question is, how to reconfigure the v_proc_sys IP to convert from RGB to YCRCB and then use the pixel_packer IP (V_8 mode) to store a gray image through VDMA?

something like mipi.v_proc_sys.colorspace(COLOR_IN_YCBCR) doesn´t work
Thank you

2

My last findings,
I reused some pipeline.py PYNQ code to write to the registers of the v_proc_sys IP:

def colorspace(color):
        if len(color) != 12:
            raise ValueError("Wrong number of elements in color specification")
        for i, c in enumerate(color):
            vproc.write(0x10 + 8 * i, int(c * 256))

def configBGR2YCBCR():            
    colorspace(COLOR_IN_YCBCR)

COLOR_IN_YCBCR = [0.114, 0.587, 0.299,
                  0.5, -0.331264, -0.168736,
                  -0.081312, -0.41866, 0.5,
                  0, 0.5, 0.5]

Then configuring and reading a frame

videomode = VideoMode(1280, 720, 8)  #pixel packer in V8 mode to store just the Y channel
mipi.stop()
mipi.configure(videomode)
configBGR2YCBCR()
mipi.start()
frame = mipi.readframe()
PIL.Image.fromarray(frame)

the result :

image
image960×507 22.9 KB

The Y values are very low.

I changed the original scale of the coeff (256) to 4096, but kept the offset scale to 256.

def colorspace(color):
        if len(color) != 12:
            raise ValueError("Wrong number of elements in color specification")
        for i, c in enumerate(color):
            vproc.write(0x50 + 8 * i, int(c * 4096)) #vproc.write(0x10 + 8 * i, int(c * 256))
        vproc.write(0x98, 0) # write the offsets 
        vproc.write(0x0a0, 128)  # 0.5*256
        vproc.write(0x0a8, 128)  # 0.5*256

The result looks correct, now.

image
image972×512 81.9 KB

So, this rises a new question. Is this a bug in the colourspace PYNQ method or is something missing in my code?
Regards

3

Hi all,
looking at the Xilinx drivers (layer 2): embeddedsw/XilinxProcessorIPLib/drivers/v_csc/src/xv_csc_l2.c at master · Xilinx/embeddedsw · GitHub
I have found the following:

#define XV_CSC_COEFF_FRACTIONAL_BITS   (12)
...
s32 scale_factor = (1<<XV_CSC_COEFF_FRACTIONAL_BITS);
s32 bpcScale = (1<<(pixPrec-8));  
...
switch(cstdOut)
  {
    case XVIDC_BT_601:
        switch(cRangeOut)
 	{
...
          case XVIDC_CR_16_235:
              RGB2YCC[0][0] = (s32) ( 0.299*(float)scale_factor);  //K11
              RGB2YCC[0][1] = (s32) ( 0.587*(float)scale_factor);  //K12
              RGB2YCC[0][2] = (s32) ( 0.144*(float)scale_factor);  //K13
              RGB2YCC[1][0] = (s32) (-0.172*(float)scale_factor);  //K21
              RGB2YCC[1][1] = (s32) (-0.339*(float)scale_factor);  //K22
              RGB2YCC[1][2] = (s32) ( 0.511*(float)scale_factor);  //K23
              RGB2YCC[2][0] = (s32) ( 0.511*(float)scale_factor);  //K31
              RGB2YCC[2][1] = (s32) (-0.428*(float)scale_factor);  //K32
              RGB2YCC[2][2] = (s32) (-0.083*(float)scale_factor);  //K33
              RGB2YCC[0][3] =  0*bpcScale;                   //R Offset
              RGB2YCC[1][3] =  128*bpcScale;                  //G Offset
              RGB2YCC[2][3] =  128*bpcScale;                  //B Offset
              break;

the scale_factor matches with my previous findings (for pixPrec=8).

My conclusion is that there is a bug in the PYNQ ColorConverter class (pipeline.py) regarding the scale factor of both coeffs and offsets.

Note: there is a bug in the driver too. Coeff 0.144 should be 0.114 (reported in: AMD Customer Community)
Regards

4

Hi @sergicuen,

Thank you for looking at this. I think you have found the solution. The description of the coefficients representation are here AMD Adaptive Computing Documentation Portal

It would be a good idea to create a driver for this IP to handle this. You could use something like this as reference PYNQ/pynq/lib/video/pipeline.py at master · Xilinx/PYNQ · GitHub

The different color conversions are defined here PYNQ/pynq/lib/video/common.py at master · Xilinx/PYNQ · GitHub

Would you like to do a PR to the PYNQ repo with the necessary changes?

Mario

5

Hi @marioruiz ,
I’m afraid I don’t know enough python to tackle the coding of the driver.
For the moment I leave here the workaround to get a gray image using the base overlay:

def colorspace(color):
        if len(color) != 12:
            raise ValueError("Wrong number of elements in color specification")
        for i, c in enumerate(color):
            if i<9:   # write coeff
                vproc.write(0x50 + 8 * i, int(c * 4096))   
            else:   # write offsets
                vproc.write(0x50 + 8 * i, int(c)) 

def configBGR2YCBCR():            
    vproc.write(0x10,0)  #BGR
    vproc.write(0x18,0)  #BGR
    colorspace(COLOR_IN_YCBCR_16_235)

def configBGR2BGR():            
    vproc.write(0x10,0)  #BGR
    vproc.write(0x18,0)  #BGR
    colorspace(COLOR_IN_BGR)

# from Xilinx drivers embeddedsw/XilinxProcessorIPLib/drivers/v_csc/src/xv_csc_l2.c
# YCBCR range 16:235
COLOR_IN_YCBCR_16_235= [0.114, 0.587, 0.299,
                  0.5, -0.331264, -0.168736,
                  -0.081312, -0.41866, 0.5,
                  0, 128, 128]

# YCBCR range 0:255
COLOR_IN_YCBCR_0_255 = [0.0979, 0.5041, 0.2568 ,
                  0.4393, -0.2910, -0.1482,
                 -0.0714, -0.3678, 0.4393, 
                  16, 128, 128]
# no color space conversion
COLOR_IN_BGR = [1, 0, 0,
                0, 1, 0,
                0, 0, 1,
                0, 0, 0]

# get the driver for v_proc_sys IP
vproc = base.mipi.v_proc_sys 
# stop DMA and configure pipeline
videomode = VideoMode(1280, 720, 8) 
mipi.stop()
mipi.configure(videomode)  # configure DMA write chanel and PixelPacker (mode V_8)
configBGR2YCBCR()         # configure V_proc_sys
mipi.start()

# take a frame and display it
frame = mipi.readframe()
PIL.Image.fromarray(frame)
2 Likes
6

Hi,

Thanks for your solution, I was also able to swap the color spaces from BGR to RGB during initialization and avoid having to slice the channels once every readframe call saving me a few crucial milliseconds and improving fps throughput.

Regards

1 Like