Getting nan values in the output of the convolution layers of CNN using DPU-PYNQ

[Afeff]

Hello,
I wanted to retrieve the output features of CNN using DPU-PYNQ. I retrained ResNet50 for iris flower classification.I run Vitis AI CPU docker container from host using ./docker_run.sh xilinx/vitis-ai-cpu:1.3.411.Then I run the quantization to obtain a .pb model . After that, I truncated the quantized model to keep only the CONV+POOL stage using a python script and I compile the truncated model to obtain Xmodel to run it on the ZCU104 board using DPU-PYNQ using the following code.

from time import time

import numpy as np
import cv2
import matplotlib.pyplot as plt
%matplotlib inline
from six.moves import urllib
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
from pynq_dpu import DpuOverlay
overlay = DpuOverlay("dpu.bit")
overlay.load_model("Resnet_iris_avg.xmodel") 
X = np.load("features.npy")
y = np.load("label.npy")
X = X / 255.0
test_data=X[338:]
test_label=y[338:]

dpu = overlay.runner

inputTensors = dpu.get_input_tensors()
outputTensors = dpu.get_output_tensors()

shapeIn = tuple(inputTensors[0].dims)
shapeOut = tuple(outputTensors[0].dims)
outputSize = int(outputTensors[0].get_data_size() / shapeIn[0])

print("shapeIn shape: {}".format(np.shape(shapeIn)))   
shapeIn shape: (4,)
print("shapeOut shape: {}".format(np.shape(shapeOut)))
shapeOut shape: (2,)
test= outputTensors
test1=outputTensors[0]
test2=outputTensors[0].get_data_size()
print("out:",outputSize)
out: 2048
print("shape:",shapeIn)
shape: (1, 224, 224, 3)
print("shape:",shapeOut)
shape: (1, 2048)
print("test:",test)
test: [<xir.Tensor named 'global_average_pooling2d/Mean/squeeze/aquant'>]
print("test1:",test1)
test1: {name: 'global_average_pooling2d/Mean/squeeze/aquant', shape: [1, 2048], type: 'xint8', attrs: {'ddr_addr': 0, 'quant_out_signed': True, 'bit_width': 8, 'quant_in_quantize_pos': -2147483648, 'round_mode': 'DPU_ROUND', 'quant_out_round_mode': 1, 'quant_in_bit_width': 8, 'location': 1, 'quant_out_quantize_pos': -2147483648, 'quant_in_round_mode': 1, 'if_signed': True, 'fix_point': -2147483648, 'reg_id': 1, 'quant_out_bit_width': 8, 'quant_in_signed': True}}
print("test2:",test2)
test2: 2048

image[0] = test_data[10]
job_id = dpu.execute_async(input_data, output_data)
dpu.wait(job_id) 
temp = output_data
x=output_data[0]
feat= output_data[0].reshape(1, outputSize)
print("output_data shape: {}".format(np.shape(output_data)))
print("temp: {}".format(np.shape(temp)))
print("feat: {}".format(np.shape(x)))
print("feat: {}".format(np.shape(feat)))
print("features =",(temp))
print("feat =",(feat))
print("feat1 =",(x))

output_data shape: (1, 1, 2048)
temp: (1, 1, 2048)
feat: (1, 2048)
feat: (1, 2048)
features = [array([[nan, nan, nan, ..., nan, nan, nan]], dtype=float32)]
feat = [[nan nan nan ... nan nan nan]]
feat1 = [[nan nan nan ... nan nan nan]]

I don’t get where there the problem is. Actually, I have done the same steps for another model and it was ok.
I suspected the quantization part, it runs successfully but I got this warning:

NFO: Checking Float Graph...
INFO: Float Graph Check Done.
2022-08-29 10:07:39.136938: W tensorflow/contrib/decent_q/utils/quantize_utils.cc:566] [DECENT_WARNING] Convert mean node global_average_pooling2d/Mean to AvgPool + Squeeze
2022-08-29 10:07:39.140287: W tensorflow/contrib/decent_q/utils/quantize_utils.cc:655] [DECENT_WARNING] Scale output of avg_pool node global_average_pooling2d/Mean to simulate DPU.
INFO: Calibrating for 32 iterations...
100% (32 of 32) |########################| Elapsed Time: 0:01:15 Time:  0:01:15
INFO: Calibration Done.
2022-08-29 10:08:55.545578: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv1_conv/Conv2D is -2147483640. It exceed range [-4, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545607: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block1_0_conv/Conv2D is -2147483644. It exceed range [-10, 16], modify quantize pos from 2 to -2147483632
2022-08-29 10:08:55.545612: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block1_1_conv/Conv2D is -2147483645. It exceed range [-9, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545616: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block1_2_conv/Conv2D is -2147483643. It exceed range [-7, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545619: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block1_3_conv/Conv2D is -2147483646. It exceed range [-10, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545622: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block2_1_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545625: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block2_2_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545628: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block2_3_conv/Conv2D is -2147483645. It exceed range [-9, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545631: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block3_1_conv/Conv2D is -2147483643. It exceed range [-7, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545634: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block3_2_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545637: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv2_block3_3_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545640: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block1_0_conv/Conv2D is -2147483644. It exceed range [-8, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545643: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block1_1_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545645: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block1_2_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545648: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block1_3_conv/Conv2D is -2147483646. It exceed range [-10, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545651: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block2_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545664: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block2_2_conv/Conv2D is -2147483645. It exceed range [-7, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545668: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block2_3_conv/Conv2D is -2147483647. It exceed range [-10, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545690: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block3_1_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545695: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block3_2_conv/Conv2D is -2147483645. It exceed range [-9, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545698: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block3_3_conv/Conv2D is -2147483647. It exceed range [-10, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545702: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block4_1_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545706: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block4_2_conv/Conv2D is -2147483645. It exceed range [-7, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545710: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv3_block4_3_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545715: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block1_0_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545718: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block1_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545722: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block1_2_conv/Conv2D is -2147483643. It exceed range [-6, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545725: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block1_3_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545730: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block2_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545733: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block2_2_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545737: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block2_3_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545740: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block3_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545743: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block3_2_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545748: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block3_3_conv/Conv2D is -2147483647. It exceed range [-9, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545751: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block4_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545755: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block4_2_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545758: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block4_3_conv/Conv2D is -2147483647. It exceed range [-9, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545762: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block5_1_conv/Conv2D is -2147483645. It exceed range [-7, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545774: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block5_2_conv/Conv2D is -2147483645. It exceed range [-8, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545778: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block5_3_conv/Conv2D is -2147483645. It exceed range [-9, 16], modify quantize pos from 4 to -2147483632
2022-08-29 10:08:55.545783: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block6_1_conv/Conv2D is -2147483645. It exceed range [-7, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545786: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block6_2_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545791: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv4_block6_3_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545794: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block1_0_conv/Conv2D is -2147483646. It exceed range [-9, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545799: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block1_1_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545810: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block1_2_conv/Conv2D is -2147483644. It exceed range [-7, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545815: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block1_3_conv/Conv2D is -2147483647. It exceed range [-10, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545818: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block2_1_conv/Conv2D is -2147483642. It exceed range [-5, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545822: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block2_2_conv/Conv2D is -2147483645. It exceed range [-7, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545825: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block2_3_conv/Conv2D is -2147483647. It exceed range [-10, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545828: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block3_1_conv/Conv2D is -2147483642. It exceed range [-5, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545840: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block3_2_conv/Conv2D is -2147483644. It exceed range [-6, 16], modify quantize pos from 6 to -2147483632
2022-08-29 10:08:55.545844: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1681] [DECENT_WARNING] Shift bias of node conv5_block3_3_conv/Conv2D is -2147483647. It exceed range [-10, 16], modify quantize pos from 5 to -2147483632
2022-08-29 10:08:55.545857: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node dense/MatMul is 2147483645. It exceed range [-5, 16], modify quantize pos from 14 to 2147483643

INFO: Generating Deploy Model...
INFO: Deploy Model Generated.
********************* Quantization Summary *********************      
INFO: Output:      
  quantize_eval_model: quantized_iris/quantize_eval_model.pb      
  deploy_model: quantized_iris/deploy_model.pb

[briansune]

Hello,

Please find the CNN MNIST tutorial useful:

MNIST + AXI Stream on PYNQ 2.7 (Attention to Details)

Hello PYNQ community: I wish you all doing great =] Previous support posts are having trouble on AXI stream HLS about CNN design and DMA: So I am going to share the details on 2020.2 HLS tool and PYNQ 2.7 design. The Github Repository Link for this tutorial: HLS AXI Stream People keep asking about tlast is missing, which this is common that after version (not sure when) the structure C no longer interpret as we expected: Old struct AXI_DMA_IO{ ap_int<16> data; ap_int<1>last; }; New …

[Afeff]

Sorry, but I don’t get how this tutorial can help me with my problem ?!

[briansune]

The IP you are using is DMA-based HLS or pure ARM CPU process?
If it is DMA-base then loading and extracting is the way you are looking for.
See how MNIST weights are being loaded and how results are extracted.
If this is not what you looking for then I misread you goal here.

[Afeff]

I am using the PYNQ DPU from xilinx.

[briansune]

I am trying with a guiding path, seems not working.
May be @marioruiz directly give you an answer.
Even DPU is based on Vitis AI synthesis logic to fit the CNN inferencing structure nothing really that huge different (correct me if this is wrongly considered).
So any weight load or inference result passing should always follow the basic tilling pipeline structure.

[briansune]

@Afeff

DPU-PYNQ

Afeff, I think this video can guide you more how DPU works (the language is English the commentary is Chinese):

DPU on PYNQ Z2 (3-1): Vivado

[Afeff]

The video is chineese I couldn’t understand anything!!

[briansune]

Afeff,

First there are English point-form that showing how DPU is invoke and build.

Second DPU is basically a DSP-based logic elements that used for tiling the CNN.
So APU is the control.
Why nan values return could basically due to wrong data format, not enough data feeding in or any miss control or even design fault from beginning.

Before you start debugging

You must make sure you understand how DPU APU runs and synthesized.
Or you just want to make things work without really knowing anything, then copy well-known good example and play around.
Otherwise there are no way to continue.