|
本帖最后由 puyanan 于 2019-8-6 15:53 编辑
环境:PC虚拟机Ubuntu16.04; rknn_toolkit-1.0.0-cp35-cp35m-linux_x86_64.whl 及其他依赖项。
网络模型:facenet的预训练模型20180402-114759.pb
该模型对应的特征提取网络是 Inception ResNet v1
(facenet官网 https://github.com/davidsandberg/facenet)
step1 模型转换:(不量化)
1.1
- from rknn.api import RKNN
- INPUT_SIZE = 160
- if __name__ == '__main__':
- # Create RKNN object
- rknn = RKNN(verbose=True, verbose_file='./test1.log')
- # Config for Model Input PreProcess
- rknn.config(channel_mean_value='0 0 0 1', reorder_channel='0 1 2')
- print('config done')
- #load tensorflow model
- print('--> Loading model')
- rknn.load_tensorflow(tf_pb='./20180402-114759.pb',
- inputs=['input','phase_train'],
- outputs=['InceptionResnetV1/Bottleneck/BatchNorm/Reshape_1'],
- input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3], [1]])
- print('done')
- # Build Model
- print('--> Building model')
- rknn.build(do_quantization=False)
- print('done')
- # Export RKNN Model
- rknn.export_rknn('./facenet_Reshape_1.rknn')
- rknn.release()
这样写会报错:
E Catch exception when loading tensorflow model: ./20180402-114759.pb!
T Traceback (most recent call last):
T File "rknn/api/rknn_base.py", line 137, in rknn.api.rknn_base.RKNNBase.load_tensorflow
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 482, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.pre_process
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 102, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.pre_proces
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 627, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.calc_2_const
T File "rknn/base/RKNNlib/converter/tf_util.py", line 371, in rknn.base.RKNNlib.converter.tf_util.TFProto_Util.query_tensor
T File "rknn/base/RKNNlib/converter/tf_util.py", line 374, in rknn.base.RKNNlib.converter.tf_util.TFProto_Util.query_tensor
T File "rknn/base/RKNNlib/converter/tf_util.py", line 388, in rknn.base.RKNNlib.converter.tf_util.TFProto_Util.query_tensor
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3972, in get_tensor_by_name
T return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3796, in as_graph_element
T return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3838, in _as_graph_element_locked
T "graph." % (repr(name), repr(op_name)))
T KeyError: "The name 'phase_train:0' refers to a Tensor which does not exist. The operation, 'phase_train', does not exist in the graph."
done
--> Building model
E Model or data is None, please load model first.
done
E RKNN model data is None, please load model first!
(我看到论坛上一位同学在inputs中添加了‘phase_train’,但是我这里却报错。 链接: http://t.rock-chips.com/forum.php?mod=viewthread&tid=419 )
1.2
修改 inputs=[], input_size_list=[]:
- #load tensorflow model
- print('--> Loading model')
- rknn.load_tensorflow(tf_pb='./20180402-114759.pb',
- inputs=['input'],
- outputs=['InceptionResnetV1/Bottleneck/BatchNorm/Reshape_1'],
- input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3]])
- print('done')
没报错,生成了 facenet_Reshape_1.rknn (后面用来做正向推理)
1.3
facenet正向推理,最终得到一个特征向量embeddings,为什么outputs不写“embeddings” 呢?
因为会报错。
- #load tensorflow model
- print('--> Loading model')
- rknn.load_tensorflow(tf_pb='./20180402-114759.pb',
- inputs=['input'],
- outputs=['embeddings'],
- input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3]])
- print('done')
出错信息:
I build output layer attach_embeddingsut0
I build input layer input:in0
D Try match Mul embeddings
I Match multiply [['embeddings']] [['Mul']] to [['multiply']]
D Try match Rsqrt embeddings/Rsqrt
W Not match node embeddings/Rsqrt Rsqrt
E Catch exception when loading tensorflow model: ./20180402-114759.pb!
T Traceback (most recent call last):
T File "rknn/api/rknn_base.py", line 185, in rknn.api.rknn_base.RKNNBase.load_tensorflow
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 589, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.match_paragraph_and_param
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 488, in rknn.base.RKNNlib.converter.convert_tf.convert_tf._tf_push_ready_node
T TypeError: 'NoneType' object is not iterable
done
--> Building model
E Model or data is None, please load model first.
done
E RKNN model data is None, please load model first!
从输出信息看,是输出节点名字不匹配?
用tensorboard查看模型:
embeddings模块内部是将网络输出的向量做一个L2-norm,得到最终的特征向量。
红色节点的名字是:embeddings/(embeddings) (这是一个512维度的向量)
该节点前一个节点的名字是:InceptionResnetV1/Bottleneck/BatchNorm/Reshape_1 (这也是一个512维度的向量)
1.4
当我修改outputs写成
outputs=['embeddings/(embeddings)'],
报错信息:
pu@pu-vm:~/RKNN/demo/facenet$ python3 test1.py
D save dump info to: ./test1.log
config done
--> Loading model
D import clients finished
WARNING: Logging before flag parsing goes to stderr.
W0806 11:09:18.194516 140165858322176 deprecation_wrapper.py:119] From /usr/local/lib/python3.5/dist-packages/rknn/api/rknn.py:62: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
W0806 11:09:18.313679 140165858322176 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/rknn/api/rknn.py:62: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
E Catch exception when loading tensorflow model: ./20180402-114759.pb!
T Traceback (most recent call last):
T File "rknn/api/rknn_base.py", line 136, in rknn.api.rknn_base.RKNNBase.load_tensorflow
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 102, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.__init__
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 50, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.__init__
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 65, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.scan_and_optim_graph
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
T return func(*args, **kwargs)
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/graph_util_impl.py", line 182, in extract_sub_graph
T _assert_nodes_are_present(name_to_node, dest_nodes)
T File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/graph_util_impl.py", line 137, in _assert_nodes_are_present
T assert d in name_to_node, "%s is not in graph" % d
T AssertionError: embeddings/(embeddings) is not in graph
done
--> Building model
E Model or data is None, please load model first.
done
E RKNN model data is None, please load model first!
从报错信息看,说embeddings/(embeddings) 不在graph中。可是明明在啊。
我后面还尝试过 “embeddings/embeddings” "(embeddings)" 都不对。那节点名字要怎么写呢??????
step2
暂且使用上面成功生成的 facenet_Reshape_1.rknn 来做正向推理。
遇到的问题是:rknn模型的输出数据与pb模型不同。
2.1
rknn模型正向推理代码:
- from rknn.api import RKNN
- import cv2
- import numpy as np
- INPUT_SIZE = 160
-
- if __name__ == '__main__':
- # Create RKNN object
- rknn = RKNN(verbose=True, verbose_file='./test2.log')
- #load rknn model
- print('--> Loading rknn model')
- rknn.load_rknn('./facenet_Reshape_1.rknn')
- print('done')
- #set inputs
- img1 = np.load('image_1.npy')
- print('img1.type:', type(img1)) #numpy.ndarray
- print('img1.shape:', img1.shape) #(1,160,160,3)
-
- # init runtime environment
- print('--> Init runtime environment')
- ret = rknn.init_runtime()
- if ret != 0:
- print('Init runtime environment failed')
- exit(ret)
- print('done')
- # Inference
- print('--> Running model')
- outputs1 = rknn.inference(inputs=[img1])
- array1 = np.array(outputs1)
- print('output_shape:', array1.shape)
- np.save('./u_Reshape_1.npy', array1)
- print('done')
-
- rknn.release()
输出数据保存在 u_Reshape_1.npy 文件中。
facenet的pb模型直接做正向推理的代码:(工具:Anaconda + spyder)- def main(args):
- #images = load_and_align_data(args.image_files, args.image_size, args.margin, args.gpu_memory_fraction)
- images = np.load('./results/image_1.npy')
- print('images_shape:', images.shape) #(1,160,160,3)
- with tf.Graph().as_default():
- with tf.Session() as sess:
-
- # Load the model
- facenet.load_model(args.model)
-
- #test
- #summary_writer = tf.summary.FileWriter("F:/pyn/spyder_projects/log", sess.graph)
- #test
-
- # Get input and output tensors
- images_placeholder = tf.get_default_graph().get_tensor_by_name("input:0")
- phase_train_placeholder = tf.get_default_graph().get_tensor_by_name("phase_train:0")
- #middle layer
- node_Reshape_1 = tf.get_default_graph().get_tensor_by_name("InceptionResnetV1/Bottleneck/BatchNorm/Reshape_1:0")
-
- # Run forward pass to calculate embeddings
- feed_dict = { images_placeholder: images, phase_train_placeholder:False }
- Reshape_1 = sess.run(node_Reshape_1, feed_dict=feed_dict)
-
- print('Reshape_1 shape:', Reshape_1.shape)
- np.save('./results/Reshape_1.npy', Reshape_1)
输出信息(截取):
D [rknn_init:940] Input Tensors:
D [printRKNNTensor:853] index=0 name= n_dims=4 dims=[1 160 160 3] n_elems=76800 size=153600 fmt=NHWC type=FP16 qnt_type=NONE fl=-128 zp=85592704 scale=0.000000
D [rknn_init:953] Output Tensors:
D [printRKNNTensor:853] index=0 name= n_dims=2 dims=[0 0 1 512] n_elems=512 size=1024 fmt=NCHW type=FP16 qnt_type=NONE fl=-128 zp=85592704 scale=0.000000
done
--> Running model
D [rknn_inputs_set:1155] 0 pass_through=0
D [rknn_inputs_set:1173] 0 input.type=3
D [__input_copy:656] UINT8->FLOAT16
I [RK_nn_ConvertTensorToData:595]Create 1024 data.
output_shape: (1, 1, 512)
done
输出数据保存在 Reshape_1.npy 文件中。
对比u_Reshape_1.npy 和 Reshape_1.npy两个文件的数据,理论上它们应该是一样的。因为pb模型转换成rknn模型的时候并没有做量化。
但实际上不同。
- import numpy as np
- pc_data = np.load('./results/Reshape_1.npy')
- u_data = np.load('./results/u_Reshape_1.npy')
- print('pc_data.shape:', pc_data.shape)
- print('u_data.shape:', u_data.shape)
- pc_data = np.squeeze(pc_data)
- pc_data_flat = pc_data.flatten()
- u_data = np.squeeze(u_data)
- print('pc_data_flat.shape:', pc_data_flat.shape)
- print('u_data.shape:', u_data.shape)
- print(pc_data_flat)
- print(u_data)
输出:
pc_data.shape: (1, 512)
u_data.shape: (1, 1, 512)
pc_data_flat.shape: (512,)
u_data.shape: (512,)
[-5.74858040e-02 7.62917578e-01 -8.15165758e-01 -1.15798497e+00
-2.25954747e+00 -4.45677876e-01 1.19305098e+00 7.95612752e-01
-1.26038074e+00 -3.16775769e-01 5.00256717e-02 3.58441472e-01
2.47351110e-01 -2.25165412e-01 6.32283926e-01 -7.77496099e-01
-9.95897710e-01 -1.48216116e+00 -8.97606254e-01 -9.74645317e-01
1.09282449e-01 -1.73147905e+00 1.52980596e-01 1.66164684e+00
1.54646719e+00 -8.88299108e-01 3.76475036e-01 -4.77487057e-01 ……]
[ 3.55224609e-01 -7.94433594e-01 -7.58300781e-01 3.95507812e-01
2.59277344e-01 3.18847656e-01 1.06933594e-01 2.07519531e-01
-4.89501953e-01 1.44409180e-01 -1.46484375e-01 4.59716797e-01
-6.03027344e-01 -3.54736328e-01 1.63940430e-01 -3.79150391e-01
4.98046875e-01 3.13476562e-01 5.80566406e-01 -4.04296875e-01
9.24682617e-02 1.11145020e-01 4.24804688e-01 -1.87500000e-01
1.63940430e-01 1.01852417e-02 4.87304688e-01 -4.87365723e-02 ……]
2.2
这个数据不同,也许是有误差,要看最终能否作为特征向量识别人脸。所以我用4张测试图,预处理后保存在images.npy,然后加载进来,images.shape=(4,160,160,3).
得出 Reshape_1节点的向量后,做 L2-norm,得到最终的embeddings。对比pb模型结果和rknn模型结果。
**pb模型正向推理出embeddings, 并计算这4个向量两两之间的距离:
Distance matrix
0 1 2 3
0 0.0000 0.8396 1.4851 1.4614
1 0.8396 0.0000 1.4103 1.3784
2 1.4851 1.4103 0.0000 0.4041
3 1.4614 1.3784 0.4041 0.0000
其中 0/1是同一个人的两张人脸图片,2/3另一个人的两张人脸图片。
可以看出,同一个人的不同照片之间的距离较小,不同人之间的人脸图片距离较大,能够正确识别。
**rknn模型正向推理出embeddings, 并计算这4个向量两两之间的距离:
Distance matrix
0 1 2 3
0 0.0000 0.8973 0.7701 0.7091
1 0.8973 0.0000 0.9278 1.0162
2 0.7701 0.9278 0.0000 0.5400
3 0.7091 1.0162 0.5400 0.0000
rknn模型结果就不能识别人脸了。
2.3
那么数据是从哪一层开始不同的呢?
我以 input节点作为输入, 经过一个 conv+relu 后的节点('InceptionResnetV1/Conv2d_1a_3x3/Relu')作为输出,比对pb模型和rknn模型数据。
在RK虚拟机中,由于需要提取中间层数据,所以重新build出新的rknn模型。
- #load tensorflow model
- print('--> Loading model')
- rknn.load_tensorflow(tf_pb='./20180402-114759.pb',
- inputs=['input'],
- outputs=['InceptionResnetV1/Conv2d_1a_3x3/Relu'],
- input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3]])
- print('done')
成功build, 模型保存在 facenet_Conv2d_1a_3x3_Relu.rknn
经过inference之后,得到输出数据,保存在 u_Conv2d_1a_3x3_Relu.npy
output.shape: (1, 1, 199712), 也就是说把(79,79,32)展开了。
pb模型也推理到节点('InceptionResnetV1/Conv2d_1a_3x3/Relu'),输出数据保存在Conv2d_1a_3x3_Relu.npy
output.shape:(1, 79, 79, 32)
对比数据
- import numpy as np
- pc_data = np.load('./results/Conv2d_1a_3x3_Relu.npy')
- u_data = np.load('./results/u_Conv2d_1a_3x3_Relu.npy')
- print('pc_data.shape:', pc_data.shape)
- print('u_data.shape:', u_data.shape)
- pc_data = np.squeeze(pc_data)
- u_data = np.squeeze(u_data)
- u_data_reshape = np.reshape(u_data, (32,79,79))
- print('pc_data.shape:', pc_data.shape)
- print('u_data_reshape.shape:', u_data_reshape.shape)
- print(pc_data[:,:,0])
- print(u_data_reshape[0,:,:])
输出:
pc_data.shape: (1, 79, 79, 32)
u_data.shape: (1, 1, 199712)
pc_data.shape: (79, 79, 32)
u_data_reshape.shape: (79, 79, 32)
Conv2d_1a_3x3_Relu.npy (pb模型推理的中间结果)
[[4.496504 3.5792322 3.279057 ... 4.1851587 4.199727 3.9860177]
[4.719177 3.80474 4.060274 ... 4.194205 4.0994143 4.004091 ]
[5.30498 4.7206473 4.2815747 ... 5.621602 5.3078156 4.4923596]
...
[2.477302 2.6923387 2.561545 ... 2.3765404 2.049178 1.906026 ]
[2.5384846 2.833478 2.7111642 ... 2.4625208 2.2596824 2.001416 ]
[2.5216506 3.0539947 3.2769814 ... 2.7133694 2.408282 2.0306435]]
u_Conv2d_1a_3x3_Relu.npy (rknn模型推理的中间结果)
[[ 93.375 400. 344.75 ... 270.75 261. 367.75 ]
[237.875 195. 222.125 ... 547. 174.25 55.28125]
[578.5 379.25 381.5 ... 368.25 264.75 209.875 ]
...
[347.25 249. 255.125 ... 276.75 266. 251.75 ]
[242.75 238.375 167. ... 357.5 308.25 290.5 ]
[ 75.375 286.25 340.25 ... 259.25 73.8125 292.5 ]]
从结果看,这两个数据不同,而且数量级都不同。
是否是输入数据有问题呢?
上文中 image_1.npy 是经过预处理的。
(1)原始图像image: [0,255],
(2)读入后为“RGB” . img = misc.imread(os.path.expanduser(image), mode='RGB')
(3)再经过归一化函数:
- def prewhiten(x):
- mean = np.mean(x)
- std = np.std(x)
- std_adj = np.maximum(std, 1.0/np.sqrt(x.size))
- y = np.multiply(np.subtract(x, mean), 1/std_adj)
- return y
rknn模型build的时候,并没有做任何处理( rknn.config(channel_mean_value='0 0 0 1', reorder_channel='0 1 2') )
那么image_1中的数据格式是 float64,是否可以直接作为rknn模型的输入呢?
在pc端运行pd模型的时候,也是输入的image_1.npy。两者输入是一样的,但rknn模型是否支持呢?会不会自己内部转换呢?
总结两个问题:
1. rknn模型在build的时候,input 、output 列表要怎么写?
参考tensorboard可视化出来的节点名字( "embeddings/(embeddings)" )为什么会出错?
2. rknn模型推理出来的数据和pd模型推理出来的数据不同,而且是经过第一个(conv+relu)之后数据就不同了。数量级都不同。
以上是哪里理解错了?
|
本帖子中包含更多资源
您需要 登录 才可以下载或查看,没有帐号?立即注册
x
|