Toybrick

标题: 人工智能开发系列(8) 表情识别 [打印本页]

作者: jax.fang    时间: 2019-6-21 09:19
标题: 人工智能开发系列(8) 表情识别
[attach]353[/attach]
                        
                                本教程视频直播回看:

[attach]354[/attach]
一、开发环境
1) 准备RK3399Pro开发板一块, USB键盘, USB鼠标,USB摄像头及HDMI显示器
2) 安装rknn-toolkit及依赖库, 按照wiki安装环境
3) 安装keras库
      pip3 install --user keras

二、资源地址:
原始代码:https://github.com/oarriaga/face_classification.git    本教程代码:  [attach]355[/attach]

三、快速上手
1)解压附件代码到开发板上, 进入rknn_test目录
2)python3 rknn_transfer.py  进行RKNN模型转换(先将训练好的HDF5模型文件转为Tensorflow模型pb文件, 再转成RKNN模型文件)
3)python3 rknn_image_demo.py test_images/1.jpg  图像表情识别
4)python3 rknn_video_demo.py 摄像头实时表情识别

四、概述
本示例代码表情识别基本流程:
加载RKNN模型 -> 图像输入 -> 人脸检测 -> 图像预处理 -> NPU模型推理 ->  输出分类概率 -> 识别结果
人脸检测采用OPENCV-PYTHON识别,用FER2013数据集训练深度卷积神经网络构建的模型, 准确度为66%, 表情分七类:0 angry;1 disgust; 2 fear;3 happy; 4 sad; 5 surprise; 6 neutral;

五、通过RKNN转换时注意点
1、rknn.config(reorder_channel='0, 1, 2')
① 由于输入数据在前处理阶段做归一化处理, 所以RKNN模型转换时不必设置channel_mean_value。
② reorder_channel设置为按输入的图像通道顺序不做调整。
③ 构建模型时不进行量化,所以quantized_dtype可不作设置。
2、rknn.load_tensorflow(tf_pb=pb_path, inputs=['input_1'], outputs=['predictions/Softmax'],
                         input_size_list=[[INPUT_WIDTH, INPUT_HEIGHT, 1]])
① inputs指定模型中的输入节点, outputs指定模型中的输出节点。
② input_size_list中注意输入数据为灰度图, 所以channel值应为1.
3、rknn.build(do_quantization=False)
① 不对模型进行量化。
4、 rknn.inference(inputs=[gray_face], data_type='float32', data_format='nhwc')
① 输入数据在预处理后已是float32类型, 所以推理阶段要求传入的数据类型必须保持一致。
② 输入数据的data_format为默认值‘nhwc’

六、代码解析
rknn_transfer.py
  1. from rknn.api import RKNN

  2. from keras.models import load_model
  3. import tensorflow as tf
  4. import os
  5. import os.path as osp
  6. from keras import backend as K

  7. model_name = 'fer2013_mini_XCEPTION.102-0.66'
  8. #model_name = 'ck_mini_XCEPTION.95-0.89'
  9. #model_name = 'ck_mini_XCEPTION.98-0.90'

  10. def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
  11.     """
  12.     Freezes the state of a session into a prunned computation graph.

  13.     Creates a new computation graph where variable nodes are replaced by
  14.     constants taking their current value in the session. The new graph will be
  15.     prunned so subgraphs that are not neccesary to compute the requested
  16.     outputs are removed.
  17.     @param session The TensorFlow session to be frozen.
  18.     @param keep_var_names A list of variable names that should not be frozen,
  19.                           or None to freeze all the variables in the graph.
  20.     @param output_names Names of the relevant graph outputs.
  21.     @param clear_devices Remove the device directives from the graph for better portability.
  22.     @return The frozen graph definition.
  23.     """
  24.     from tensorflow.python.framework.graph_util import convert_variables_to_constants
  25.     graph = session.graph
  26.     with graph.as_default():
  27.         freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))
  28.         output_names = output_names or []
  29.         output_names += [v.op.name for v in tf.global_variables()]
  30.         input_graph_def = graph.as_graph_def()
  31.         if clear_devices:
  32.             for node in input_graph_def.node:
  33.                 node.device = ""
  34.         frozen_graph = convert_variables_to_constants(session, input_graph_def,
  35.                                                       output_names, freeze_var_names)
  36.         return frozen_graph

  37. def pb_transfer():
  38.     #input_fld = sys.path[0]
  39.     input_fld = './trained_models/emotion_models/'
  40.     weight_file = model_name + '.hdf5'

  41.     output_fld = './'
  42.     output_graph_name = model_name + '.pb'

  43.     if not os.path.isdir(output_fld):
  44.         os.mkdir(output_fld)
  45.     weight_file_path = osp.join(input_fld, weight_file)

  46.     K.set_learning_phase(0)
  47.     net_model = load_model(weight_file_path)

  48.     print('input is :', net_model.input.name)
  49.     print ('output is:', net_model.output.name)

  50.     sess = K.get_session()

  51.     frozen_graph = freeze_session(K.get_session(), output_names=[net_model.output.op.name])

  52.     from tensorflow.python.framework import graph_io

  53.     graph_io.write_graph(frozen_graph, output_fld, output_graph_name, as_text=False)

  54.     print('saved the constant graph (ready for inference) at: ', osp.join(output_fld, output_graph_name))


  55. INPUT_WIDTH = 64
  56. INPUT_HEIGHT = 64

  57. def transfer(pb_path, rknn_name):
  58.     # 创建RKNN执行对象
  59.     #rknn = RKNN(verbose=True, verbose_file='./mini_XCEPTION_build.log')
  60.     rknn = RKNN()
  61. # 配置模型输入,用于NPU对数据输入的预处理
  62. # channel_mean_value='0 0 0 255',那么模型推理时,将会对RGB数据做如下转换
  63. # (R - 0)/255, (G - 0)/255, (B - 0)/255。推理时,RKNN模型会自动做均值和归一化处理
  64. # reorder_channel=’0 1 2’用于指定是否调整图像通道顺序,设置成0 1 2即按输入的图像通道顺序不做调整
  65. # reorder_channel=’2 1 0’表示交换0和2通道,如果输入是RGB,将会被调整为BGR。如果是BGR将会被调整为RGB
  66. #图像通道顺序不做调整
  67.     #rknn.config(channel_mean_value='0 0 0 255', reorder_channel='0 1 2')
  68.     rknn.config(quantized_dtype='dynamic_fixed_point-8')

  69. # 加载TensorFlow模型
  70. # tf_pb='digital_gesture.pb'指定待转换的TensorFlow模型
  71. # inputs指定模型中的输入节点
  72. # outputs指定模型中输出节点
  73. # input_size_list指定模型输入的大小
  74.     print('--> Loading model')
  75.     ret = rknn.load_tensorflow(tf_pb=pb_path,
  76.                          inputs=['input_1'],
  77.                          outputs=['predictions/Softmax'],
  78.                          input_size_list=[[INPUT_WIDTH, INPUT_HEIGHT, 1]])
  79.     if ret != 0:
  80.         print('Load Model failed!')
  81.         exit(ret)
  82.     print('done')

  83. # 创建解析pb模型
  84. # do_quantization=False指定不进行量化
  85. # 量化会减小模型的体积和提升运算速度,但是会有精度的丢失
  86.     print('--> Building model')
  87.     ret = rknn.build(do_quantization=False)
  88.     if ret != 0:
  89.         print('Build Model failed!')
  90.         exit(ret)
  91.     print('done')

  92.     # 导出保存rknn模型文件
  93.     print('--> Export RKNN model')
  94.     ret = rknn.export_rknn(rknn_name)
  95.     if ret != 0:
  96.         print('Export Model failed!')
  97.         exit(ret)
  98.     print('done')

  99.     # Release RKNN Context
  100.     rknn.release()

  101. if __name__ == '__main__':
  102.     pb_transfer()
  103.     transfer(model_name + '.pb', model_name + '.rknn')
复制代码
rknn_image_demo.py
  1. import numpy as np
  2. import cv2
  3. from rknn.api import RKNN
  4. from time import time
  5. #from utils.builddata import preprocess_input
  6. import sys, getopt
  7. import sys

  8. from utils.inference import detect_faces
  9. from utils.inference import draw_text
  10. from utils.inference import draw_bounding_box
  11. from utils.inference import apply_offsets
  12. from utils.inference import load_detection_model
  13. from utils.inference import load_image
  14. from utils.preprocessor import preprocess_input

  15. # parameters for loading data and images
  16. detection_model_path = './trained_models/detection_models/haarcascade_frontalface_default.xml'

  17. def get_labels(dataset_name):
  18.     if dataset_name == 'fer2013':
  19.         return {0: 'angry', 1: 'disgust', 2: 'fear', 3: 'happy',
  20.                 4: 'sad', 5: 'surprise', 6: 'neutral'}

  21. emotion_labels = get_labels('fer2013')
  22. font = cv2.FONT_HERSHEY_SIMPLEX

  23. INPUT_SIZE = 64

  24. def load_model(modle_path):
  25.         # Create RKNN object
  26.         rknn = RKNN()

  27.         print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
  28.         print('-->loading model')
  29.         rknn.load_rknn(modle_path)
  30.         print('loading model done')

  31.         # init runtime environment
  32.         print('--> Init runtime environment')
  33.         ret = rknn.init_runtime()
  34.         if ret != 0:
  35.                 print('Init runtime environment failed')
  36.                 exit(ret)
  37.         print('done')
  38.         print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
  39.         return rknn

  40. def model_predict(rknn, gray_face):
  41.         #指定输入数据格式, 进行模型推理, 并返回推理结果, 结果为7个表情标签的概率值
  42.     emotion_prediction = rknn.inference(inputs=[gray_face], data_type='float32', data_format='nhwc')
  43.         #根据表情分类预测概率,识别表情
  44.     emotion_probability = np.max(emotion_prediction)
  45.     emotion_label_arg = np.argmax(emotion_prediction)
  46.     emotion_text = emotion_labels[emotion_label_arg]
  47.     return emotion_text


  48. if __name__ == '__main__':
  49.     image_path = sys.argv[1]

  50.     # hyper-parameters for bounding boxes shape
  51.     emotion_offsets = (0, 0)

  52.     # loading models
  53.     rknn = load_model('./fer2013_mini_XCEPTION.102-0.66.rknn')
  54.         # 加载人脸检测模型
  55.     face_detection = load_detection_model(detection_model_path)

  56.     # 设置模型输入图片像素大小, 这里为64x64
  57.     emotion_target_size = (INPUT_SIZE, INPUT_SIZE)

  58.     # 加载图片, 并转为灰度图
  59.     rgb_image = load_image(image_path, grayscale=False)
  60.     gray_image = load_image(image_path, grayscale=True)
  61.     gray_image = np.squeeze(gray_image)
  62.     gray_image = gray_image.astype('uint8')

  63.     start = time()
  64.         # 检测图片中人脸个数, 并返回人脸数据集
  65.     faces = detect_faces(face_detection, gray_image)
  66.     print('detect face cost time: %f'%(time()-start))

  67.     for face_coordinates in faces:
  68.         start = time()
  69.                 #从所有人脸数据集中截取单个人脸数据
  70.         x1, x2, y1, y2 = apply_offsets(face_coordinates, emotion_offsets)
  71.         gray_face = gray_image[y1:y2, x1:x2]

  72.         try:
  73.                         #将截取的人脸数据设置为64x64像素
  74.             gray_face = cv2.resize(gray_face, (emotion_target_size))
  75.         except:
  76.             continue

  77.                 #人脸数据归一化处理, 并转化为RKNN模型要求的输入数据格式
  78.         gray_face = preprocess_input(gray_face, True)
  79.         gray_face = np.expand_dims(gray_face, 0)
  80.         gray_face = np.expand_dims(gray_face, -1)

  81.         emotion_text = model_predict(rknn, gray_face)
  82.         print('predict emotion cost time: %f'%(time()-start))
  83.         print('predict emotion result: ' + emotion_text)

  84.         color = (255, 0, 0)
  85.         draw_bounding_box(face_coordinates, rgb_image, color)
  86.         draw_text(face_coordinates, rgb_image, emotion_text, color, 0, 0, 1, 2)

  87.     bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2BGR)
  88.     cv2.imwrite('./predicted_test_image_rknn.png', bgr_image)
复制代码


作者: TryYourBest    时间: 2019-7-1 14:32
我通过rknn_transfer.py文件在开发板上转换simple_CNN.81-0.96.pb文件为rknn过程中提示模型加载失败,这是为什么,该如何解决,请指教,谢谢!
错误内容如下:

  1. --> Loading model
  2. E Catch exception when loading tensorflow model: simple_CNN.81-0.96.pb!
  3. T Traceback (most recent call last):
  4. T   File "rknn/api/rknn_base.py", line 136, in rknn.api.rknn_base.RKNNBase.load_tensorflow
  5. T   File "rknn/base/RKNNlib/converter/convert_tf.py", line 102, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.__init__
  6. T   File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 53, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.__init__
  7. T AttributeError: 'NoneType' object has no attribute 'op'
  8. Load Model failed!
复制代码

作者: jax.fang    时间: 2019-7-4 09:51
TryYourBest 发表于 2019-7-1 14:32
我通过rknn_transfer.py文件在开发板上转换simple_CNN.81-0.96.pb文件为rknn过程中提示模型加载失败,这是 ...

可以将rknn=RKNN()改成rknn=RKNN(verbose=True)看下详细的日志信息, 可以看到是哪个OP不支持

作者: TryYourBest    时间: 2019-7-4 10:15
本帖最后由 TryYourBest 于 2019-7-4 10:16 编辑
jax.fang 发表于 2019-7-4 09:51
可以将rknn=RKNN()改成rknn=RKNN(verbose=True)看下详细的日志信息, 可以看到是哪个OP不支持
...

将rknn = RKNN()改成rknn = RKNN(verbose = True)后,提示如下错误,多了第一行的D import~,由于知识尚浅,不明白啥意思

  1. --> Loading model
  2. D import clients finished
  3. E Catch exception when loading tensorflow model: simple_CNN.81-0.96.pb!
  4. T Traceback (most recent call last):
  5. T   File "rknn/api/rknn_base.py", line 136, in rknn.api.rknn_base.RKNNBase.load_tensorflow
  6. T   File "rknn/base/RKNNlib/converter/convert_tf.py", line 102, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.__init__
  7. T   File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 53, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.__init__
  8. T AttributeError: 'NoneType' object has no attribute 'op'
  9. Load Model failed!
复制代码

作者: jax.fang    时间: 2019-7-4 16:18
TryYourBest 发表于 2019-7-4 10:15
将rknn = RKNN()改成rknn = RKNN(verbose = True)后,提示如下错误,多了第一行的D import~,由于知识 ...

TensorFlow版本是多少
作者: TryYourBest    时间: 2019-7-4 16:45
jax.fang 发表于 2019-7-4 16:18
TensorFlow版本是多少

TensorFlow的版本是1.10.1的
作者: jax.fang    时间: 2019-7-4 19:39
TryYourBest 发表于 2019-7-4 16:45
TensorFlow的版本是1.10.1的

这个模型用TensorFlow能加载吗?从报错的信息来看,TensorFlow加载这个模型就失败了。这个模型好像是用比较新的版本的TensorFlow导出的吧
作者: TryYourBest    时间: 2019-7-5 09:56
jax.fang 发表于 2019-7-4 19:39
这个模型用TensorFlow能加载吗?从报错的信息来看,TensorFlow加载这个模型就失败了。这个模型好像是用比 ...

我PC用的是1.13.1的,我尝试下1.10版本的,支持开发板的tensorflow还有比1.10.1更高版本的么,网上找了找没找到
作者: allforgot    时间: 2019-7-23 14:17
请问有人将这个项目移植到安卓上吗
作者: 小宁哥    时间: 2019-12-9 10:46
在image_demo里load_image 报错,invalid interpolation method,我改为opencv 读图后,结果图为空,是怎么回事
作者: 小宁哥    时间: 2019-12-9 11:07
小宁哥 发表于 2019-12-9 10:46
在image_demo里load_image 报错,invalid interpolation method,我改为opencv 读图后,结果图为空,是怎么 ...

后面两张图 可以正确识别
作者: liuwenhua    时间: 2019-12-12 10:21
小宁哥 发表于 2019-12-9 10:46
在image_demo里load_image 报错,invalid interpolation method,我改为opencv 读图后,结果图为空,是怎么 ...

降低OpenCV版本试试
作者: wzp    时间: 2020-4-20 20:11
小白初次接触TB-RK3399Pro开发板,运行上边的表情识别案例时遇到以下问题:
1、运行python3 rknn_image_demo.py test_images/1.jpg  时,出现以下提示:
File “rknn_image_demo.py”, line 59, in <module>
image_path = sys.argv[1]
IndexError: list index out of range

2、开发板运行python3 rknn_video_demo.py 可以在正常跑通,但是在电脑虚拟机的Ubuntu上ssh远程开发板运行python3 rknn_video_demo.py 出现以下提示:
qt.qpa.screen: QXcbConnection: Could not connect to display
Could not connect to any X display.
求大神解答。
作者: NIELSONGOD    时间: 2020-5-27 14:40
小宁哥 发表于 2019-12-9 10:46
在image_demo里load_image 报错,invalid interpolation method,我改为opencv 读图后,结果图为空,是怎么 ...

同样的问题,请问你解决了么?求指导
作者: jax.fang    时间: 2020-6-1 16:15
wzp 发表于 2020-4-20 20:11
小白初次接触TB-RK3399Pro开发板,运行上边的表情识别案例时遇到以下问题:
1、运行python3 rknn_image_dem ...

1. rknn_image_demo.py 后面需要加一个图片路径的参数
2. SSH远程运行需安装支持Xserver的客户端软件
作者: jax.fang    时间: 2020-6-1 16:37
NIELSONGOD 发表于 2020-5-27 14:40
同样的问题,请问你解决了么?求指导

可能是keras的版本不对导致无效接口定义, 可试试降低TensorFlow的版本?
作者: 李雪梅    时间: 2020-11-3 17:20
tensorflow版本是1.11.0版本,执行python3 rknn_transfer.py后总是提醒:ModuleNotFoundError: No module named 'tensorflow.keras.layers.experimental',怎么解决?系统提醒安装tensorflow2.2版本的,但安装不了,怎么解决?
作者: ccb1129086912    时间: 2020-11-16 10:03
李雪梅 发表于 2020-11-3 17:20
tensorflow版本是1.11.0版本,执行python3 rknn_transfer.py后总是提醒:ModuleNotFoundError: No module n ...

我也是,请问你解决了吗

作者: qingtian123    时间: 2022-6-13 16:22
运行python3 rknn_transfer.py报错如下( python3==3.6.9 , tensorflow==1.14.0, keras==2.2.5), 帮忙看下什么原因

W:tensorflow:From /home/xx/workspace/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

Traceback (most recent call last):
  File "rknn_transfer.py", line 124, in <module>
    pb_transfer()
  File "rknn_transfer.py", line 55, in pb_transfer
    net_model = load_model(weight_file_path)
  File "/home/xx/workspace/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in load_wrapper
    return load_function(*args, **kwargs)
  File "/home/xx/workspace/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 550, in load_model
    model = _deserialize_model(h5dict, custom_objects, compile)
  File "/home/xx/workspace/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 242, in _deserialize_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
作者: ziyifly    时间: 2022-10-20 20:44
你好,请问TensorFlow的版本和keras的版本分别应该多少呢? 既然是官方推荐的帖子,希望能描述详细一些帮到我们
我现在用的是官方1.4.0的docker,rknn-toolkit2-1.40,TensorFlow和Keras 两个版本都是 2.6.2 ,但转换模型时提示:

  1. root@2c548c52c948:/rknn_test# python3 rknn_transfer.py
  2. 2022-10-20 20:42:44.727550: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
  3. 2022-10-20 20:42:44.727594: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  4. /usr/local/lib/python3.8/dist-packages/keras/backend.py:401: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
  5.   warnings.warn('`tf.keras.backend.set_learning_phase` is deprecated and '
  6. 2022-10-20 20:42:45.534988: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
  7. 2022-10-20 20:42:45.535040: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
  8. 2022-10-20 20:42:45.535060: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2c548c52c948): /proc/driver/nvidia/version does not exist
  9. 2022-10-20 20:42:45.535192: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
  10. To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  11. W:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
  12. input is : input_1
  13. output is: predictions/Softmax:0
  14. Traceback (most recent call last):
  15.   File "rknn_transfer.py", line 124, in <module>
  16.     pb_transfer()
  17.   File "rknn_transfer.py", line 62, in pb_transfer
  18.     frozen_graph = freeze_session(K.get_session(), output_names=[net_model.output.op.name])
  19.   File "/usr/local/lib/python3.8/dist-packages/keras/engine/keras_tensor.py", line 222, in op
  20.     raise TypeError('Keras symbolic inputs/outputs do not '
  21. TypeError: Keras symbolic inputs/outputs do not implement `op`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
  22. root@2c548c52c948:/rknn_test#
复制代码






欢迎光临 Toybrick (https://t.rock-chips.com/) Powered by Discuz! X3.3