人工智能开发系列(8) 表情识别 - Toybrick

[attach]353[/attach]

本教程视频直播回看：

一、开发环境
1）准备RK3399Pro开发板一块， USB键盘， USB鼠标，USB摄像头及HDMI显示器
2）安装rknn-toolkit及依赖库，按照wiki安装环境
3）安装keras库

pip3 install --user keras

二、资源地址：
原始代码：https://github.com/oarriaga/face_classification.git 本教程代码: [attach]355[/attach]

1）解压附件代码到开发板上, 进入rknn_test目录

2）python3 rknn_transfer.py 进行RKNN模型转换(先将训练好的HDF5模型文件转为Tensorflow模型pb文件，再转成RKNN模型文件)

3）python3 rknn_image_demo.py test_images/1.jpg 图像表情识别

4）python3 rknn_video_demo.py 摄像头实时表情识别

本示例代码表情识别基本流程：

加载RKNN模型 -> 图像输入 -> 人脸检测 -> 图像预处理 -> NPU模型推理 -> 输出分类概率 -> 识别结果

人脸检测采用OPENCV-PYTHON识别，用FER2013数据集训练深度卷积神经网络构建的模型，准确度为66%，表情分七类：0 angry；1 disgust; 2 fear；3 happy; 4 sad; 5 surprise; 6 neutral;

五、通过RKNN转换时注意点

1、rknn.config(reorder_channel='0, 1, 2')

① 由于输入数据在前处理阶段做归一化处理，所以RKNN模型转换时不必设置channel_mean_value。

② reorder_channel设置为按输入的图像通道顺序不做调整。

③ 构建模型时不进行量化，所以quantized_dtype可不作设置。

2、rknn.load_tensorflow(tf_pb=pb_path, inputs=['input_1'], outputs=['predictions/Softmax'],

input_size_list=[[INPUT_WIDTH, INPUT_HEIGHT, 1]])

① inputs指定模型中的输入节点， outputs指定模型中的输出节点。

② input_size_list中注意输入数据为灰度图，所以channel值应为1.

3、rknn.build(do_quantization=False)

① 不对模型进行量化。

4、 rknn.inference(inputs=[gray_face], data_type='float32', data_format='nhwc')

① 输入数据在预处理后已是float32类型，所以推理阶段要求传入的数据类型必须保持一致。

② 输入数据的data_format为默认值‘nhwc’

from rknn.api import RKNN



from keras.models import load_model

import tensorflow as tf

import os

import os.path as osp

from keras import backend as K



model_name = 'fer2013_mini_XCEPTION.102-0.66'

#model_name = 'ck_mini_XCEPTION.95-0.89'

#model_name = 'ck_mini_XCEPTION.98-0.90'



def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):

    """

    Freezes the state of a session into a prunned computation graph.



    Creates a new computation graph where variable nodes are replaced by

    constants taking their current value in the session. The new graph will be

    prunned so subgraphs that are not neccesary to compute the requested

    outputs are removed.

    @param session The TensorFlow session to be frozen.

    @param keep_var_names A list of variable names that should not be frozen,

                          or None to freeze all the variables in the graph.

    @param output_names Names of the relevant graph outputs.

    @param clear_devices Remove the device directives from the graph for better portability.

    @return The frozen graph definition.

    """

    from tensorflow.python.framework.graph_util import convert_variables_to_constants

    graph = session.graph

    with graph.as_default():

        freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))

        output_names = output_names or []

        output_names += [v.op.name for v in tf.global_variables()]

        input_graph_def = graph.as_graph_def()

        if clear_devices:

            for node in input_graph_def.node:

                node.device = ""

        frozen_graph = convert_variables_to_constants(session, input_graph_def,

                                                      output_names, freeze_var_names)

        return frozen_graph



def pb_transfer():

    #input_fld = sys.path[0]

    input_fld = './trained_models/emotion_models/'

    weight_file = model_name + '.hdf5'



    output_fld = './'

    output_graph_name = model_name + '.pb'



    if not os.path.isdir(output_fld):

        os.mkdir(output_fld)

    weight_file_path = osp.join(input_fld, weight_file)



    K.set_learning_phase(0)

    net_model = load_model(weight_file_path)



    print('input is :', net_model.input.name)

    print ('output is:', net_model.output.name)



    sess = K.get_session()



    frozen_graph = freeze_session(K.get_session(), output_names=[net_model.output.op.name])



    from tensorflow.python.framework import graph_io



    graph_io.write_graph(frozen_graph, output_fld, output_graph_name, as_text=False)



    print('saved the constant graph (ready for inference) at: ', osp.join(output_fld, output_graph_name))





INPUT_WIDTH = 64

INPUT_HEIGHT = 64



def transfer(pb_path, rknn_name):

    # 创建RKNN执行对象

    #rknn = RKNN(verbose=True, verbose_file='./mini_XCEPTION_build.log')

    rknn = RKNN()

# 配置模型输入，用于NPU对数据输入的预处理

# channel_mean_value='0 0 0 255'，那么模型推理时，将会对RGB数据做如下转换

# (R - 0)/255, (G - 0)/255, (B - 0)/255。推理时，RKNN模型会自动做均值和归一化处理

# reorder_channel=’0 1 2’用于指定是否调整图像通道顺序，设置成0 1 2即按输入的图像通道顺序不做调整

# reorder_channel=’2 1 0’表示交换0和2通道，如果输入是RGB，将会被调整为BGR。如果是BGR将会被调整为RGB

#图像通道顺序不做调整

    #rknn.config(channel_mean_value='0 0 0 255', reorder_channel='0 1 2')

    rknn.config(quantized_dtype='dynamic_fixed_point-8')

 

# 加载TensorFlow模型

# tf_pb='digital_gesture.pb'指定待转换的TensorFlow模型

# inputs指定模型中的输入节点

# outputs指定模型中输出节点

# input_size_list指定模型输入的大小

    print('--> Loading model')

    ret = rknn.load_tensorflow(tf_pb=pb_path,

                         inputs=['input_1'],

                         outputs=['predictions/Softmax'],

                         input_size_list=[[INPUT_WIDTH, INPUT_HEIGHT, 1]])

    if ret != 0:

        print('Load Model failed!')

        exit(ret)

    print('done')

 

# 创建解析pb模型

# do_quantization=False指定不进行量化

# 量化会减小模型的体积和提升运算速度，但是会有精度的丢失

    print('--> Building model')

    ret = rknn.build(do_quantization=False)

    if ret != 0:

        print('Build Model failed!')

        exit(ret)

    print('done')

 

    # 导出保存rknn模型文件

    print('--> Export RKNN model')

    ret = rknn.export_rknn(rknn_name)

    if ret != 0:

        print('Export Model failed!')

        exit(ret)

    print('done')

 

    # Release RKNN Context

    rknn.release()



if __name__ == '__main__':

    pb_transfer()

    transfer(model_name + '.pb', model_name + '.rknn')

复制代码

rknn_image_demo.py

import numpy as np

import cv2

from rknn.api import RKNN

from time import time

#from utils.builddata import preprocess_input

import sys, getopt

import sys



from utils.inference import detect_faces

from utils.inference import draw_text

from utils.inference import draw_bounding_box

from utils.inference import apply_offsets

from utils.inference import load_detection_model

from utils.inference import load_image

from utils.preprocessor import preprocess_input



# parameters for loading data and images

detection_model_path = './trained_models/detection_models/haarcascade_frontalface_default.xml'



def get_labels(dataset_name):

    if dataset_name == 'fer2013':

        return {0: 'angry', 1: 'disgust', 2: 'fear', 3: 'happy',

                4: 'sad', 5: 'surprise', 6: 'neutral'}



emotion_labels = get_labels('fer2013')

font = cv2.FONT_HERSHEY_SIMPLEX



INPUT_SIZE = 64



def load_model(modle_path):

        # Create RKNN object

        rknn = RKNN()



        print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

        print('-->loading model')

        rknn.load_rknn(modle_path)

        print('loading model done')



        # init runtime environment

        print('--> Init runtime environment')

        ret = rknn.init_runtime()

        if ret != 0:

                print('Init runtime environment failed')

                exit(ret)

        print('done')

        print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

        return rknn



def model_predict(rknn, gray_face):

        #指定输入数据格式， 进行模型推理， 并返回推理结果， 结果为7个表情标签的概率值

    emotion_prediction = rknn.inference(inputs=[gray_face], data_type='float32', data_format='nhwc')

        #根据表情分类预测概率，识别表情 

    emotion_probability = np.max(emotion_prediction)

    emotion_label_arg = np.argmax(emotion_prediction)

    emotion_text = emotion_labels[emotion_label_arg]

    return emotion_text





if __name__ == '__main__':

    image_path = sys.argv[1]



    # hyper-parameters for bounding boxes shape

    emotion_offsets = (0, 0)



    # loading models

    rknn = load_model('./fer2013_mini_XCEPTION.102-0.66.rknn')

        # 加载人脸检测模型

    face_detection = load_detection_model(detection_model_path)



    # 设置模型输入图片像素大小， 这里为64x64

    emotion_target_size = (INPUT_SIZE, INPUT_SIZE)



    # 加载图片， 并转为灰度图

    rgb_image = load_image(image_path, grayscale=False)

    gray_image = load_image(image_path, grayscale=True)

    gray_image = np.squeeze(gray_image)

    gray_image = gray_image.astype('uint8')



    start = time()

        # 检测图片中人脸个数， 并返回人脸数据集

    faces = detect_faces(face_detection, gray_image)

    print('detect face cost time: %f'%(time()-start))



    for face_coordinates in faces:

        start = time()

                #从所有人脸数据集中截取单个人脸数据

        x1, x2, y1, y2 = apply_offsets(face_coordinates, emotion_offsets)

        gray_face = gray_image[y1:y2, x1:x2]



        try:

                        #将截取的人脸数据设置为64x64像素

            gray_face = cv2.resize(gray_face, (emotion_target_size))

        except:

            continue



                #人脸数据归一化处理， 并转化为RKNN模型要求的输入数据格式

        gray_face = preprocess_input(gray_face, True)

        gray_face = np.expand_dims(gray_face, 0)

        gray_face = np.expand_dims(gray_face, -1)



        emotion_text = model_predict(rknn, gray_face)

        print('predict emotion cost time: %f'%(time()-start))

        print('predict emotion result: ' + emotion_text)



        color = (255, 0, 0)

        draw_bounding_box(face_coordinates, rgb_image, color)

        draw_text(face_coordinates, rgb_image, emotion_text, color, 0, 0, 1, 2)



    bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2BGR)

    cv2.imwrite('./predicted_test_image_rknn.png', bgr_image)
复制代码

--> Loading model
E Catch exception when loading tensorflow model: simple_CNN.81-0.96.pb!
T Traceback (most recent call last):
T File "rknn/api/rknn_base.py", line 136, in rknn.api.rknn_base.RKNNBase.load_tensorflow
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 102, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.__init__
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 53, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.__init__
T AttributeError: 'NoneType' object has no attribute 'op'
Load Model failed!

--> Loading model
D import clients finished
E Catch exception when loading tensorflow model: simple_CNN.81-0.96.pb!
T Traceback (most recent call last):
T File "rknn/api/rknn_base.py", line 136, in rknn.api.rknn_base.RKNNBase.load_tensorflow
T File "rknn/base/RKNNlib/converter/convert_tf.py", line 102, in rknn.base.RKNNlib.converter.convert_tf.convert_tf.__init__
T File "rknn/base/RKNNlib/converter/tensorflowloader.py", line 53, in rknn.base.RKNNlib.converter.tensorflowloader.TF_Graph_Preprocess.__init__
T AttributeError: 'NoneType' object has no attribute 'op'
Load Model failed!

root@2c548c52c948:/rknn_test# python3 rknn_transfer.py
2022-10-20 20:42:44.727550: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-10-20 20:42:44.727594: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/usr/local/lib/python3.8/dist-packages/keras/backend.py:401: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
warnings.warn('`tf.keras.backend.set_learning_phase` is deprecated and '
2022-10-20 20:42:45.534988: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-10-20 20:42:45.535040: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-10-20 20:42:45.535060: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2c548c52c948): /proc/driver/nvidia/version does not exist
2022-10-20 20:42:45.535192: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
W:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
input is : input_1
output is: predictions/Softmax:0
Traceback (most recent call last):
File "rknn_transfer.py", line 124, in <module>
pb_transfer()
File "rknn_transfer.py", line 62, in pb_transfer
frozen_graph = freeze_session(K.get_session(), output_names=[net_model.output.op.name])
File "/usr/local/lib/python3.8/dist-packages/keras/engine/keras_tensor.py", line 222, in op
raise TypeError('Keras symbolic inputs/outputs do not '
TypeError: Keras symbolic inputs/outputs do not implement `op`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
root@2c548c52c948:/rknn_test#