Toybrick

标题: rk1808进行speech_comand语音识别时的问题 [打印本页]

作者: vkaoke 时间: 2019-10-14 09:12
标题: rk1808进行speech_comand语音识别时的问题
进行模型转换的代码：
#!/usr/bin/env python3
import numpy as np
import re
import math
import random
import cv2

from rknn.api import RKNN

if __name__ == '__main__':
# Create RKNN object
rknn = RKNN(verbose=False, verbose_file='./speech_command_build.log')

# Config for Model Input PreProcess
#rknn.config(quantized_dtype='dynamic_fixed_point-8')
#rknn.config(quantized_dtype='asymmetric_quantized-u8')
rknn.config()
# Load TensorFlow Model
print('--> Loading model')
rknn.load_tensorflow(tf_pb='./my_frozen_graph.pb',
                     inputs=['Reshape'],
                     outputs=['labels_softmax'],
                     input_size_list=[[1,3920]])
print('done')

# Build Model
print('--> Building model')
#rknn.build(do_quantization=False, dataset='./dataset.txt', pre_compile=False)
rknn.build(do_quantization=False)
print('done')

# Export RKNN Model
#rknn.export_rknn('./speech_command_quantized.rknn')
rknn.export_rknn('./speech_new.rknn')

语音识别的运行时代码：
#!/usr/bin/env python3

import numpy as np

import re
import math
import random

import tensorflow as tf

from rknn.api import RKNN
import soundfile as sf
import cv2
import tensorflow as tf
from tensorflow.contrib.framework.python.ops import audio_ops as contrib_audio
from tensorflow.python.framework import graph_util
def load_labels(filename):
  """Read in labels, one label per line."""
  return [line.rstrip() for line in tf.gfile.GFile(filename)]

if __name__ == '__main__':

# Create RKNN object
#rknn = RKNN(verbose=True, verbose_file='./speech_command_build.log')
rknn = RKNN()

# Load TensorFlow Model
print('--> Loading model')
ret = rknn.load_rknn(path='./speech_new.rknn')
if ret != 0:
print('load_rknn failed')
exit(ret)
print('load_rknn done')

print('--> Init runtime environment')
ret = rknn.init_runtime()
if ret != 0:
print('Init runtime environment failed')
exit(ret)
print('init_runtime done')
wav_file = open("on.wav", "rb")
wav_data = wav_file.read()
decoded_sample_data = contrib_audio.decode_wav(wav_data, desired_channels=1, desired_samples=16000, name='decoded_sample_data')
spectrogram = contrib_audio.audio_spectrogram(decoded_sample_data.audio, window_size=480, stride=160, magnitude_squared=True)
fingerprint_input = contrib_audio.mfcc(spectrogram, 16000,  dct_coefficient_count=40)
with tf.Session() as sess:
      fingerprint_input_npy = fingerprint_input.eval()
outputs, = rknn.inference(inputs=fingerprint_input_npy,data_type='float32')
print('inference done')
print('inference result: ', outputs)

labels = load_labels("./conv_labels.txt")
predictions = np.array(outputs)
top_k = predictions[0].argsort()[-3:][::-1]
for node_id in top_k:
            human_string = labels[node_id]
            score = predictions[0][node_id]
            print('%s (score = %.5f)' % (human_string, score))

发现的问题：
1.可以正常进行转换成rknn模型，但是转换后，用此模型进行语音识别，发现识别的是完全错误的,但是tensorflow进行识别时,人家结果是正确的。
[attach]634[/attach]
这里实在虚拟机上进行测试的，在rk1808的开发板上运行时，同样的代码和模型，每次运行的结果都不相同
[attach]635[/attach]

这是什么原因导致的呢？希望能给个思路

作者: vkaoke 时间: 2019-10-15 14:30

hisping 发表于 2019-10-14 14:51
帖子附件有更新，加了pb模型，我们内部测试ok；另外rk3399pro的麦克风只支持4段式的，建议你用苹果耳机试试 ...

用附件里的pb模型，识别还是一样，我这个板子是rk1808。我没录音，直接将wav文件进行特征提取后，存为npy格式的文件，导入接口的。

作者: vkaoke 时间: 2019-10-24 13:18

hisping 发表于 2019-10-16 09:15
你有执行test.py测试吗？这个会测试“yes”语音，看下结果对不对

结果不对

作者: ScottGreen 时间: 2020-6-23 11:30
同问，我这里使用提供的demo的精度也是不对的
~/speech/sc$ python3 test.py
--> Loading model
load_rknn done
--> Init runtime environment
init_runtime done
inference done
inference result: [[0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]

~/speech/sc$ python3 test.py
--> Loading model
load_rknn done
--> Init runtime environment
init_runtime done
inference done
inference result: [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

欢迎光临 Toybrick (https://t.rock-chips.com/)