Toybrick

TB-RK3399ProD 3G(2G+1G)转换模型后推理报找不到NPU

ooo

注册会员

积分
158
发表于 2022-7-31 16:40:34    查看: 9300|回复: 26 | [复制链接]    打印 | 显示全部楼层
以下是整个运行流程(请官方技术解答一下):

toybrick@debian10:~/tensorflow/ssd_mobilenet_v1$ python3 test.py--> Config modeldone--> Loading modelW:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/rknn/api/rknn.py:75: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.Instructions for updating:Use `tf.compat.v1.graph_util.extract_sub_graph`W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !W Network may not support batch > 1 !/home/toybrick/.local/lib/python3.7/site-packages/rknn/api/rknn.py:75: FutureWarning: Possible nested set at position 7  std_values=std_values, size_with_batch=size_with_batch)done--> Building modelW The target_platform is not set in config, using default target platform rk1808.W:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py:915: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.Instructions for updating:tf.py_func is deprecated in TF V2. Instead, there are two    options available in V2.    - tf.py_function takes a python function which manipulates tf eager    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to    an ndarray (just call tensor.numpy()) but having access to eager tensors    means `tf.py_function`s can use accelerators such as GPUs as well as    being differentiable using a gradient tape.    - tf.numpy_function maintains the semantics of the deprecated tf.py_func    (it is not differentiable, and manipulates numpy arrays). It drops the    stateful argument making all functions stateful.    W:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.Instructions for updating:If using Keras pass *_constraint arguments to layers.done--> Export RKNN modeldone--> Init runtime environmentE Only support ntb mode on ARM64 platform. But can not find device with ntb mode.E Catch exception when init runtime!E Traceback (most recent call last):E   File "rknn/api/rknn_base.py", line 815, in rknn.api.rknn_base.RKNNBase.init_runtimeE   File "rknn/api/rknn_runtime.py", line 170, in rknn.api.rknn_runtime.RKNNRuntime.__init__E   File "rknn/api/rknn_platform_utils.py", line 307, in rknn.api.rknn_platform_utils.start_ntp_or_adbE Exception: Init runtime environment failed!Init runtime environment failed

回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 16:44:26 | 显示全部楼层
本帖最后由 ooo 于 2022-7-31 16:46 编辑

上面格式不行,下面重新上传
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 16:45:46 | 显示全部楼层
toybrick@debian10:~/tensorflow/ssd_mobilenet_v1$ python3 test.py
--> Config model
done
--> Loading model
W:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/rknn/api/rknn.py:75: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
W Network may not support batch > 1 !
/home/toybrick/.local/lib/python3.7/site-packages/rknn/api/rknn.py:75: FutureWarning: Possible nested set at position 7
  std_values=std_values, size_with_batch=size_with_batch)
done
--> Building model
W The target_platform is not set in config, using default target platform rk1808.
W:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py:915: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
   
W:tensorflow:From /home/toybrick/.local/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
done
--> Export RKNN model
done
--> Init runtime environment
E Only support ntb mode on ARM64 platform. But can not find device with ntb mode.
E Catch exception when init runtime!
E Traceback (most recent call last):
E   File "rknn/api/rknn_base.py", line 815, in rknn.api.rknn_base.RKNNBase.init_runtime
E   File "rknn/api/rknn_runtime.py", line 170, in rknn.api.rknn_runtime.RKNNRuntime.__init__
E   File "rknn/api/rknn_platform_utils.py", line 307, in rknn.api.rknn_platform_utils.start_ntp_or_adb
E Exception: Init runtime environment failed!
Init runtime environment failed
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 16:50:20 | 显示全部楼层
补充:试过烧录当前固件下载页面的Linux固件(TB-RK3399ProD: 百度网盘 OneDrive),还是如此
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 16:53:10 | 显示全部楼层
补充:Android8.1下跑RK的例程,没有检测框(估计是没用到npu),官方技术解答一下
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 17:21:18 | 显示全部楼层
请官方技术解答一下,在线等
回复

使用道具 举报

jefferyzhang

版主

积分
13093
发表于 2022-7-31 23:05:53 | 显示全部楼层
看过去是Init runtime时候target写错了吧
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 23:09:54 | 显示全部楼层
import numpy as np

import re
import math
import random
import cv2

from rknn.api import RKNN

INPUT_SIZE = 300

NUM_RESULTS = 1917
NUM_CLASSES = 2

Y_SCALE = 10.0
X_SCALE = 10.0
H_SCALE = 5.0
W_SCALE = 5.0


def expit(x):
    return 1. / (1. + math.exp(-x))


def unexpit(y):
    return -1.0 * math.log((1.0 / y) - 1.0);


def CalculateOverlap(xmin0, ymin0, xmax0, ymax0, xmin1, ymin1, xmax1, ymax1):
    w = max(0.0, min(xmax0, xmax1) - max(xmin0, xmin1))
    h = max(0.0, min(ymax0, ymax1) - max(ymin0, ymin1))
    i = w * h
    u = (xmax0 - xmin0) * (ymax0 - ymin0) + (xmax1 - xmin1) * (ymax1 - ymin1) - i

    if u <= 0.0:
        return 0.0

    return i / u


def load_box_priors():
    box_priors_ = []
    fp = open('./box_priors.txt', 'r')
    ls = fp.readlines()
    for s in ls:
        aList = re.findall('([-+]?\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?', s)
        for ss in aList:
            aNum = float((ss[0]+ss[2]))
            box_priors_.append(aNum)
    fp.close()

    box_priors = np.array(box_priors_)
    box_priors = box_priors.reshape(4, NUM_RESULTS)

    return box_priors


if __name__ == '__main__':

    # Create RKNN object
    rknn = RKNN()

    # Config for Model Input PreProcess
    print('--> Config model')
    rknn.config(mean_values=[[127.5, 127.5, 127.5]], std_values=[[127.5, 127.5, 127.5]], reorder_channel='0 1 2')
    print('done')

    # Load TensorFlow Model
    print('--> Loading model')
    ret = rknn.load_tensorflow(tf_pb='./tflite_graph.pb',
                               inputs=['normalized_input_image_tensor'],
                               outputs=['concat', 'concat_1'],
                               input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3]])
    if ret != 0:
        print('Load model failed!')
        exit(ret)
    print('done')

    # Build Model
    print('--> Building model')
    ret = rknn.build(do_quantization=True, dataset='./dataset.txt')
    if ret != 0:
        print('Build model failed!')
        exit(ret)
    print('done')

    # Export RKNN Model
    print('--> Export RKNN model')
    rknn.export_rknn('./ssd.rknn')
    if ret != 0:
        print('Export RKNN model failed!')
        exit(ret)
    print('done')

    # Direct Load RKNN Model
    # rknn.load_rknn('./ssd.rknn')

    # Set inputs
    orig_img = cv2.imread('./horizontal.bmp')
    img = cv2.cvtColor(orig_img, cv2.COLOR_BGR2RGB)
    img = cv2.resize(img, (INPUT_SIZE, INPUT_SIZE), interpolation=cv2.INTER_CUBIC)

    # init runtime environment
    print('--> Init runtime environment')
    ret = rknn.init_runtime()
    if ret != 0:
        print('Init runtime environment failed')
        exit(ret)
    print('done')

    # Inference
    print('--> Running model')
    outputs = rknn.inference(inputs=[img])
    print('done')

    predictions = outputs[0].reshape((1, NUM_RESULTS, 4))
    outputClasses = outputs[1].reshape((1, NUM_RESULTS, NUM_CLASSES))
    candidateBox = np.zeros([2, NUM_RESULTS], dtype=int)
    vaildCnt = 0

    box_priors = load_box_priors()

    # Post Process
    # got valid candidate box
    for i in range(0, NUM_RESULTS):
        topClassScore = -1000
        topClassScoreIndex = -1

        # Skip the first catch-all class.
        for j in range(1, NUM_CLASSES):
            score = expit(outputClasses[0][i][j]);

            if score > topClassScore:
                topClassScoreIndex = j
                topClassScore = score

        if topClassScore > 0.4:
            candidateBox[0][vaildCnt] = i
            candidateBox[1][vaildCnt] = topClassScoreIndex
            vaildCnt += 1

    # calc position
    for i in range(0, vaildCnt):
        if candidateBox[0][i] == -1:
            continue

        n = candidateBox[0][i]
        ycenter = predictions[0][n][0] / Y_SCALE * box_priors[2][n] + box_priors[0][n]
        xcenter = predictions[0][n][1] / X_SCALE * box_priors[3][n] + box_priors[1][n]
        h = math.exp(predictions[0][n][2] / H_SCALE) * box_priors[2][n]
        w = math.exp(predictions[0][n][3] / W_SCALE) * box_priors[3][n]

        ymin = ycenter - h / 2.
        xmin = xcenter - w / 2.
        ymax = ycenter + h / 2.
        xmax = xcenter + w / 2.

        predictions[0][n][0] = ymin
        predictions[0][n][1] = xmin
        predictions[0][n][2] = ymax
        predictions[0][n][3] = xmax

    # NMS
    for i in range(0, vaildCnt):
        if candidateBox[0][i] == -1:
            continue

        n = candidateBox[0][i]
        xmin0 = predictions[0][n][1]
        ymin0 = predictions[0][n][0]
        xmax0 = predictions[0][n][3]
        ymax0 = predictions[0][n][2]

        for j in range(i+1, vaildCnt):
            m = candidateBox[0][j]

            if m == -1:
                continue

            xmin1 = predictions[0][m][1]
            ymin1 = predictions[0][m][0]
            xmax1 = predictions[0][m][3]
            ymax1 = predictions[0][m][2]

            iou = CalculateOverlap(xmin0, ymin0, xmax0, ymax0, xmin1, ymin1, xmax1, ymax1)

            if iou >= 0.45:
                candidateBox[0][j] = -1

    # Draw result
    for i in range(0, vaildCnt):
        if candidateBox[0][i] == -1:
            continue

        n = candidateBox[0][i]

        xmin = max(0.0, min(1.0, predictions[0][n][1])) * INPUT_SIZE
        ymin = max(0.0, min(1.0, predictions[0][n][0])) * INPUT_SIZE
        xmax = max(0.0, min(1.0, predictions[0][n][3])) * INPUT_SIZE
        ymax = max(0.0, min(1.0, predictions[0][n][2])) * INPUT_SIZE

        # print("%d @ (%d, %d) (%d, %d) score=%f" % (topClassScoreIndex, xmin, ymin, xmax, ymax, topClassScore))
        cv2.rectangle(orig_img, (int(xmin), int(ymin)), (int(xmax), int(ymax)),
                      (random.random()*255, random.random()*255, random.random()*255), 3)

    cv2.imwrite("out.jpg", orig_img)

    # Evaluate Perf on Simulator
    print('--> Evaluate model performance')
    rknn.eval_perf(inputs=[img], is_print=True)
    print('done')

    # Release RKNN Context
    rknn.release()
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 23:17:37 | 显示全部楼层
jefferyzhang 发表于 2022-7-31 23:05
看过去是Init runtime时候target写错了吧

>>> from rknn.api import RKNN
>>> rknn = RKNN()
>>> rknn.list_devices()
*************************
None devices connected.
*************************
([], [])
>>>
回复

使用道具 举报

ooo

注册会员

积分
158
 楼主| 发表于 2022-7-31 23:26:54 | 显示全部楼层
补充:我这有两个TB-RK3399ProD 3G(2G+1G),但是另外一个却可以正常转换模型后推理,同样的文件,同样的操作
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

产品中心 购买渠道 开源社区 Wiki教程 资料下载 关于Toybrick


快速回复 返回顶部 返回列表