toolkit_lite2推理的dump error问题

[复制链接] · 发表于 2023-11-14 16:55:35

本帖最后由 BoyceL 于 2023-11-15 10:11 编辑

我在ubuntu上使用toolkit2将onnx转换成了rknn模型，并且在pc端使用python对比了onnx与rknn的推理结果，结果保持一致。但在3588板端加载该rknn模型，并使用相同的输入，却得到了完全不同的结果。注意到在推理过程中具有如下报错，但并没有跳出程序。（推理错误问题已经解决，但dump error并未解决）

D RKNN: [08:39:53.242] dump npy tensor to: dumps/008_Gather_348_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/008_Gather_348_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/009_Mul_354_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/009_Mul_354_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/010_Reshape_358_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/010_Reshape_358_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/011_Gather_350_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/011_Gather_350_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/012_Mul_352_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/012_Mul_352_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/013_Reshape_356_shape_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/013_Reshape_356_shape_1_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/014_Concat_359_shape_2_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/014_Concat_359_shape_2_float.npy

D RKNN: [08:39:53.242] dump npy tensor to: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy

E RKNN: [08:39:53.242] failed open dump path: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/023_Sub_370_shape_1_512_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/023_Sub_370_shape_1_512_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/024_Div_373_shape_1_512_1_2_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/024_Div_373_shape_1_512_1_2_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/025_Gather_375_shape_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/025_Gather_375_shape_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/026_Mul_381_shape_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/026_Mul_381_shape_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/027_Reshape_385_shape_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/027_Reshape_385_shape_1_float.npy

D RKNN: [08:39:53.243] dump npy tensor to: dumps/028_Gather_377_shape_1_float.npy

E RKNN: [08:39:53.243] failed open dump path: dumps/028_Gather_377_shape_1_float.npy

D RKNN: [08:39:53.244] dump npy tensor to: dumps/029_Mul_379_shape_1_float.npy

E RKNN: [08:39:53.244] failed open dump path: dumps/029_Mul_379_shape_1_float.npy

复制代码

这样的错误几乎存在于各个网络层，想问这个错误可能是由什么原因导致的？有什么样的解决方案？并且与推理结果错误是否有直接联系？

ps：我在网络中使用的matmul算子，发现并不能在npu上运行，转到cpu上运行要10-30ms，是由于我的feature不满足npu matmul的约束规格吗？（我的channel为4，约束为>=8）

只看该作者 · 发表于 2023-11-15 10:10:00

目前已经找到问题在于torch.onnx.export时，使用的opset_version=13，将其修改为11即可推理正确，但还是没能解决dump error的问题，这个问题会不会影响推理时间？似乎每次推理的过程都会出现

只看该作者 · 发表于 2023-11-15 11:17:23

这dump是你自己开了分析每一层输出的选项吧？每一层都输出结果，但是保存出错

只看该作者 · 发表于 2023-11-20 06:57:41

这个是打开了调试，的输出，可以不管，也可以创建个dumps目录，来保存中间数据，就不报错了