Toybrick

标题: toolkit_lite2推理的dump error问题 [打印本页]
作者: BoyceL    时间: 2023-11-14 16:55
标题: toolkit_lite2推理的dump error问题
本帖最后由 BoyceL 于 2023-11-15 10:11 编辑

我在ubuntu上使用toolkit2将onnx转换成了rknn模型，并且在pc端使用python对比了onnx与rknn的推理结果，结果保持一致。但在3588板端加载该rknn模型，并使用相同的输入，却得到了完全不同的结果。注意到在推理过程中具有如下报错，但并没有跳出程序。（推理错误问题已经解决，但dump error并未解决）
  1. D RKNN: [08:39:53.242] dump npy tensor to: dumps/008_Gather_348_shape_1_float.npy
  2. E RKNN: [08:39:53.242] failed open dump path: dumps/008_Gather_348_shape_1_float.npy
  3. D RKNN: [08:39:53.242] dump npy tensor to: dumps/009_Mul_354_shape_1_float.npy
  4. E RKNN: [08:39:53.242] failed open dump path: dumps/009_Mul_354_shape_1_float.npy
  5. D RKNN: [08:39:53.242] dump npy tensor to: dumps/010_Reshape_358_shape_1_float.npy
  6. E RKNN: [08:39:53.242] failed open dump path: dumps/010_Reshape_358_shape_1_float.npy
  7. D RKNN: [08:39:53.242] dump npy tensor to: dumps/011_Gather_350_shape_1_float.npy
  8. E RKNN: [08:39:53.242] failed open dump path: dumps/011_Gather_350_shape_1_float.npy
  9. D RKNN: [08:39:53.242] dump npy tensor to: dumps/012_Mul_352_shape_1_float.npy
  10. E RKNN: [08:39:53.242] failed open dump path: dumps/012_Mul_352_shape_1_float.npy
  11. D RKNN: [08:39:53.242] dump npy tensor to: dumps/013_Reshape_356_shape_1_float.npy
  12. E RKNN: [08:39:53.242] failed open dump path: dumps/013_Reshape_356_shape_1_float.npy
  13. D RKNN: [08:39:53.242] dump npy tensor to: dumps/014_Concat_359_shape_2_float.npy
  14. E RKNN: [08:39:53.242] failed open dump path: dumps/014_Concat_359_shape_2_float.npy
  15. D RKNN: [08:39:53.242] dump npy tensor to: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy
  16. E RKNN: [08:39:53.242] failed open dump path: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy
  17. D RKNN: [08:39:53.243] dump npy tensor to: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy
  18. E RKNN: [08:39:53.243] failed open dump path: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy
  19. D RKNN: [08:39:53.243] dump npy tensor to: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy
  20. E RKNN: [08:39:53.243] failed open dump path: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy
  21. D RKNN: [08:39:53.243] dump npy tensor to: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy
  22. E RKNN: [08:39:53.243] failed open dump path: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy
  23. D RKNN: [08:39:53.243] dump npy tensor to: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy
  24. E RKNN: [08:39:53.243] failed open dump path: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy
  25. D RKNN: [08:39:53.243] dump npy tensor to: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy
  26. E RKNN: [08:39:53.243] failed open dump path: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy
  27. D RKNN: [08:39:53.243] dump npy tensor to: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy
  28. E RKNN: [08:39:53.243] failed open dump path: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy
  29. D RKNN: [08:39:53.243] dump npy tensor to: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy
  30. E RKNN: [08:39:53.243] failed open dump path: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy
  31. D RKNN: [08:39:53.243] dump npy tensor to: dumps/023_Sub_370_shape_1_512_1_2_float.npy
  32. E RKNN: [08:39:53.243] failed open dump path: dumps/023_Sub_370_shape_1_512_1_2_float.npy
  33. D RKNN: [08:39:53.243] dump npy tensor to: dumps/024_Div_373_shape_1_512_1_2_float.npy
  34. E RKNN: [08:39:53.243] failed open dump path: dumps/024_Div_373_shape_1_512_1_2_float.npy
  35. D RKNN: [08:39:53.243] dump npy tensor to: dumps/025_Gather_375_shape_1_float.npy
  36. E RKNN: [08:39:53.243] failed open dump path: dumps/025_Gather_375_shape_1_float.npy
  37. D RKNN: [08:39:53.243] dump npy tensor to: dumps/026_Mul_381_shape_1_float.npy
  38. E RKNN: [08:39:53.243] failed open dump path: dumps/026_Mul_381_shape_1_float.npy
  39. D RKNN: [08:39:53.243] dump npy tensor to: dumps/027_Reshape_385_shape_1_float.npy
  40. E RKNN: [08:39:53.243] failed open dump path: dumps/027_Reshape_385_shape_1_float.npy
  41. D RKNN: [08:39:53.243] dump npy tensor to: dumps/028_Gather_377_shape_1_float.npy
  42. E RKNN: [08:39:53.243] failed open dump path: dumps/028_Gather_377_shape_1_float.npy
  43. D RKNN: [08:39:53.244] dump npy tensor to: dumps/029_Mul_379_shape_1_float.npy
  44. E RKNN: [08:39:53.244] failed open dump path: dumps/029_Mul_379_shape_1_float.npy
复制代码这样的错误几乎存在于各个网络层，想问这个错误可能是由什么原因导致的？有什么样的解决方案？并且与推理结果错误是否有直接联系？

ps：我在网络中使用的matmul算子，发现并不能在npu上运行，转到cpu上运行要10-30ms，是由于我的feature不满足npu matmul的约束规格吗？（我的channel为4，约束为>=8）
作者: BoyceL    时间: 2023-11-15 10:10
目前已经找到问题在于torch.onnx.export时，使用的opset_version=13，将其修改为11即可推理正确，但还是没能解决dump error的问题，这个问题会不会影响推理时间？似乎每次推理的过程都会出现
作者: jefferyzhang    时间: 2023-11-15 11:17
这dump是你自己开了分析每一层输出的选项吧？每一层都输出结果，但是保存出错
作者: wudiking    时间: 2023-11-20 06:57
这个是打开了调试，的输出，可以不管，也可以创建个dumps目录，来保存中间数据，就不报错了



欢迎光临 Toybrick (https://t.rock-chips.com/) Powered by Discuz! X3.3