Toybrick

toolkit_lite2推理的dump error问题

BoyceL

新手上路

积分
15
发表于 2023-11-14 16:55:35    查看: 2199|回复: 3 | [复制链接]    打印 | 显示全部楼层
本帖最后由 BoyceL 于 2023-11-15 10:11 编辑

我在ubuntu上使用toolkit2将onnx转换成了rknn模型,并且在pc端使用python对比了onnx与rknn的推理结果,结果保持一致。但在3588板端加载该rknn模型,并使用相同的输入,却得到了完全不同的结果。注意到在推理过程中具有如下报错,但并没有跳出程序。(推理错误问题已经解决,但dump error并未解决)
  1. D RKNN: [08:39:53.242] dump npy tensor to: dumps/008_Gather_348_shape_1_float.npy
  2. E RKNN: [08:39:53.242] failed open dump path: dumps/008_Gather_348_shape_1_float.npy
  3. D RKNN: [08:39:53.242] dump npy tensor to: dumps/009_Mul_354_shape_1_float.npy
  4. E RKNN: [08:39:53.242] failed open dump path: dumps/009_Mul_354_shape_1_float.npy
  5. D RKNN: [08:39:53.242] dump npy tensor to: dumps/010_Reshape_358_shape_1_float.npy
  6. E RKNN: [08:39:53.242] failed open dump path: dumps/010_Reshape_358_shape_1_float.npy
  7. D RKNN: [08:39:53.242] dump npy tensor to: dumps/011_Gather_350_shape_1_float.npy
  8. E RKNN: [08:39:53.242] failed open dump path: dumps/011_Gather_350_shape_1_float.npy
  9. D RKNN: [08:39:53.242] dump npy tensor to: dumps/012_Mul_352_shape_1_float.npy
  10. E RKNN: [08:39:53.242] failed open dump path: dumps/012_Mul_352_shape_1_float.npy
  11. D RKNN: [08:39:53.242] dump npy tensor to: dumps/013_Reshape_356_shape_1_float.npy
  12. E RKNN: [08:39:53.242] failed open dump path: dumps/013_Reshape_356_shape_1_float.npy
  13. D RKNN: [08:39:53.242] dump npy tensor to: dumps/014_Concat_359_shape_2_float.npy
  14. E RKNN: [08:39:53.242] failed open dump path: dumps/014_Concat_359_shape_2_float.npy
  15. D RKNN: [08:39:53.242] dump npy tensor to: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy
  16. E RKNN: [08:39:53.242] failed open dump path: dumps/015_Reshape_361_shape4_Div_15_2mul_shape_1_2_1_1_float.npy
  17. D RKNN: [08:39:53.243] dump npy tensor to: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy
  18. E RKNN: [08:39:53.243] failed open dump path: dumps/016_ReduceMax_365_shape_1_1_1_1_float.npy
  19. D RKNN: [08:39:53.243] dump npy tensor to: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy
  20. E RKNN: [08:39:53.243] failed open dump path: dumps/017_Mul_367_shape4_shape_1_1_1_1_float.npy
  21. D RKNN: [08:39:53.243] dump npy tensor to: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy
  22. E RKNN: [08:39:53.243] failed open dump path: dumps/018_Expand_372_shape4_Div_24_expand_for_Div_24_shape_1_512_1_2_float.npy
  23. D RKNN: [08:39:53.243] dump npy tensor to: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy
  24. E RKNN: [08:39:53.243] failed open dump path: dumps/019_Mul_364_shape4_shape_1_2_1_1_float.npy
  25. D RKNN: [08:39:53.243] dump npy tensor to: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy
  26. E RKNN: [08:39:53.243] failed open dump path: dumps/020_Reshape_369_shape4_Sub_21_shape_1_1_1_2_float.npy
  27. D RKNN: [08:39:53.243] dump npy tensor to: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy
  28. E RKNN: [08:39:53.243] failed open dump path: dumps/021_Conv_Sub_21_conv_out_shape_1_512_1_2_float.npy
  29. D RKNN: [08:39:53.243] dump npy tensor to: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy
  30. E RKNN: [08:39:53.243] failed open dump path: dumps/022_Reshape_keypoints0_shape4_Sub_21_shape_1_512_1_2_float.npy
  31. D RKNN: [08:39:53.243] dump npy tensor to: dumps/023_Sub_370_shape_1_512_1_2_float.npy
  32. E RKNN: [08:39:53.243] failed open dump path: dumps/023_Sub_370_shape_1_512_1_2_float.npy
  33. D RKNN: [08:39:53.243] dump npy tensor to: dumps/024_Div_373_shape_1_512_1_2_float.npy
  34. E RKNN: [08:39:53.243] failed open dump path: dumps/024_Div_373_shape_1_512_1_2_float.npy
  35. D RKNN: [08:39:53.243] dump npy tensor to: dumps/025_Gather_375_shape_1_float.npy
  36. E RKNN: [08:39:53.243] failed open dump path: dumps/025_Gather_375_shape_1_float.npy
  37. D RKNN: [08:39:53.243] dump npy tensor to: dumps/026_Mul_381_shape_1_float.npy
  38. E RKNN: [08:39:53.243] failed open dump path: dumps/026_Mul_381_shape_1_float.npy
  39. D RKNN: [08:39:53.243] dump npy tensor to: dumps/027_Reshape_385_shape_1_float.npy
  40. E RKNN: [08:39:53.243] failed open dump path: dumps/027_Reshape_385_shape_1_float.npy
  41. D RKNN: [08:39:53.243] dump npy tensor to: dumps/028_Gather_377_shape_1_float.npy
  42. E RKNN: [08:39:53.243] failed open dump path: dumps/028_Gather_377_shape_1_float.npy
  43. D RKNN: [08:39:53.244] dump npy tensor to: dumps/029_Mul_379_shape_1_float.npy
  44. E RKNN: [08:39:53.244] failed open dump path: dumps/029_Mul_379_shape_1_float.npy
复制代码
这样的错误几乎存在于各个网络层,想问这个错误可能是由什么原因导致的?有什么样的解决方案?并且与推理结果错误是否有直接联系?

ps:我在网络中使用的matmul算子,发现并不能在npu上运行,转到cpu上运行要10-30ms,是由于我的feature不满足npu matmul的约束规格吗?(我的channel为4,约束为>=8)
回复

使用道具 举报

BoyceL

新手上路

积分
15
 楼主| 发表于 2023-11-15 10:10:00 | 显示全部楼层
目前已经找到问题在于torch.onnx.export时,使用的opset_version=13,将其修改为11即可推理正确,但还是没能解决dump error的问题,这个问题会不会影响推理时间?似乎每次推理的过程都会出现
回复

使用道具 举报

jefferyzhang

版主

积分
13204
发表于 2023-11-15 11:17:23 | 显示全部楼层
这dump是你自己开了分析每一层输出的选项吧?每一层都输出结果,但是保存出错
回复

使用道具 举报

wudiking

高级会员

积分
693
发表于 2023-11-20 06:57:41 | 显示全部楼层
这个是打开了调试,的输出,可以不管,也可以创建个dumps目录,来保存中间数据,就不报错了
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

产品中心 购买渠道 开源社区 Wiki教程 资料下载 关于Toybrick


快速回复 返回顶部 返回列表