Toybrick

instance segmentation model on rockchip

nvnn

注册会员

积分
124
楼主
发表于 2020-2-3 13:29:03    查看: 26425|回复: 10 | [复制链接]    打印 | 只看该作者
Hi, I am trying to convert some segmentation model on rockchip, However, I didn't find any example about that. I am trying to convert some model to rknn, but always get error.  
Some error is like File "rknn / base / RKNNlib / converter / convert_darknet.py ", line 394, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assignE AttributeError: 'NoneType' object has no attribute 'format'
Does anyone has any success on running segmentation model on rockchip? Any advices?
Thanks

回复

使用道具 举报

jefferyzhang

版主

积分
13514
沙发
发表于 2020-2-3 15:55:03 | 只看该作者
Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.
回复

使用道具 举报

nvnn

注册会员

积分
124
板凳
 楼主| 发表于 2020-2-3 16:36:05 | 只看该作者
本帖最后由 nvnn 于 2020-2-3 16:39 编辑
jefferyzhang 发表于 2020-2-3 15:55
Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.

Hi, Thanks for your reply. I already try the latest version (1.3.0).
To be more specific, I am trying to add few more layers (only convolution and upsampling layers) to Yolov3-tiny to do both object detection and segmentation.
The models and conversion code are here:
https://drive.google.com/drive/f ... 8tBO2lN?usp=sharing

my conversion code is:
  1. from rknn.api import RKNN

  2. if __name__ == '__main__':

  3.   rknn = RKNN(verbose = True)
  4.   #darknet model
  5.   rknn.load_darknet(model='yolov3-tiny-seg.cfg', weight='yolov3-tiny-seg.weights')
  6.   rknn.config(channel_mean_value='0 0 0 255', reorder_channel='0 1 2')

  7.   rknn.build(do_quantization=True, dataset='./dataset.txt')
  8.   rknn.export_rknn('models/converted.rknn')
  9.   exit(0)
复制代码

The part I added to yolov3-tiny is:
  1. [route]
  2. layers = 13

  3. [convolutional]
  4. batch_normalize=1
  5. size=3
  6. stride=1
  7. pad=1
  8. filters=128
  9. activation=leaky

  10. [convolutional]
  11. batch_normalize=1
  12. filters=2
  13. size=3
  14. stride=1
  15. pad=1
  16. activation=linear

  17. [upsample]
  18. stride=2

  19. [upsample]
  20. stride=2

  21. [upsample]
  22. stride=2

  23. [upsample]
  24. stride=2

  25. [upsample]
  26. stride=2
复制代码
And the error is:
  1. ....
  2. D Process convolution_45 ...
  3. D RKNN output shape(convolution): (0 26 26 255)
  4. D Process output_46 ...
  5. D RKNN output shape(output): (0 26 26 255)
  6. I Build yolov3-tiny-seg complete.
  7. E Catch exception when loading darknet model: yolov3-tiny-seg.cfg!
  8. T Traceback (most recent call last):
  9. T   File "rknn/api/rknn_base.py", line 382, in rknn.api.rknn_base.RKNNBase.load_darknet
  10. T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 471, in rknn.base.RKNNlib.converter.convert_darknet.Converter.load_blobs
  11. T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 211, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assign
  12. T AttributeError: 'NoneType' object has no attribute 'format'
  13. Traceback (most recent call last):
  14.   File "rknn_transform.py", line 28, in <module>
  15.     rknn.build(do_quantization=True, dataset='./dataset.txt')
  16.   File "/home/nvnn/Downloads/env_testrknn/lib/python3.6/site-packages/rknn/api/rknn.py", line 163, in build
  17.     inputs = self.rknn_base.net.get_input_layers()
  18. AttributeError: 'NoneType' object has no attribute 'get_input_layers'
复制代码

I think there is no problem with this model because I can load and made prediction by opencv. The part I added is just convolution and upsampling layers, so it should be fine to convert to rknn model.
回复

使用道具 举报

jefferyzhang

版主

积分
13514
地板
发表于 2020-2-3 16:56:00 | 只看该作者
It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or caffe instead?
回复

使用道具 举报

nvnn

注册会员

积分
124
5#
 楼主| 发表于 2020-2-3 17:35:31 | 只看该作者
jefferyzhang 发表于 2020-2-3 16:56
It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or c ...

thanks for your reply.
It seems like there is problem with pytorch too,
I convert to that model pytorch and try converting to rknn.
The error log is:
  1. E Catch exception when loading pytorch model: ./last.pt!
  2. E Traceback (most recent call last):
  3. E   File "rknn/api/rknn_base.py", line 567, in rknn.api.rknn_base.RKNNBase.load_pytorch
  4. E   File "rknn/base/RKNNlib/app/importer/import_pytorch.py", line 95, in rknn.base.RKNNlib.app.importer.import_pytorch.ImportPytorch.run
  5. E   File "rknn/base/RKNNlib/converter/convert_pytorch.py", line 499, in rknn.base.RKNNlib.converter.convert_pytorch.convert_pytorch.__init__
  6. E   File "/home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/jit/__init__.py", line 235, in load
  7. E     cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
  8. E RuntimeError: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory
  9. E frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7f0108abbd37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libc10.so)
  10. E frame #1: caffe2::serialize::PyTorchStreamReader::valid(char const*, char const*) + 0x72 (0x7f010bc3eb62 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  11. E frame #2: caffe2::serialize::PyTorchStreamReader::init() + 0xa1 (0x7f010bc41b31 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  12. E frame #3: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7f010bc44c04 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  13. E frame #4: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::string const&, c10::optional<c10::Device>, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >&) + 0x35 (0x7f010cfa4845 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  14. E frame #5: <unknown function> + 0x776ffb (0x7f0153c25ffb in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  15. E frame #6: <unknown function> + 0x2961c4 (0x7f01537451c4 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  16. E frame #7: python() [0x4f8925]
  17. E <omitting python frames>
  18. E frame #9: python() [0x4f6128]
  19. E frame #11: <unknown function> + 0x18578 (0x7f01b8475578 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  20. E frame #12: <unknown function> + 0x18e37 (0x7f01b8475e37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  21. E frame #13: <unknown function> + 0xb04d5 (0x7f01b850d4d5 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  22. E frame #14: python() [0x5a105c]
  23. E frame #15: python() [0x512be5]
  24. E frame #16: python() [0x53ee21]
  25. E frame #17: <unknown function> + 0x6a4c (0x7f01c0948a4c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  26. E frame #18: <unknown function> + 0xc67c (0x7f01c094e67c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  27. E frame #19: <unknown function> + 0x1255c (0x7f01c095455c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  28. E frame #20: <unknown function> + 0x20bdd (0x7f01c9e0ebdd in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  29. E frame #21: <unknown function> + 0x20dde (0x7f01c9e0edde in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  30. E frame #22: <unknown function> + 0xce615 (0x7f01c9ebc615 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  31. E frame #24: python() [0x4f88ba]
  32. E frame #26: python() [0x4f6128]
  33. E frame #27: python() [0x4f7d60]
  34. E frame #28: python() [0x4f876d]
  35. E frame #30: python() [0x4f6128]
  36. E frame #32: python() [0x6415b2]
  37. E frame #37: __libc_start_main + 0xe7 (0x7f01cbddbb97 in /lib/x86_64-linux-gnu/libc.so.6)
  38. Traceback (most recent call last):
  39.   File "rknn_transform.py", line 28, in <module>
  40.     rknn.build(do_quantization=True, dataset='./dataset.txt')
  41.   File "/home/nvnn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn.py", line 220, in build
  42.     inputs = self.rknn_base.net.get_input_layers()
  43. AttributeError: 'NoneType' object has no attribute 'get_input_layers'
复制代码
回复

使用道具 举报

jefferyzhang

版主

积分
13514
6#
发表于 2020-2-3 20:17:23 | 只看该作者
本帖最后由 jefferyzhang 于 2020-2-3 20:19 编辑

Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron.
回复

使用道具 举报

nvnn

注册会员

积分
124
7#
 楼主| 发表于 2020-2-4 08:26:08 | 只看该作者
jefferyzhang 发表于 2020-2-3 20:17
Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron. ...

according to the manual, i don't need to set the input name while loading pytorch model.
This is the sample code from manual:
  1. # Load the pytorch model resnet18 in the current path
  2. ret = rknn. load_pytorch(model = './resnet18.pt',
  3. input_size_list=[[3,224,224]])
复制代码
回复

使用道具 举报

jefferyzhang

版主

积分
13514
8#
发表于 2020-2-6 20:05:06 | 只看该作者
My colleague want to know did you add new layers after the origin model?
回复

使用道具 举报

raul

中级会员

积分
235
9#
发表于 2020-2-6 20:52:31 | 只看该作者
1. For darknet model, did you update the .weight file to add weight and bias for these layers you added in .cfg file?
2. For pytorch, we only support >=1.0.0 and <= 1.2.0. And need a model in the torchscript format.
回复

使用道具 举报

nvnn

注册会员

积分
124
10#
 楼主| 发表于 2020-2-7 08:05:09 | 只看该作者
Hi jefferyzhang, raul.

Thanks for your reply. Finally, I can have a working model.
I did add new layers after the origin model and retrain the new weights file.
For pytorch, I used rknn-toolkit 1.3.0.

My solution was just add a blank yolo layer at the end of the cfg file.

Thanks for your support
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

产品中心 购买渠道 开源社区 Wiki教程 资料下载 关于Toybrick


快速回复 返回顶部 返回列表