Toybrick

标题: instance segmentation model on rockchip [打印本页]
作者: nvnn    时间: 2020-2-3 13:29
标题: instance segmentation model on rockchip
Hi, I am trying to convert some segmentation model on rockchip, However, I didn't find any example about that. I am trying to convert some model to rknn, but always get error.  
Some error is like File "rknn / base / RKNNlib / converter / convert_darknet.py ", line 394, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assignE AttributeError: 'NoneType' object has no attribute 'format'
Does anyone has any success on running segmentation model on rockchip? Any advices?
Thanks

作者: jefferyzhang    时间: 2020-2-3 15:55
Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.
作者: nvnn    时间: 2020-2-3 16:36
本帖最后由 nvnn 于 2020-2-3 16:39 编辑
jefferyzhang 发表于 2020-2-3 15:55
Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.

Hi, Thanks for your reply. I already try the latest version (1.3.0).
To be more specific, I am trying to add few more layers (only convolution and upsampling layers) to Yolov3-tiny to do both object detection and segmentation.
The models and conversion code are here:
https://drive.google.com/drive/f ... 8tBO2lN?usp=sharing

my conversion code is:
  1. from rknn.api import RKNN

  3. if __name__ == '__main__':

  5.   rknn = RKNN(verbose = True)
  6.   #darknet model
  7.   rknn.load_darknet(model='yolov3-tiny-seg.cfg', weight='yolov3-tiny-seg.weights')
  8.   rknn.config(channel_mean_value='0 0 0 255', reorder_channel='0 1 2')

  10.   rknn.build(do_quantization=True, dataset='./dataset.txt')
  11.   rknn.export_rknn('models/converted.rknn')
  12.   exit(0)
复制代码
The part I added to yolov3-tiny is:
  1. [route]
  2. layers = 13

  4. [convolutional]
  5. batch_normalize=1
  6. size=3
  7. stride=1
  8. pad=1
  9. filters=128
  10. activation=leaky

  12. [convolutional]
  13. batch_normalize=1
  14. filters=2
  15. size=3
  16. stride=1
  17. pad=1
  18. activation=linear

  20. [upsample]
  21. stride=2

  23. [upsample]
  24. stride=2

  26. [upsample]
  27. stride=2

  29. [upsample]
  30. stride=2

  32. [upsample]
  33. stride=2
复制代码And the error is:
  1. ....
  2. D Process convolution_45 ...
  3. D RKNN output shape(convolution): (0 26 26 255)
  4. D Process output_46 ...
  5. D RKNN output shape(output): (0 26 26 255)
  6. I Build yolov3-tiny-seg complete.
  7. E Catch exception when loading darknet model: yolov3-tiny-seg.cfg!
  8. T Traceback (most recent call last):
  9. T   File "rknn/api/rknn_base.py", line 382, in rknn.api.rknn_base.RKNNBase.load_darknet
  10. T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 471, in rknn.base.RKNNlib.converter.convert_darknet.Converter.load_blobs
  11. T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 211, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assign
  12. T AttributeError: 'NoneType' object has no attribute 'format'
  13. Traceback (most recent call last):
  14.   File "rknn_transform.py", line 28, in <module>
  15.     rknn.build(do_quantization=True, dataset='./dataset.txt')
  16.   File "/home/nvnn/Downloads/env_testrknn/lib/python3.6/site-packages/rknn/api/rknn.py", line 163, in build
  17.     inputs = self.rknn_base.net.get_input_layers()
  18. AttributeError: 'NoneType' object has no attribute 'get_input_layers'
复制代码
I think there is no problem with this model because I can load and made prediction by opencv. The part I added is just convolution and upsampling layers, so it should be fine to convert to rknn model.
作者: jefferyzhang    时间: 2020-2-3 16:56
It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or caffe instead?
作者: nvnn    时间: 2020-2-3 17:35
jefferyzhang 发表于 2020-2-3 16:56
It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or c ...

thanks for your reply.
It seems like there is problem with pytorch too,
I convert to that model pytorch and try converting to rknn.
The error log is:
  1. E Catch exception when loading pytorch model: ./last.pt!
  2. E Traceback (most recent call last):
  3. E   File "rknn/api/rknn_base.py", line 567, in rknn.api.rknn_base.RKNNBase.load_pytorch
  4. E   File "rknn/base/RKNNlib/app/importer/import_pytorch.py", line 95, in rknn.base.RKNNlib.app.importer.import_pytorch.ImportPytorch.run
  5. E   File "rknn/base/RKNNlib/converter/convert_pytorch.py", line 499, in rknn.base.RKNNlib.converter.convert_pytorch.convert_pytorch.__init__
  6. E   File "/home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/jit/__init__.py", line 235, in load
  7. E     cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
  8. E RuntimeError: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory
  9. E frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7f0108abbd37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libc10.so)
  10. E frame #1: caffe2::serialize::PyTorchStreamReader::valid(char const*, char const*) + 0x72 (0x7f010bc3eb62 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  11. E frame #2: caffe2::serialize::PyTorchStreamReader::init() + 0xa1 (0x7f010bc41b31 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  12. E frame #3: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7f010bc44c04 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  13. E frame #4: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::string const&, c10::optional<c10::Device>, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >&) + 0x35 (0x7f010cfa4845 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)
  14. E frame #5: <unknown function> + 0x776ffb (0x7f0153c25ffb in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  15. E frame #6: <unknown function> + 0x2961c4 (0x7f01537451c4 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  16. E frame #7: python() [0x4f8925]
  17. E <omitting python frames>
  18. E frame #9: python() [0x4f6128]
  19. E frame #11: <unknown function> + 0x18578 (0x7f01b8475578 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  20. E frame #12: <unknown function> + 0x18e37 (0x7f01b8475e37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  21. E frame #13: <unknown function> + 0xb04d5 (0x7f01b850d4d5 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)
  22. E frame #14: python() [0x5a105c]
  23. E frame #15: python() [0x512be5]
  24. E frame #16: python() [0x53ee21]
  25. E frame #17: <unknown function> + 0x6a4c (0x7f01c0948a4c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  26. E frame #18: <unknown function> + 0xc67c (0x7f01c094e67c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  27. E frame #19: <unknown function> + 0x1255c (0x7f01c095455c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)
  28. E frame #20: <unknown function> + 0x20bdd (0x7f01c9e0ebdd in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  29. E frame #21: <unknown function> + 0x20dde (0x7f01c9e0edde in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  30. E frame #22: <unknown function> + 0xce615 (0x7f01c9ebc615 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)
  31. E frame #24: python() [0x4f88ba]
  32. E frame #26: python() [0x4f6128]
  33. E frame #27: python() [0x4f7d60]
  34. E frame #28: python() [0x4f876d]
  35. E frame #30: python() [0x4f6128]
  36. E frame #32: python() [0x6415b2]
  37. E frame #37: __libc_start_main + 0xe7 (0x7f01cbddbb97 in /lib/x86_64-linux-gnu/libc.so.6)
  38. Traceback (most recent call last):
  39.   File "rknn_transform.py", line 28, in <module>
  40.     rknn.build(do_quantization=True, dataset='./dataset.txt')
  41.   File "/home/nvnn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn.py", line 220, in build
  42.     inputs = self.rknn_base.net.get_input_layers()
  43. AttributeError: 'NoneType' object has no attribute 'get_input_layers'
复制代码
作者: jefferyzhang    时间: 2020-2-3 20:17
本帖最后由 jefferyzhang 于 2020-2-3 20:19 编辑

Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron.
作者: nvnn    时间: 2020-2-4 08:26
jefferyzhang 发表于 2020-2-3 20:17
Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron. ...

according to the manual, i don't need to set the input name while loading pytorch model.
This is the sample code from manual:
  1. # Load the pytorch model resnet18 in the current path
  2. ret = rknn. load_pytorch(model = './resnet18.pt',
  3. input_size_list=[[3,224,224]])
复制代码
作者: jefferyzhang    时间: 2020-2-6 20:05
My colleague want to know did you add new layers after the origin model?
作者: raul    时间: 2020-2-6 20:52
1. For darknet model, did you update the .weight file to add weight and bias for these layers you added in .cfg file?
2. For pytorch, we only support >=1.0.0 and <= 1.2.0. And need a model in the torchscript format.
作者: nvnn    时间: 2020-2-7 08:05
Hi jefferyzhang, raul.

Thanks for your reply. Finally, I can have a working model.
I did add new layers after the origin model and retrain the new weights file.
For pytorch, I used rknn-toolkit 1.3.0.

My solution was just add a blank yolo layer at the end of the cfg file.

Thanks for your support
作者: 1105060120    时间: 2020-7-24 08:40
你好，请问是否有TTS模型在嵌入式上面运行的例子



欢迎光临 Toybrick (https://t.rock-chips.com/) Powered by Discuz! X3.3