instance segmentation model on rockchip

[复制链接] · 发表于 2020-2-3 13:29:03

Hi, I am trying to convert some segmentation model on rockchip, However, I didn't find any example about that. I am trying to convert some model to rknn, but always get error.
Some error is like File "rknn / base / RKNNlib / converter / convert_darknet.py ", line 394, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assignE AttributeError: 'NoneType' object has no attribute 'format'
Does anyone has any success on running segmentation model on rockchip? Any advices?
Thanks

只看该作者 · 发表于 2020-2-3 15:55:03

Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.

只看该作者 · 发表于 2020-2-3 16:36:05

本帖最后由 nvnn 于 2020-2-3 16:39 编辑

jefferyzhang 发表于 2020-2-3 15:55
Try latest version of rknn-toolkit (1.3.0) and enable verbose for details.

Hi, Thanks for your reply. I already try the latest version (1.3.0).
To be more specific, I am trying to add few more layers (only convolution and upsampling layers) to Yolov3-tiny to do both object detection and segmentation.
The models and conversion code are here:
https://drive.google.com/drive/f ... 8tBO2lN?usp=sharing

my conversion code is:

from rknn.api import RKNN



if __name__ == '__main__':



  rknn = RKNN(verbose = True)

  #darknet model

  rknn.load_darknet(model='yolov3-tiny-seg.cfg', weight='yolov3-tiny-seg.weights')

  rknn.config(channel_mean_value='0 0 0 255', reorder_channel='0 1 2')



  rknn.build(do_quantization=True, dataset='./dataset.txt')

  rknn.export_rknn('models/converted.rknn')

  exit(0)

复制代码

The part I added to yolov3-tiny is:

[route]

layers = 13



[convolutional]

batch_normalize=1

size=3

stride=1

pad=1

filters=128

activation=leaky



[convolutional]

batch_normalize=1

filters=2

size=3

stride=1

pad=1

activation=linear



[upsample]

stride=2



[upsample]

stride=2



[upsample]

stride=2



[upsample]

stride=2



[upsample]

stride=2
复制代码

And the error is:

....

D Process convolution_45 ...

D RKNN output shape(convolution): (0 26 26 255)

D Process output_46 ...

D RKNN output shape(output): (0 26 26 255)

I Build yolov3-tiny-seg complete.

E Catch exception when loading darknet model: yolov3-tiny-seg.cfg!

T Traceback (most recent call last):

T   File "rknn/api/rknn_base.py", line 382, in rknn.api.rknn_base.RKNNBase.load_darknet

T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 471, in rknn.base.RKNNlib.converter.convert_darknet.Converter.load_blobs

T   File "rknn/base/RKNNlib/converter/convert_darknet.py", line 211, in rknn.base.RKNNlib.converter.convert_darknet.Convolution_Blob_Assign.assign

T AttributeError: 'NoneType' object has no attribute 'format'

Traceback (most recent call last):

  File "rknn_transform.py", line 28, in <module>

    rknn.build(do_quantization=True, dataset='./dataset.txt')

  File "/home/nvnn/Downloads/env_testrknn/lib/python3.6/site-packages/rknn/api/rknn.py", line 163, in build

    inputs = self.rknn_base.net.get_input_layers()

AttributeError: 'NoneType' object has no attribute 'get_input_layers'

复制代码

I think there is no problem with this model because I can load and made prediction by opencv. The part I added is just convolution and upsampling layers, so it should be fine to convert to rknn model.

只看该作者 · 发表于 2020-2-3 16:56:00

It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or caffe instead?

只看该作者 · 发表于 2020-2-3 17:35:31

jefferyzhang 发表于 2020-2-3 16:56
It looks like darknet support issue, I will report to NPU team.
Could you try to use tensorflow or c ...

thanks for your reply.
It seems like there is problem with pytorch too,
I convert to that model pytorch and try converting to rknn.
The error log is:

E Catch exception when loading pytorch model: ./last.pt!

E Traceback (most recent call last):

E   File "rknn/api/rknn_base.py", line 567, in rknn.api.rknn_base.RKNNBase.load_pytorch

E   File "rknn/base/RKNNlib/app/importer/import_pytorch.py", line 95, in rknn.base.RKNNlib.app.importer.import_pytorch.ImportPytorch.run

E   File "rknn/base/RKNNlib/converter/convert_pytorch.py", line 499, in rknn.base.RKNNlib.converter.convert_pytorch.convert_pytorch.__init__

E   File "/home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/jit/__init__.py", line 235, in load

E     cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)

E RuntimeError: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory

E frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7f0108abbd37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libc10.so)

E frame #1: caffe2::serialize::PyTorchStreamReader::valid(char const*, char const*) + 0x72 (0x7f010bc3eb62 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)

E frame #2: caffe2::serialize::PyTorchStreamReader::init() + 0xa1 (0x7f010bc41b31 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)

E frame #3: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7f010bc44c04 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)

E frame #4: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::string const&, c10::optional<c10::Device>, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >&) + 0x35 (0x7f010cfa4845 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch.so)

E frame #5: <unknown function> + 0x776ffb (0x7f0153c25ffb in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

E frame #6: <unknown function> + 0x2961c4 (0x7f01537451c4 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

E frame #7: python() [0x4f8925]

E <omitting python frames>

E frame #9: python() [0x4f6128]

E frame #11: <unknown function> + 0x18578 (0x7f01b8475578 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #12: <unknown function> + 0x18e37 (0x7f01b8475e37 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #13: <unknown function> + 0xb04d5 (0x7f01b850d4d5 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/converter/convert_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #14: python() [0x5a105c]

E frame #15: python() [0x512be5]

E frame #16: python() [0x53ee21]

E frame #17: <unknown function> + 0x6a4c (0x7f01c0948a4c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #18: <unknown function> + 0xc67c (0x7f01c094e67c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #19: <unknown function> + 0x1255c (0x7f01c095455c in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/base/RKNNlib/app/importer/import_pytorch.cpython-36m-x86_64-linux-gnu.so)

E frame #20: <unknown function> + 0x20bdd (0x7f01c9e0ebdd in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)

E frame #21: <unknown function> + 0x20dde (0x7f01c9e0edde in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)

E frame #22: <unknown function> + 0xce615 (0x7f01c9ebc615 in /home/nrknn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn_base.cpython-36m-x86_64-linux-gnu.so)

E frame #24: python() [0x4f88ba]

E frame #26: python() [0x4f6128]

E frame #27: python() [0x4f7d60]

E frame #28: python() [0x4f876d]

E frame #30: python() [0x4f6128]

E frame #32: python() [0x6415b2]

E frame #37: __libc_start_main + 0xe7 (0x7f01cbddbb97 in /lib/x86_64-linux-gnu/libc.so.6)

Traceback (most recent call last):

  File "rknn_transform.py", line 28, in <module>

    rknn.build(do_quantization=True, dataset='./dataset.txt')

  File "/home/nvnn/Downloads/env_rknn11/lib/python3.6/site-packages/rknn/api/rknn.py", line 220, in build

    inputs = self.rknn_base.net.get_input_layers()

AttributeError: 'NoneType' object has no attribute 'get_input_layers'

复制代码

只看该作者 · 发表于 2020-2-3 20:17:23

本帖最后由 jefferyzhang 于 2020-2-3 20:19 编辑

Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron.

只看该作者 · 发表于 2020-2-4 08:26:08

jefferyzhang 发表于 2020-2-3 20:17
Did you setup correct input name in rknn.load_xxxx?
You can check the name of input layer by Netron. ...

according to the manual, i don't need to set the input name while loading pytorch model.
This is the sample code from manual:

# Load the pytorch model resnet18 in the current path

ret = rknn. load_pytorch(model = './resnet18.pt',

input_size_list=[[3,224,224]])
复制代码

只看该作者 · 发表于 2020-2-6 20:05:06

My colleague want to know did you add new layers after the origin model?

只看该作者 · 发表于 2020-2-6 20:52:31

1. For darknet model, did you update the .weight file to add weight and bias for these layers you added in .cfg file?
2. For pytorch, we only support >=1.0.0 and <= 1.2.0. And need a model in the torchscript format.

只看该作者 · 发表于 2020-2-7 08:05:09

Hi jefferyzhang, raul.

Thanks for your reply. Finally, I can have a working model.
I did add new layers after the origin model and retrain the new weights file.
For pytorch, I used rknn-toolkit 1.3.0.

My solution was just add a blank yolo layer at the end of the cfg file.

Thanks for your support