Toybrick

yolov3 608 量化 Resource exhausted: OOM

hanrb

新手上路

积分
46
发表于 2020-4-29 19:43:50    查看: 1486|回复: 4 | [复制链接]    打印 | 显示全部楼层
原来在1080ti 显卡上对yolov3 416 做量化, 可以正常进行,

然后对yolov3 608 量化后, 报异常Resource exhausted: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
帮忙看看, 谢谢

--> Building model
W:tensorflow:From /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W:tensorflow:From /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:1941: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
    tf.py_function, which takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
   
/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/scipy/ndimage/interpolation.py:611: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)
2020-04-29 19:35:11.595813: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.72GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-04-29 19:35:21.751863: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.41GiB.  Current allocation summary follows.
2020-04-29 19:35:21.753365: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *******_______________*********************************************_________________________________
2020-04-29 19:35:21.753420: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E Catch exception when building RKNN model!
E Traceback (most recent call last):
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
E     return fn(*args)
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
E     options, feed_dict, fetch_list, target_list, run_metadata)
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
E     run_metadata)
E tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E          [[{{node yolov3/darknet53_body/Conv/Conv2D_251_2/Add-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E          [[{{node yolov3/yolov3_head/Conv_14/BiasAdd_8_2/cond_1/Merge}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E During handling of the above exception, another exception occurred:
E Traceback (most recent call last):
E   File "rknn/api/rknn_base.py", line 737, in rknn.api.rknn_base.RKNNBase.build
E   File "rknn/api/rknn_base.py", line 1644, in rknn.api.rknn_base.RKNNBase._quantize2
E   File "rknn/base/RKNNlib/app/medusa/quantization.py", line 105, in rknn.base.RKNNlib.app.medusa.quantization.Quantization.run
E   File "rknn/base/RKNNlib/app/medusa/quantization.py", line 44, in rknn.base.RKNNlib.app.medusa.quantization.Quantization._run_quantization
E   File "rknn/base/RKNNlib/app/medusa/workspace.py", line 135, in rknn.base.RKNNlib.app.medusa.workspace.Workspace.run
E   File "rknn/base/RKNNlib/app/medusa/workspace.py", line 116, in rknn.base.RKNNlib.app.medusa.workspace.Workspace._run_iteration
E   File "rknn/base/RKNNlib/RKNN_session.py", line 30, in rknn.base.RKNNlib.RKNN_session.RKNNSession.run
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
E     run_metadata_ptr)
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
E     feed_dict_tensor, options, run_metadata)
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
E     run_metadata)
E   File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
E     raise type(e)(node_def, op, message)
E tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E          [[{{node yolov3/darknet53_body/Conv/Conv2D_251_2/Add-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E          [[node yolov3/yolov3_head/Conv_14/BiasAdd_8_2/cond_1/Merge (defined at /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/rknn/base/RKNNlib/dtype/quantized_dtype.py:112) ]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
done
E RKNN model data is None, please load model first!



回复

使用道具 举报

jefferyzhang

超级版主

积分
7776
发表于 2020-4-29 20:49:57 | 显示全部楼层
用CPU啊
回复

使用道具 举报

hanrb

新手上路

积分
46
 楼主| 发表于 2020-4-30 11:15:50 | 显示全部楼层

用cpu可以, 谢谢
回复

使用道具 举报

zwj1234

注册会员

积分
83
发表于 2020-6-5 14:53:33 | 显示全部楼层

为何我用cpu依然内存溢出错误
回复

使用道具 举报

gsq

新手上路

积分
30
发表于 2021-2-23 10:24:04 | 显示全部楼层

请问怎么设置参数,才能使用CPU进行量化
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

产品中心 购买渠道 开源社区 Wiki教程 资料下载 关于Toybrick


快速回复 返回顶部 返回列表