|
|
原来在1080ti 显卡上对yolov3 416 做量化, 可以正常进行,
然后对yolov3 608 量化后, 报异常Resource exhausted: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
帮忙看看, 谢谢
--> Building model
W:tensorflow:From /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W:tensorflow:From /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:1941: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
tf.py_function, which takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/scipy/ndimage/interpolation.py:611: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
"the returned array has changed.", UserWarning)
2020-04-29 19:35:11.595813: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.72GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-04-29 19:35:21.751863: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.41GiB. Current allocation summary follows.
2020-04-29 19:35:21.753365: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *******_______________*********************************************_________________________________
2020-04-29 19:35:21.753420: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E Catch exception when building RKNN model!
E Traceback (most recent call last):
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
E return fn(*args)
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
E options, feed_dict, fetch_list, target_list, run_metadata)
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
E run_metadata)
E tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E [[{{node yolov3/darknet53_body/Conv/Conv2D_251_2/Add-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E [[{{node yolov3/yolov3_head/Conv_14/BiasAdd_8_2/cond_1/Merge}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E During handling of the above exception, another exception occurred:
E Traceback (most recent call last):
E File "rknn/api/rknn_base.py", line 737, in rknn.api.rknn_base.RKNNBase.build
E File "rknn/api/rknn_base.py", line 1644, in rknn.api.rknn_base.RKNNBase._quantize2
E File "rknn/base/RKNNlib/app/medusa/quantization.py", line 105, in rknn.base.RKNNlib.app.medusa.quantization.Quantization.run
E File "rknn/base/RKNNlib/app/medusa/quantization.py", line 44, in rknn.base.RKNNlib.app.medusa.quantization.Quantization._run_quantization
E File "rknn/base/RKNNlib/app/medusa/workspace.py", line 135, in rknn.base.RKNNlib.app.medusa.workspace.Workspace.run
E File "rknn/base/RKNNlib/app/medusa/workspace.py", line 116, in rknn.base.RKNNlib.app.medusa.workspace.Workspace._run_iteration
E File "rknn/base/RKNNlib/RKNN_session.py", line 30, in rknn.base.RKNNlib.RKNN_session.RKNNSession.run
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
E run_metadata_ptr)
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
E feed_dict_tensor, options, run_metadata)
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
E run_metadata)
E File "/data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
E raise type(e)(node_def, op, message)
E tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[100,608,608,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E [[{{node yolov3/darknet53_body/Conv/Conv2D_251_2/Add-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E [[node yolov3/yolov3_head/Conv_14/BiasAdd_8_2/cond_1/Merge (defined at /data01/hanrb/virtualenv/edge_device36/lib/python3.6/site-packages/rknn/base/RKNNlib/dtype/quantized_dtype.py:112) ]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
done
E RKNN model data is None, please load model first!
|
|
闽公网安备 35010202001061号