Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues with CUDA OOM #134

Open
Sachin-Bhat opened this issue Sep 20, 2024 · 3 comments
Open

Issues with CUDA OOM #134

Sachin-Bhat opened this issue Sep 20, 2024 · 3 comments

Comments

@Sachin-Bhat
Copy link

Hello there,

Firstly I would like to thank you for creating this tool.I am getting the following error with CUDA and was hoping I could get some assistance in debugging the issue. I am running a RTX 3060 GPU on my Laptop and I am currently unsure if my device meets the minimum requirements to run on the GPU.

I am currently running Artix Linux in case the distro information is needed.

Namespace(pdf='1706.03762.pdf', output='output', batch_size=128, vis=False, render=False)
2024-09-20 11:06:26
Started!
CustomVisionEncoderDecoderModel init
CustomMBartForCausalLM init
CustomMBartDecoder init
�[32m[09/20 11:06:39 detectron2]: �[0mRank of current process: 0. World size: 1
�[32m[09/20 11:06:39 detectron2]: �[0mEnvironment info:
-------------------------------  ---------------------------------------------------------------------------------------
sys.platform                     linux
Python                           3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
numpy                            1.26.4
detectron2                       0.6 @/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/detectron2
Compiler                         GCC 11.4
CUDA compiler                    not available
DETECTRON2_ENV_MODULE            <not set>
PyTorch                          2.3.1+cu121 @/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch
PyTorch debug build              False
torch._C._GLIBCXX_USE_CXX11_ABI  False
GPU available                    Yes
GPU 0                            NVIDIA GeForce RTX 3060 Laptop GPU (arch=8.6)
Driver version                   560.35.03
CUDA_HOME                        /opt/cuda
Pillow                           8.4.0
torchvision                      0.18.1+cu121 @/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torchvision
torchvision arch flags           5.0, 6.0, 7.0, 7.5, 8.0, 8.6, 9.0
fvcore                           0.1.5.post20221221
iopath                           0.1.9
cv2                              4.6.0
-------------------------------  ---------------------------------------------------------------------------------------
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

�[32m[09/20 11:06:39 detectron2]: �[0mCommand line arguments: {'config_file': 'modules/layoutlmv3/layoutlmv3_base_inference.yaml', 'resume': False, 'eval_only': False, 'num_gpus': 1, 'num_machines': 1, 'machine_rank': 0, 'dist_url': 'tcp://127.0.0.1:57823', 'opts': ['MODEL.WEIGHTS', 'models/Layout/model_final.pth']}
�[32m[09/20 11:06:39 detectron2]: �[0mContents of args.config_file=modules/layoutlmv3/layoutlmv3_base_inference.yaml:
�[38;5;204mAUG�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mDETR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;204mCACHE_DIR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m~/cache/huggingface�[39m
�[38;5;204mCUDNN_BENCHMARK�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;204mDATALOADER�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mASPECT_RATIO_GROUPING�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m  �[39m�[38;5;204mFILTER_EMPTY_ANNOTATIONS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m  �[39m�[38;5;204mNUM_WORKERS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4�[39m
�[38;5;15m  �[39m�[38;5;204mREPEAT_THRESHOLD�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0�[39m
�[38;5;15m  �[39m�[38;5;204mSAMPLER_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mTrainingSampler�[39m
�[38;5;204mDATASETS�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mPRECOMPUTED_PROPOSAL_TOPK_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m  �[39m�[38;5;204mPRECOMPUTED_PROPOSAL_TOPK_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2000�[39m
�[38;5;15m  �[39m�[38;5;204mPROPOSAL_FILES_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;15m[�[39m�[38;5;15m]�[39m
�[38;5;15m  �[39m�[38;5;204mPROPOSAL_FILES_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;15m[�[39m�[38;5;15m]�[39m
�[38;5;15m  �[39m�[38;5;204mTEST�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mscihub_train�[39m
�[38;5;15m  �[39m�[38;5;204mTRAIN�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mscihub_train�[39m
�[38;5;204mGLOBAL�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mHACK�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;204mICDAR_DATA_DIR_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;204mICDAR_DATA_DIR_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;204mINPUT�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mCROP�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mSIZE�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m384�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m600�[39m
�[38;5;15m    �[39m�[38;5;204mTYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mabsolute_range�[39m
�[38;5;15m  �[39m�[38;5;204mFORMAT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mRGB�[39m
�[38;5;15m  �[39m�[38;5;204mMASK_FORMAT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mpolygon�[39m
�[38;5;15m  �[39m�[38;5;204mMAX_SIZE_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1333�[39m
�[38;5;15m  �[39m�[38;5;204mMAX_SIZE_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1333�[39m
�[38;5;15m  �[39m�[38;5;204mMIN_SIZE_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m800�[39m
�[38;5;15m  �[39m�[38;5;204mMIN_SIZE_TRAIN�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m480�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m544�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m576�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m608�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m640�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m672�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m704�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m736�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m768�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m800�[39m
�[38;5;15m  �[39m�[38;5;204mMIN_SIZE_TRAIN_SAMPLING�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mchoice�[39m
�[38;5;15m  �[39m�[38;5;204mRANDOM_FLIP�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mhorizontal�[39m
�[38;5;204mMODEL�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mANCHOR_GENERATOR�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mANGLES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m-90�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m90�[39m
�[38;5;15m    �[39m�[38;5;204mASPECT_RATIOS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m2.0�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mDefaultAnchorGenerator�[39m
�[38;5;15m    �[39m�[38;5;204mOFFSET�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0�[39m
�[38;5;15m    �[39m�[38;5;204mSIZES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m32�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m64�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m128�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m  �[39m�[38;5;204mBACKBONE�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mFREEZE_AT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mbuild_vit_fpn_backbone�[39m
�[38;5;15m  �[39m�[38;5;204mCONFIG_PATH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;15m  �[39m�[38;5;204mDEVICE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mcuda�[39m
�[38;5;15m  �[39m�[38;5;204mFPN�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mFUSE_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141msum�[39m
�[38;5;15m    �[39m�[38;5;204mIN_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer5�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer7�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer11�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;15m    �[39m�[38;5;204mOUT_CHANNELS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m  �[39m�[38;5;204mIMAGE_ONLY�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m  �[39m�[38;5;204mKEYPOINT_ON�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m  �[39m�[38;5;204mLOAD_PROPOSALS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m  �[39m�[38;5;204mMASK_ON�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m  �[39m�[38;5;204mMETA_ARCHITECTURE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mVLGeneralizedRCNN�[39m
�[38;5;15m  �[39m�[38;5;204mPANOPTIC_FPN�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mCOMBINE�[39m�[38;5;15m:�[39m
�[38;5;15m      �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m      �[39m�[38;5;204mINSTANCES_CONFIDENCE_THRESH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m      �[39m�[38;5;204mOVERLAP_THRESH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m      �[39m�[38;5;204mSTUFF_AREA_LIMIT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4096�[39m
�[38;5;15m    �[39m�[38;5;204mINSTANCE_LOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m  �[39m�[38;5;204mPIXEL_MEAN�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;204mPIXEL_STD�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m127.5�[39m
�[38;5;15m  �[39m�[38;5;204mPROPOSAL_GENERATOR�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mMIN_SIZE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mRPN�[39m
�[38;5;15m  �[39m�[38;5;204mRESNETS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mDEFORM_MODULATED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;204mDEFORM_NUM_GROUPS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mDEFORM_ON_PER_STAGE�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;204mDEPTH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m50�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mFrozenBN�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_GROUPS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mOUT_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mres4�[39m
�[38;5;15m    �[39m�[38;5;204mRES2_OUT_CHANNELS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m    �[39m�[38;5;204mRES5_DILATION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mSTEM_OUT_CHANNELS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m64�[39m
�[38;5;15m    �[39m�[38;5;204mSTRIDE_IN_1X1�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mWIDTH_PER_GROUP�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m64�[39m
�[38;5;15m  �[39m�[38;5;204mRETINANET�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_LOSS_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141msmooth_l1�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_WEIGHTS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mFOCAL_LOSS_ALPHA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.25�[39m
�[38;5;15m    �[39m�[38;5;204mFOCAL_LOSS_GAMMA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2.0�[39m
�[38;5;15m    �[39m�[38;5;204mIN_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp4�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp5�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp6�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp7�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_LABELS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m-1�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_THRESHOLDS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.4�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;204mNMS_THRESH_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CLASSES�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m10�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CONVS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4�[39m
�[38;5;15m    �[39m�[38;5;204mPRIOR_PROB�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.01�[39m
�[38;5;15m    �[39m�[38;5;204mSCORE_THRESH_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.05�[39m
�[38;5;15m    �[39m�[38;5;204mSMOOTH_L1_LOSS_BETA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.1�[39m
�[38;5;15m    �[39m�[38;5;204mTOPK_CANDIDATES_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m  �[39m�[38;5;204mROI_BOX_CASCADE_HEAD�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_WEIGHTS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m5.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m5.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m20.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m20.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m30.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m30.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m15.0�[39m
�[38;5;15m      �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m15.0�[39m
�[38;5;15m    �[39m�[38;5;204mIOUS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.6�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.7�[39m
�[38;5;15m  �[39m�[38;5;204mROI_BOX_HEAD�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_LOSS_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141msmooth_l1�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_LOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_WEIGHTS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m5.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m5.0�[39m
�[38;5;15m    �[39m�[38;5;204mCLS_AGNOSTIC_BBOX_REG�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mCONV_DIM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m    �[39m�[38;5;204mFC_DIM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1024�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mFastRCNNConvFCHead�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CONV�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_FC�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_RESOLUTION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m7�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_SAMPLING_RATIO�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mROIAlignV2�[39m
�[38;5;15m    �[39m�[38;5;204mSMOOTH_L1_BETA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0�[39m
�[38;5;15m    �[39m�[38;5;204mTRAIN_ON_PRED_BOXES�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m  �[39m�[38;5;204mROI_HEADS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mBATCH_SIZE_PER_IMAGE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;204mIN_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp2�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp4�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp5�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_LABELS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_THRESHOLDS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mCascadeROIHeads�[39m
�[38;5;15m    �[39m�[38;5;204mNMS_THRESH_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CLASSES�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m10�[39m
�[38;5;15m    �[39m�[38;5;204mPOSITIVE_FRACTION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.25�[39m
�[38;5;15m    �[39m�[38;5;204mPROPOSAL_APPEND_GT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mSCORE_THRESH_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.05�[39m
�[38;5;15m  �[39m�[38;5;204mROI_KEYPOINT_HEAD�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mCONV_DIMS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m512�[39m
�[38;5;15m    �[39m�[38;5;204mLOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mMIN_KEYPOINTS_PER_IMAGE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mKRCNNConvDeconvUpsampleHead�[39m
�[38;5;15m    �[39m�[38;5;204mNORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_KEYPOINTS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m17�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_RESOLUTION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m14�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_SAMPLING_RATIO�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mROIAlignV2�[39m
�[38;5;15m  �[39m�[38;5;204mROI_MASK_HEAD�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mCLS_AGNOSTIC_MASK�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;204mCONV_DIM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mMaskRCNNConvUpsampleHead�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;186m'�[39m�[38;5;186m'�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CONV�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_RESOLUTION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m14�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_SAMPLING_RATIO�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;204mPOOLER_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mROIAlignV2�[39m
�[38;5;15m  �[39m�[38;5;204mRPN�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mBATCH_SIZE_PER_IMAGE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m256�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_LOSS_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141msmooth_l1�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_LOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mBBOX_REG_WEIGHTS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mBOUNDARY_THRESH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m-1�[39m
�[38;5;15m    �[39m�[38;5;204mCONV_DIMS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m-1�[39m
�[38;5;15m    �[39m�[38;5;204mHEAD_NAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mStandardRPNHead�[39m
�[38;5;15m    �[39m�[38;5;204mIN_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp2�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp4�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp5�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp6�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_LABELS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m-1�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m    �[39m�[38;5;204mIOU_THRESHOLDS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m0.7�[39m
�[38;5;15m    �[39m�[38;5;204mLOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mNMS_THRESH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.7�[39m
�[38;5;15m    �[39m�[38;5;204mPOSITIVE_FRACTION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.5�[39m
�[38;5;15m    �[39m�[38;5;204mPOST_NMS_TOPK_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m    �[39m�[38;5;204mPOST_NMS_TOPK_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2000�[39m
�[38;5;15m    �[39m�[38;5;204mPRE_NMS_TOPK_TEST�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m    �[39m�[38;5;204mPRE_NMS_TOPK_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2000�[39m
�[38;5;15m    �[39m�[38;5;204mSMOOTH_L1_BETA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0�[39m
�[38;5;15m  �[39m�[38;5;204mSEM_SEG_HEAD�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mCOMMON_STRIDE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4�[39m
�[38;5;15m    �[39m�[38;5;204mCONVS_DIM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m128�[39m
�[38;5;15m    �[39m�[38;5;204mIGNORE_VALUE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m255�[39m
�[38;5;15m    �[39m�[38;5;204mIN_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp2�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp4�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mp5�[39m
�[38;5;15m    �[39m�[38;5;204mLOSS_WEIGHT�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mSemSegFPNHead�[39m
�[38;5;15m    �[39m�[38;5;204mNORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mGN�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_CLASSES�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m10�[39m
�[38;5;15m  �[39m�[38;5;204mVIT�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mDROP_PATH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.1�[39m
�[38;5;15m    �[39m�[38;5;204mIMG_SIZE�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m224�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m224�[39m
�[38;5;15m    �[39m�[38;5;204mNAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mlayoutlmv3_base�[39m
�[38;5;15m    �[39m�[38;5;204mOUT_FEATURES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer3�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer5�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer7�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141mlayer11�[39m
�[38;5;15m    �[39m�[38;5;204mPOS_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mabs�[39m
�[38;5;15m  �[39m�[38;5;204mWEIGHTS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m
�[38;5;204mOUTPUT_DIR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m
�[38;5;204mSCIHUB_DATA_DIR_TRAIN�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m~/publaynet/layout_scihub/train�[39m
�[38;5;204mSEED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m42�[39m
�[38;5;204mSOLVER�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mAMP�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m  �[39m�[38;5;204mBACKBONE_MULTIPLIER�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m  �[39m�[38;5;204mBASE_LR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0002�[39m
�[38;5;15m  �[39m�[38;5;204mBIAS_LR_FACTOR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m  �[39m�[38;5;204mCHECKPOINT_PERIOD�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2000�[39m
�[38;5;15m  �[39m�[38;5;204mCLIP_GRADIENTS�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mCLIP_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfull_model�[39m
�[38;5;15m    �[39m�[38;5;204mCLIP_VALUE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1.0�[39m
�[38;5;15m    �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mNORM_TYPE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2.0�[39m
�[38;5;15m  �[39m�[38;5;204mGAMMA�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.1�[39m
�[38;5;15m  �[39m�[38;5;204mGRADIENT_ACCUMULATION_STEPS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1�[39m
�[38;5;15m  �[39m�[38;5;204mIMS_PER_BATCH�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m32�[39m
�[38;5;15m  �[39m�[38;5;204mLR_SCHEDULER_NAME�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mWarmupCosineLR�[39m
�[38;5;15m  �[39m�[38;5;204mMAX_ITER�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m20000�[39m
�[38;5;15m  �[39m�[38;5;204mMOMENTUM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.9�[39m
�[38;5;15m  �[39m�[38;5;204mNESTEROV�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m  �[39m�[38;5;204mOPTIMIZER�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mADAMW�[39m
�[38;5;15m  �[39m�[38;5;204mREFERENCE_WORLD_SIZE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m
�[38;5;15m  �[39m�[38;5;204mSTEPS�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m10000�[39m
�[38;5;15m  �[39m�[38;5;204mWARMUP_FACTOR�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.01�[39m
�[38;5;15m  �[39m�[38;5;204mWARMUP_ITERS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m333�[39m
�[38;5;15m  �[39m�[38;5;204mWARMUP_METHOD�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mlinear�[39m
�[38;5;15m  �[39m�[38;5;204mWEIGHT_DECAY�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.05�[39m
�[38;5;15m  �[39m�[38;5;204mWEIGHT_DECAY_BIAS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mnull�[39m
�[38;5;15m  �[39m�[38;5;204mWEIGHT_DECAY_NORM�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0.0�[39m
�[38;5;204mTEST�[39m�[38;5;15m:�[39m
�[38;5;15m  �[39m�[38;5;204mAUG�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;204mFLIP�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mtrue�[39m
�[38;5;15m    �[39m�[38;5;204mMAX_SIZE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m4000�[39m
�[38;5;15m    �[39m�[38;5;204mMIN_SIZES�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m400�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m500�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m600�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m700�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m800�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m900�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1100�[39m
�[38;5;15m    �[39m�[38;5;15m-�[39m�[38;5;15m �[39m�[38;5;141m1200�[39m
�[38;5;15m  �[39m�[38;5;204mDETECTIONS_PER_IMAGE�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m100�[39m
�[38;5;15m  �[39m�[38;5;204mEVAL_PERIOD�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m1000�[39m
�[38;5;15m  �[39m�[38;5;204mEXPECTED_RESULTS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;15m[�[39m�[38;5;15m]�[39m
�[38;5;15m  �[39m�[38;5;204mKEYPOINT_OKS_SIGMAS�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;15m[�[39m�[38;5;15m]�[39m
�[38;5;15m  �[39m�[38;5;204mPRECISE_BN�[39m�[38;5;15m:�[39m
�[38;5;15m    �[39m�[38;5;204mENABLED�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141mfalse�[39m
�[38;5;15m    �[39m�[38;5;204mNUM_ITER�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m200�[39m
�[38;5;204mVERSION�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m2�[39m
�[38;5;204mVIS_PERIOD�[39m�[38;5;15m:�[39m�[38;5;15m �[39m�[38;5;141m0�[39m

�[32m[09/20 11:06:41 d2.checkpoint.detection_checkpoint]: �[0m[DetectionCheckpointer] Loading from models/Layout/model_final.pth ...
�[32m[09/20 11:06:41 fvcore.common.checkpoint]: �[0m[Checkpointer] Loading from models/Layout/model_final.pth ...
[2024/09/20 11:06:41] ppocr DEBUG: Namespace(help='==SUPPRESS==', use_gpu=True, use_xpu=False, use_npu=False, ir_optim=True, use_tensorrt=False, min_subgraph_size=15, precision='fp32', gpu_mem=500, gpu_id=0, image_dir=None, page_num=0, det_algorithm='DB', det_model_dir='/home/sachin/.paddleocr/whl/det/ch/ch_PP-OCRv4_det_infer', det_limit_side_len=960, det_limit_type='max', det_box_type='quad', det_db_thresh=0.3, det_db_box_thresh=0.3, det_db_unclip_ratio=1.5, max_batch_size=10, use_dilation=False, det_db_score_mode='fast', det_east_score_thresh=0.8, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_sast_score_thresh=0.5, det_sast_nms_thresh=0.2, det_pse_thresh=0, det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, scales=[8, 16, 32], alpha=1.0, beta=1.0, fourier_degree=5, rec_algorithm='SVTR_LCNet', rec_model_dir='/home/sachin/.paddleocr/whl/rec/ch/ch_PP-OCRv4_rec_infer', rec_image_inverse=True, rec_image_shape='3, 48, 320', rec_batch_num=6, max_text_length=25, rec_char_dict_path='/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/paddleocr/ppocr/utils/ppocr_keys_v1.txt', use_space_char=True, vis_font_path='./doc/fonts/simfang.ttf', drop_score=0.5, e2e_algorithm='PGNet', e2e_model_dir=None, e2e_limit_side_len=768, e2e_limit_type='max', e2e_pgnet_score_thresh=0.5, e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_pgnet_valid_set='totaltext', e2e_pgnet_mode='fast', use_angle_cls=False, cls_model_dir='/home/sachin/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_image_shape='3, 48, 192', label_list=['0', '180'], cls_batch_num=6, cls_thresh=0.9, enable_mkldnn=False, cpu_threads=10, use_pdserving=False, warmup=False, sr_model_dir=None, sr_image_shape='3, 32, 128', sr_batch_num=1, draw_img_save_dir='./inference_results', save_crop_res=False, crop_res_save_dir='./output', use_mp=False, total_process_num=1, process_id=0, benchmark=False, save_log_path='./log_output/', show_log=True, use_onnx=False, output='./output', table_max_len=488, table_algorithm='TableAttn', table_model_dir=None, merge_no_span_structure=True, table_char_dict_path=None, layout_model_dir=None, layout_dict_path=None, layout_score_threshold=0.5, layout_nms_threshold=0.5, kie_algorithm='LayoutXLM', ser_model_dir=None, re_model_dir=None, use_visual_backbone=True, ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ocr_order_method=None, mode='structure', image_orientation=False, layout=True, table=True, ocr=True, recovery=False, use_pdf2docx_api=False, invert=False, binarize=False, alphacolor=(255, 255, 255), lang='ch', det=True, rec=True, type='ocr', ocr_version='PP-OCRv4', structure_version='PP-StructureV2')
2024-09-20 11:06:26
Model init done!
total files: 1
pdf index: 0 pages: 15
Traceback (most recent call last):
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/pdf_extract.py", line 131, in <module>
    layout_res = layout_model(image, ignore_catids=[])
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/model_init.py", line 124, in __call__
    outputs = self.predictor(image)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/detectron2/engine/defaults.py", line 319, in __call__
    predictions = self.model([inputs])[0]
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/rcnn_vl.py", line 55, in forward
    return self.inference(batched_inputs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/rcnn_vl.py", line 113, in inference
    features = self.backbone(input)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/detectron2/modeling/backbone/fpn.py", line 139, in forward
    bottom_up_features = self.bottom_up(x)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/backbone.py", line 106, in forward
    return self.backbone.forward(
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 949, in forward
    encoder_outputs = self.encoder(
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 647, in forward
    layer_outputs = layer_module(
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 435, in forward
    self_attention_outputs = self.attention(
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 394, in forward
    self_outputs = self.self(
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sachin/miniforge3/envs/pdf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 335, in forward
    attention_probs = self.cogview_attn(attention_scores)  # to stablize training
  File "/home/sachin/Documents/FYP/pdf-extract-kit/content/PDF-Extract-Kit/modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py", line 271, in cogview_attn
    new_attention_scores = (scaled_attention_scores - max_value) * alpha
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB. GPU

Once again any assistance on this would be greatly appreciated.

Cheers,
Sachin
@wangbinDL
Copy link
Member

Hello @Sachin-Bhat

Your RTX 3060 GPU might be running out of memory due to the high batch size during inference. To resolve this, try reducing the batch size using the --batch-size parameter.

For example, start with a batch size of 16:

python pdf_extract.py --pdf assets/examples/example.pdf --batch-size 16

If you still encounter memory issues, further reduce the batch size:

python pdf_extract.py --pdf assets/examples/example.pdf --batch-size 8
python pdf_extract.py --pdf assets/examples/example.pdf --batch-size 4

This should help manage the GPU memory usage better.

@Sachin-Bhat
Copy link
Author

Hey @wangbinDL

Thank you for the prompt response. I did try what you said. However even setting the batch size to 1 did not fix the issues. So I am unsure if this is a bug. Please let me know if I can try something else to work around this issue.

Cheers,
Sachin

@myhloli
Copy link
Collaborator

myhloli commented Sep 21, 2024

Hey @wangbinDL

Thank you for the prompt response. I did try what you said. However even setting the batch size to 1 did not fix the issues. So I am unsure if this is a bug. Please let me know if I can try something else to work around this issue.

Cheers,

Sachin

Maybe you need more than 8GB vram to run this project.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@myhloli @Sachin-Bhat @wangbinDL