TAO PTM Inference using TAO-Deploy¶

Transfer learning is the process of transferring learned features from one application to another. It is a commonly used training technique where you use a model trained on one task and re-train to use it on a different task.

Train Adapt Optimize (TAO) Toolkit is a simple and easy-to-use Python based AI toolkit for taking purpose-built AI models and customizing them with users' own data.

Learning Objectives¶

In this notebook, you will learn how to leverage the simplicity and convenience of TAO to:

  • Download a TAO special use-case pre-trained model
  • Export the downloaded etlt model to TensorRT engine
  • Run inference on TRT engine file using TAO-Deploy
  • Visualize the inferences

Table of Contents¶

This notebook shows an example of classifying gestures using GestureNet in the Train Adapt Optimize (TAO) Toolkit.

  1. Get the TensorRT tar file
  2. Setup GPU environment
    2.1 Connect to GPU Instance
    2.2 Mounting Google drive
  3. Choose PTM model
  4. Untar TensorRT file if needed
  5. Install dependencies
  6. Download the etlt model of the ptm chosen
  7. Generate TRT engine file for the ptm's etlt file
  8. Run inference on TRT engine file
  9. Visualize inference

FIXME¶

  1. ptm_model_name - set this to one of the available PTM models
  2. trt_tar_path - set this path of the uploaded TensorRT tar.gz file after browser download
  3. trt_untar_folder_path - set to path of the folder where the TensoRT tar.gz file has to be untarred into
  4. trt_version - set this to the version of TRT you have downloaded
  5. COLAB_NOTEBOOKS_PATH - set this to the path where the repo is to be cloned/repo is already downloaded to
  6. ptm_download_folder: set the folder path where you want to download the file into
  7. (Optional FIXME) model_to_download_map(dictionary_value for chosen PTM key) - change the version tag of the PTM model if you want a version different from the below defaults
  8. data_type - choose between fp32 and fp16
  9. trt_out_folder - set this to the output folder for TensorRT engine file writing
  10. inference_out_folder - set this to the output folder to write the inference results to
  11. inference_input_images_folder - set this to the folder path containing images to run the inference on

1. Get the TensorRT tar file¶

  1. Visit https://developer.nvidia.com/tensorrt
  2. Clicking Download now from step one directs you to https://developer.nvidia.com/nvidia-tensorrt-download where you have to Login/Join Now for Nvidia Developer Program Membership
  3. Now, in the download page: Choose TensorRT 8 in available versions
  4. Agree to Terms and Conditions
  5. Click on TensorRT 8.5 GA to expand the available options
  6. Click on 'TensorRT 8.5 GA for Linux x86_64 and CUDA 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7 and 11.8 TAR Package' to dowload the TAR file
  7. Upload the the tar file to your Google Drive

2. Setup GPU environment ¶

2.1 Connect to GPU Instance ¶

  1. Move any data saved to the Colab Instance storage to Google Drive
  2. Change Runtime type to GPU by Runtime(Top Left tab)->Change Runtime Type->GPU(Hardware Accelerator)
  3. Then click on Connect (Top Right)

2.2 Mounting Google drive ¶

Mount your Google drive storage to this Colab instance

In [1]:
try:
    import google.colab
    %env GOOGLE_COLAB=1
    from google.colab import drive
    drive.mount('/content/drive', force_remount=True)
except:
    %env GOOGLE_COLAB=0
    print("Warning: Not a Colab Environment")

env: GOOGLE_COLAB=1
Mounted at /content/drive

3. Choose PTM model ¶

In [2]:
# Define model_name
# Available models (#FIXME 1):
# 1. PeopleNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet
# 2. PeopleSemSegNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesemsegnet
# 3. TrafficCamNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet
# 4. DashCamNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet
# 5. FaceDetectIR - https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir
# 6. FaceDetect - https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetect
# 7. VehicleMakeNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet
# 8. VehicleTypeNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet
# 9. LicensePlateDetection - https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet
# 10. LicensePlateRecognition - https://ngc.nvidia.com/catalog/models/nvidia:tao:lprnet

ptm_model_name = "PeopleNet" # FIXME1 (Add the model name from the above mentioned list)

4. Untar TensorRT tar file if needed ¶

  1. Set the variable trt_tarfile_path to the path where you uploaded the TensorRT tar file
  2. Set the folder you want to untar into in trt_untar_folder_path
  3. Set the TRT version you have downloaded in trt_version
    • If the TRT tar file's name is TensorRT-8.5.1.7.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz, then the version is 8.5.1.7
In [4]:
# FIXME 2: set this path of the uploaded TensorRT tar.gz file after browser download
trt_tar_path="/content/drive/MyDrive/TensorRT-8.5.3.1.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz"

import os
if not os.path.exists(trt_tar_path):
  raise Exception("TAR file not found in the provided path")

# FIXME 3: set to path of the folder where the TensoRT tar.gz file has to be untarred into
%env trt_untar_folder_path=/content/drive/MyDrive/trt_untar
# FIXME 4: set this to the version of TRT you have downloaded
%env trt_version=8.5.3.1

!mkdir -p $trt_untar_folder_path

import os

untar = True
for fname in os.listdir(os.environ.get("trt_untar_folder_path", None)):
  if fname.startswith("TensorRT-"+os.environ.get("trt_version")) and not fname.endswith(".tar.gz"):
    untar = False

if untar:
  !tar -xzf $trt_tar_path -C $trt_untar_folder_path

env: trt_untar_folder_path=/content/drive/MyDrive/trt_untar
env: trt_version=8.5.3.1
In [5]:
if not os.path.exists(f'{os.environ.get("trt_untar_folder_path")}/TensorRT-{os.environ.get("trt_version")}'):
  raise Exception("TensorRT not untarred properly. Please download and untar properly")

5. Install dependencies ¶

In [6]:
#FIXME 5 - COLAB_NOTEBOOKS_PATH: set this to the path where the repo is to be cloned/repo is already downloaded to
%env COLAB_NOTEBOOKS_PATH=/content/drive/MyDrive/nvidia-tao

if os.environ["GOOGLE_COLAB"] == "1":
    if not os.path.exists(os.path.join(os.environ["COLAB_NOTEBOOKS_PATH"])):
        !git clone https://github.com/NVIDIA-AI-IOT/nvidia-tao.git $COLAB_NOTEBOOKS_PATH
else:
    if not os.path.exists(os.environ["COLAB_NOTEBOOKS_PATH"]):
        raise Exception("Error, enter the path of the colab notebooks repo correctly")

!sed -i "s|PATH_TO_TRT|$trt_untar_folder_path|g" $COLAB_NOTEBOOKS_PATH/tao_deploy/setup_env_colab.sh
!sed -i "s|TRT_VERSION|$trt_version|g" $COLAB_NOTEBOOKS_PATH/tao_deploy/setup_env_colab.sh

!sh $COLAB_NOTEBOOKS_PATH/tao_deploy/setup_env_colab.sh

env: COLAB_NOTEBOOKS_PATH=/content/drive/MyDrive/nvidia-tao
Get:1 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu focal InRelease [18.1 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:4 http://ppa.launchpad.net/cran/libgit2/ubuntu focal InRelease
Get:5 https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/ InRelease [3,622 B]
Hit:6 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Get:9 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu focal InRelease [24.3 kB]
Hit:10 http://ppa.launchpad.net/ubuntugis/ppa/ubuntu focal InRelease
Get:11 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:12 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu focal/main Sources [2,573 kB]
Get:13 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu focal/main amd64 Packages [1,213 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,158 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,343 kB]
Get:16 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu focal/main amd64 Packages [46.8 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,681 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,046 kB]
Fetched 12.4 MB in 2s (6,282 kB/s)
Reading package lists... Done
Hit:1 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu focal InRelease
Hit:2 http://ppa.launchpad.net/cran/libgit2/ubuntu focal InRelease
Hit:3 https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/ InRelease
Hit:4 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease
Hit:5 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu focal InRelease
Hit:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Hit:7 http://ppa.launchpad.net/ubuntugis/ppa/ubuntu focal InRelease
Hit:8 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:9 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:10 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:11 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3.8 is already the newest version (3.8.10-0ubuntu1~20.04.7).
python3.8 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  python-pip-whl python3-setuptools python3-wheel
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  python-pip-whl python3-pip python3-setuptools python3-wheel
0 upgraded, 4 newly installed, 0 to remove and 24 not upgraded.
Need to get 2,389 kB of archives.
After this operation, 4,933 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.8 [1,805 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-wheel all 0.34.2-1ubuntu0.1 [23.9 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.8 [231 kB]
Fetched 2,389 kB in 1s (2,699 kB/s)
Selecting previously unselected package python-pip-whl.
(Reading database ... 122519 files and directories currently installed.)
Preparing to unpack .../python-pip-whl_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python-pip-whl (20.0.2-5ubuntu1.8) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../python3-setuptools_45.2.0-1ubuntu0.1_all.deb ...
Unpacking python3-setuptools (45.2.0-1ubuntu0.1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../python3-wheel_0.34.2-1ubuntu0.1_all.deb ...
Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../python3-pip_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python3-pip (20.0.2-5ubuntu1.8) ...
Setting up python3-setuptools (45.2.0-1ubuntu0.1) ...
Setting up python3-wheel (0.34.2-1ubuntu0.1) ...
Setting up python-pip-whl (20.0.2-5ubuntu1.8) ...
Setting up python3-pip (20.0.2-5ubuntu1.8) ...
Processing triggers for man-db (2.9.1-1) ...
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'python3-distutils' instead of 'python3.8-distutils'
python3-distutils is already the newest version (3.8.10-0ubuntu1~20.04).
python3-distutils set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3.8-dev is already the newest version (3.8.10-0ubuntu1~20.04.7).
python3.8-dev set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting pip
  Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 13.4 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-23.1.2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting cython
  Downloading Cython-0.29.34-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 54.5 MB/s eta 0:00:00
Installing collected packages: cython
Successfully installed cython-0.29.34
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting nvidia-ml-py
  Downloading nvidia_ml_py-11.525.112-py3-none-any.whl (35 kB)
Installing collected packages: nvidia-ml-py
Successfully installed nvidia-ml-py-11.525.112
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting nvidia-pyindex
  Downloading nvidia-pyindex-1.0.9.tar.gz (10 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: nvidia-pyindex
  Building wheel for nvidia-pyindex (setup.py) ... done
  Created wheel for nvidia-pyindex: filename=nvidia_pyindex-1.0.9-py3-none-any.whl size=8398 sha256=cb3bd386678208dd4e1a85e5064058a09046af6d85b501bae791fb7ca0d45245
  Stored in directory: /root/.cache/pip/wheels/e0/c2/fb/5cf4e1cfaf28007238362cb746fb38fc2dd76348331a748d54
Successfully built nvidia-pyindex
Installing collected packages: nvidia-pyindex
Successfully installed nvidia-pyindex-1.0.9
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (45.2.0)
Collecting setuptools
  Downloading setuptools-67.7.2-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 39.8 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 45.2.0
    Uninstalling setuptools-45.2.0:
      Successfully uninstalled setuptools-45.2.0
Successfully installed setuptools-67.7.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Collecting pycuda==2020.1
  Downloading pycuda-2020.1.tar.gz (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 49.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pytools>=2011.2 (from pycuda==2020.1)
  Downloading pytools-2022.1.14.tar.gz (74 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.6/74.6 kB 10.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting decorator>=3.2.0 (from pycuda==2020.1)
  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting appdirs>=1.4.0 (from pycuda==2020.1)
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting mako (from pycuda==2020.1)
  Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 10.1 MB/s eta 0:00:00
Collecting platformdirs>=2.2.0 (from pytools>=2011.2->pycuda==2020.1)
  Downloading platformdirs-3.5.1-py3-none-any.whl (15 kB)
Collecting typing_extensions>=4.0 (from pytools>=2011.2->pycuda==2020.1)
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting MarkupSafe>=0.9.2 (from mako->pycuda==2020.1)
  Downloading MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Building wheels for collected packages: pycuda, pytools
  Building wheel for pycuda (setup.py) ... done
  Created wheel for pycuda: filename=pycuda-2020.1-cp38-cp38-linux_x86_64.whl size=651027 sha256=cad3cdc929edbdb20e402489c35ec0d2cbd74e88194baf4f515827f885642b4a
  Stored in directory: /root/.cache/pip/wheels/0d/9e/20/bdd7c08a85b14ccaeea840e9d3f010354c92bdd5a77d935f7f
  Building wheel for pytools (setup.py) ... done
  Created wheel for pytools: filename=pytools-2022.1.14-py2.py3-none-any.whl size=69854 sha256=c50f24d2bbfe8b162cf75a2ae453b0e07a4f58660d0d4bd14b346f9ba334eaf1
  Stored in directory: /root/.cache/pip/wheels/cb/fc/a9/1e7e56fe02d7f58eaff555f22e79d4fc2d817012291254bae2
Successfully built pycuda pytools
Installing collected packages: appdirs, typing_extensions, platformdirs, MarkupSafe, decorator, pytools, mako, pycuda
Successfully installed MarkupSafe-2.1.2 appdirs-1.4.4 decorator-5.1.1 mako-1.2.4 platformdirs-3.5.1 pycuda-2020.1 pytools-2022.1.14 typing_extensions-4.5.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Collecting nvidia-eff-tao-encryption==0.1.7
  Downloading nvidia_eff_tao_encryption-0.1.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 35.7 MB/s eta 0:00:00
Collecting cryptography (from nvidia-eff-tao-encryption==0.1.7)
  Downloading cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl (3.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 63.5 MB/s eta 0:00:00
Collecting pybind11>=2.6 (from nvidia-eff-tao-encryption==0.1.7)
  Downloading pybind11-2.10.4-py3-none-any.whl (222 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.3/222.3 kB 27.6 MB/s eta 0:00:00
Collecting pyarmor (from nvidia-eff-tao-encryption==0.1.7)
  Downloading pyarmor-8.2.0-py2.py3-none-any.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 77.2 MB/s eta 0:00:00
Collecting pyinstaller (from nvidia-eff-tao-encryption==0.1.7)
  Downloading pyinstaller-5.11.0-py3-none-manylinux2014_x86_64.whl (653 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 653.4/653.4 kB 45.9 MB/s eta 0:00:00
Collecting black (from nvidia-eff-tao-encryption==0.1.7)
  Downloading black-23.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 84.8 MB/s eta 0:00:00
Collecting isort[requirements]<5 (from nvidia-eff-tao-encryption==0.1.7)
  Downloading isort-4.3.21-py2.py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.3/42.3 kB 5.4 MB/s eta 0:00:00
Collecting pipreqs (from isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading pipreqs-0.4.13-py2.py3-none-any.whl (33 kB)
Collecting pip-api (from isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading pip_api-0.0.30-py3-none-any.whl (111 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.6/111.6 kB 16.8 MB/s eta 0:00:00
Collecting click>=8.0.0 (from black->nvidia-eff-tao-encryption==0.1.7)
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 14.6 MB/s eta 0:00:00
Collecting mypy-extensions>=0.4.3 (from black->nvidia-eff-tao-encryption==0.1.7)
  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting packaging>=22.0 (from black->nvidia-eff-tao-encryption==0.1.7)
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 7.0 MB/s eta 0:00:00
Collecting pathspec>=0.9.0 (from black->nvidia-eff-tao-encryption==0.1.7)
  Downloading pathspec-0.11.1-py3-none-any.whl (29 kB)
Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff-tao-encryption==0.1.7) (3.5.1)
Collecting tomli>=1.1.0 (from black->nvidia-eff-tao-encryption==0.1.7)
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Requirement already satisfied: typing-extensions>=3.10.0.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff-tao-encryption==0.1.7) (4.5.0)
Collecting cffi>=1.12 (from cryptography->nvidia-eff-tao-encryption==0.1.7)
  Downloading cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.7/442.7 kB 42.4 MB/s eta 0:00:00
Collecting pyarmor.cli.core~=3.2.0 (from pyarmor->nvidia-eff-tao-encryption==0.1.7)
  Downloading pyarmor.cli.core-3.2.0-cp38-none-manylinux1_x86_64.whl (714 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 714.0/714.0 kB 65.8 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=42.0.0 in /usr/local/lib/python3.8/dist-packages (from pyinstaller->nvidia-eff-tao-encryption==0.1.7) (67.7.2)
Collecting altgraph (from pyinstaller->nvidia-eff-tao-encryption==0.1.7)
  Downloading altgraph-0.17.3-py2.py3-none-any.whl (21 kB)
Collecting pyinstaller-hooks-contrib>=2021.4 (from pyinstaller->nvidia-eff-tao-encryption==0.1.7)
  Downloading pyinstaller_hooks_contrib-2023.3-py2.py3-none-any.whl (263 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 263.6/263.6 kB 32.6 MB/s eta 0:00:00
Collecting pycparser (from cffi>=1.12->cryptography->nvidia-eff-tao-encryption==0.1.7)
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 18.6 MB/s eta 0:00:00
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (from pip-api->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7) (23.1.2)
Collecting docopt (from pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
Collecting yarg (from pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading yarg-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting requests (from yarg->pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading requests-2.30.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 kB 9.5 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 195.9/195.9 kB 20.5 MB/s eta 0:00:00
Collecting idna<4,>=2.5 (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 9.1 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading urllib3-2.0.2-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.2/123.2 kB 18.6 MB/s eta 0:00:00
Collecting certifi>=2017.4.17 (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff-tao-encryption==0.1.7)
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB 23.1 MB/s eta 0:00:00
Building wheels for collected packages: docopt
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13706 sha256=dbc40bea630cb74c07076a1327867e7223d9d9ebf354fbec287e6b0fcdc30002
  Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c
Successfully built docopt
Installing collected packages: pyarmor.cli.core, docopt, altgraph, urllib3, tomli, pyinstaller-hooks-contrib, pycparser, pybind11, pyarmor, pip-api, pathspec, packaging, mypy-extensions, isort, idna, click, charset-normalizer, certifi, requests, pyinstaller, cffi, black, yarg, cryptography, pipreqs, nvidia-eff-tao-encryption
Successfully installed altgraph-0.17.3 black-23.3.0 certifi-2023.5.7 cffi-1.15.1 charset-normalizer-3.1.0 click-8.1.3 cryptography-40.0.2 docopt-0.6.2 idna-3.4 isort-4.3.21 mypy-extensions-1.0.0 nvidia-eff-tao-encryption-0.1.7 packaging-23.1 pathspec-0.11.1 pip-api-0.0.30 pipreqs-0.4.13 pyarmor-8.2.0 pyarmor.cli.core-3.2.0 pybind11-2.10.4 pycparser-2.21 pyinstaller-5.11.0 pyinstaller-hooks-contrib-2023.3 requests-2.30.0 tomli-2.0.1 urllib3-2.0.2 yarg-0.1.9
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Collecting nvidia-eff==0.6.2
  Downloading nvidia_eff-0.6.2-py38-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (673 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 673.7/673.7 kB 22.9 MB/s eta 0:00:00
Collecting ruamel.yaml (from nvidia-eff==0.6.2)
  Downloading ruamel.yaml-0.17.26-py3-none-any.whl (109 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.1/109.1 kB 16.2 MB/s eta 0:00:00
Requirement already satisfied: pyarmor in /usr/local/lib/python3.8/dist-packages (from nvidia-eff==0.6.2) (8.2.0)
Requirement already satisfied: pyinstaller in /usr/local/lib/python3.8/dist-packages (from nvidia-eff==0.6.2) (5.11.0)
Collecting wrapt (from nvidia-eff==0.6.2)
  Downloading wrapt-1.15.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (81 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.5/81.5 kB 12.2 MB/s eta 0:00:00
Requirement already satisfied: black in /usr/local/lib/python3.8/dist-packages (from nvidia-eff==0.6.2) (23.3.0)
Requirement already satisfied: isort[requirements]<5 in /usr/local/lib/python3.8/dist-packages (from nvidia-eff==0.6.2) (4.3.21)
Requirement already satisfied: pipreqs in /usr/local/lib/python3.8/dist-packages (from isort[requirements]<5->nvidia-eff==0.6.2) (0.4.13)
Requirement already satisfied: pip-api in /usr/local/lib/python3.8/dist-packages (from isort[requirements]<5->nvidia-eff==0.6.2) (0.0.30)
Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (8.1.3)
Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (1.0.0)
Requirement already satisfied: packaging>=22.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (23.1)
Requirement already satisfied: pathspec>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (0.11.1)
Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (3.5.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (2.0.1)
Requirement already satisfied: typing-extensions>=3.10.0.0 in /usr/local/lib/python3.8/dist-packages (from black->nvidia-eff==0.6.2) (4.5.0)
Requirement already satisfied: pyarmor.cli.core~=3.2.0 in /usr/local/lib/python3.8/dist-packages (from pyarmor->nvidia-eff==0.6.2) (3.2.0)
Requirement already satisfied: setuptools>=42.0.0 in /usr/local/lib/python3.8/dist-packages (from pyinstaller->nvidia-eff==0.6.2) (67.7.2)
Requirement already satisfied: altgraph in /usr/local/lib/python3.8/dist-packages (from pyinstaller->nvidia-eff==0.6.2) (0.17.3)
Requirement already satisfied: pyinstaller-hooks-contrib>=2021.4 in /usr/local/lib/python3.8/dist-packages (from pyinstaller->nvidia-eff==0.6.2) (2023.3)
Collecting ruamel.yaml.clib>=0.2.7 (from ruamel.yaml->nvidia-eff==0.6.2)
  Downloading ruamel.yaml.clib-0.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (555 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 555.3/555.3 kB 45.1 MB/s eta 0:00:00
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (from pip-api->isort[requirements]<5->nvidia-eff==0.6.2) (23.1.2)
Requirement already satisfied: docopt in /usr/local/lib/python3.8/dist-packages (from pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (0.6.2)
Requirement already satisfied: yarg in /usr/local/lib/python3.8/dist-packages (from pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (0.1.9)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from yarg->pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (2.30.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (2.0.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->yarg->pipreqs->isort[requirements]<5->nvidia-eff==0.6.2) (2023.5.7)
Installing collected packages: wrapt, ruamel.yaml.clib, ruamel.yaml, nvidia-eff
Successfully installed nvidia-eff-0.6.2 ruamel.yaml-0.17.26 ruamel.yaml.clib-0.2.7 wrapt-1.15.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Requirement already satisfied: cffi in /usr/local/lib/python3.8/dist-packages (1.15.1)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi) (2.21)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Processing ./drive/MyDrive/trt_untar/TensorRT-8.5.3.1/python/tensorrt-8.5.3.1-cp38-none-linux_x86_64.whl
Installing collected packages: tensorrt
Successfully installed tensorrt-8.5.3.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Processing ./drive/MyDrive/trt_untar/TensorRT-8.5.3.1/onnx_graphsurgeon/onnx_graphsurgeon-0.3.12-py2.py3-none-any.whl
Collecting numpy (from onnx-graphsurgeon==0.3.12)
  Using cached numpy-1.24.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting onnx (from onnx-graphsurgeon==0.3.12)
  Downloading onnx-1.14.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.6/14.6 MB 86.6 MB/s eta 0:00:00
Collecting protobuf>=3.20.2 (from onnx->onnx-graphsurgeon==0.3.12)
  Downloading protobuf-4.23.0-cp37-abi3-manylinux2014_x86_64.whl (304 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 304.5/304.5 kB 37.4 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.8/dist-packages (from onnx->onnx-graphsurgeon==0.3.12) (4.5.0)
Installing collected packages: protobuf, numpy, onnx, onnx-graphsurgeon
Successfully installed numpy-1.24.3 onnx-1.14.0 onnx-graphsurgeon-0.3.12 protobuf-4.23.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Processing ./drive/MyDrive/trt_untar/TensorRT-8.5.3.1/graphsurgeon/graphsurgeon-0.4.6-py2.py3-none-any.whl
Installing collected packages: graphsurgeon
Successfully installed graphsurgeon-0.4.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Processing ./drive/MyDrive/trt_untar/TensorRT-8.5.3.1/uff/uff-0.6.9-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.8/dist-packages (from uff==0.6.9) (1.24.3)
Requirement already satisfied: protobuf>=3.3.0 in /usr/local/lib/python3.8/dist-packages (from uff==0.6.9) (4.23.0)
Installing collected packages: uff
Successfully installed uff-0.6.9
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Collecting nvidia-tao-deploy==4.0.0.1
  Downloading nvidia_tao_deploy-4.0.0.1-py3-none-any.whl (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 11.5 MB/s eta 0:00:00
Collecting Pillow<9.0.0,>=8.1.0 (from nvidia-tao-deploy==4.0.0.1)
  Downloading Pillow-8.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 104.3 MB/s eta 0:00:00
Collecting PyYAML>=5.1 (from nvidia-tao-deploy==4.0.0.1)
  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 701.2/701.2 kB 63.8 MB/s eta 0:00:00
Collecting absl-py>=0.7.1 (from nvidia-tao-deploy==4.0.0.1)
  Downloading absl_py-1.4.0-py3-none-any.whl (126 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.5/126.5 kB 19.1 MB/s eta 0:00:00
Collecting h5py==3.7.0 (from nvidia-tao-deploy==4.0.0.1)
  Downloading h5py-3.7.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 97.5 MB/s eta 0:00:00
Collecting hydra-core==1.2.0 (from nvidia-tao-deploy==4.0.0.1)
  Downloading hydra_core-1.2.0-py3-none-any.whl (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.1/151.1 kB 22.6 MB/s eta 0:00:00
Collecting matplotlib>=3.0.3 (from nvidia-tao-deploy==4.0.0.1)
  Downloading matplotlib-3.7.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.2/9.2 MB 114.2 MB/s eta 0:00:00
Collecting mpi4py>=3.0.3 (from nvidia-tao-deploy==4.0.0.1)
  Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 80.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting omegaconf==2.2.2 (from nvidia-tao-deploy==4.0.0.1)
  Downloading omegaconf-2.2.2-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.1/79.1 kB 12.4 MB/s eta 0:00:00
Requirement already satisfied: onnx in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy==4.0.0.1) (1.14.0)
Collecting onnxruntime (from nvidia-tao-deploy==4.0.0.1)
  Downloading onnxruntime-1.14.1-cp38-cp38-manylinux_2_27_x86_64.whl (5.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 100.8 MB/s eta 0:00:00
Collecting opencv-python (from nvidia-tao-deploy==4.0.0.1)
  Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.8/61.8 MB 10.3 MB/s eta 0:00:00
Collecting protobuf==3.20.1 (from nvidia-tao-deploy==4.0.0.1)
  Downloading protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 76.2 MB/s eta 0:00:00
Collecting pycocotools-fix (from nvidia-tao-deploy==4.0.0.1)
  Downloading pycocotools-fix-2.0.0.9.tar.gz (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.0/124.0 kB 16.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pynvml==11.0.0 (from nvidia-tao-deploy==4.0.0.1)
  Downloading pynvml-11.0.0-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.1/46.1 kB 6.9 MB/s eta 0:00:00
Collecting scikit-image==0.17.2 (from nvidia-tao-deploy==4.0.0.1)
  Downloading scikit_image-0.17.2-cp38-cp38-manylinux1_x86_64.whl (12.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.4/12.4 MB 94.8 MB/s eta 0:00:00
Collecting scikit-learn==0.24.2 (from nvidia-tao-deploy==4.0.0.1)
  Downloading scikit_learn-0.24.2-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.9/24.9 MB 67.9 MB/s eta 0:00:00
Collecting scipy==1.5.4 (from nvidia-tao-deploy==4.0.0.1)
  Downloading scipy-1.5.4-cp38-cp38-manylinux1_x86_64.whl (25.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.8/25.8 MB 68.0 MB/s eta 0:00:00
Collecting seaborn==0.7.1 (from nvidia-tao-deploy==4.0.0.1)
  Downloading seaborn-0.7.1.tar.gz (158 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.1/158.1 kB 23.3 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting six>=1.12.0 (from nvidia-tao-deploy==4.0.0.1)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tqdm==4.64.0 (from nvidia-tao-deploy==4.0.0.1)
  Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 11.9 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.8/dist-packages (from h5py==3.7.0->nvidia-tao-deploy==4.0.0.1) (1.24.3)
Collecting antlr4-python3-runtime==4.9.* (from hydra-core==1.2.0->nvidia-tao-deploy==4.0.0.1)
  Downloading antlr4-python3-runtime-4.9.3.tar.gz (117 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.0/117.0 kB 14.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy==4.0.0.1) (23.1)
Collecting importlib-resources (from hydra-core==1.2.0->nvidia-tao-deploy==4.0.0.1)
  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting networkx>=2.0 (from scikit-image==0.17.2->nvidia-tao-deploy==4.0.0.1)
  Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 68.7 MB/s eta 0:00:00
Collecting imageio>=2.3.0 (from scikit-image==0.17.2->nvidia-tao-deploy==4.0.0.1)
  Downloading imageio-2.28.1-py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 75.5 MB/s eta 0:00:00
Collecting tifffile>=2019.7.26 (from scikit-image==0.17.2->nvidia-tao-deploy==4.0.0.1)
  Downloading tifffile-2023.4.12-py3-none-any.whl (219 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 219.4/219.4 kB 25.0 MB/s eta 0:00:00
Collecting PyWavelets>=1.1.1 (from scikit-image==0.17.2->nvidia-tao-deploy==4.0.0.1)
  Downloading PyWavelets-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 65.0 MB/s eta 0:00:00
Collecting joblib>=0.11 (from scikit-learn==0.24.2->nvidia-tao-deploy==4.0.0.1)
  Downloading joblib-1.2.0-py3-none-any.whl (297 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.0/298.0 kB 37.5 MB/s eta 0:00:00
Collecting threadpoolctl>=2.0.0 (from scikit-learn==0.24.2->nvidia-tao-deploy==4.0.0.1)
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting pandas (from seaborn==0.7.1->nvidia-tao-deploy==4.0.0.1)
  Downloading pandas-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 101.7 MB/s eta 0:00:00
Collecting contourpy>=1.0.1 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached contourpy-1.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
Collecting cycler>=0.10 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached fonttools-4.39.4-py3-none-any.whl (1.0 MB)
Collecting kiwisolver>=1.0.1 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting pyparsing>=2.3.1 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7 (from matplotlib>=3.0.3->nvidia-tao-deploy==4.0.0.1)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
INFO: pip is looking at multiple versions of onnx to determine which version is compatible with other requirements. This could take a while.
Collecting onnx (from nvidia-tao-deploy==4.0.0.1)
  Downloading onnx-1.13.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 80.3 MB/s eta 0:00:00
  Downloading onnx-1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 65.9 MB/s eta 0:00:00
  Downloading onnx-1.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.1/13.1 MB 84.8 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.8/dist-packages (from onnx->nvidia-tao-deploy==4.0.0.1) (4.5.0)
Collecting coloredlogs (from onnxruntime->nvidia-tao-deploy==4.0.0.1)
  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 6.5 MB/s eta 0:00:00
Collecting flatbuffers (from onnxruntime->nvidia-tao-deploy==4.0.0.1)
  Downloading flatbuffers-23.5.9-py2.py3-none-any.whl (26 kB)
Collecting sympy (from onnxruntime->nvidia-tao-deploy==4.0.0.1)
  Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 84.6 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy==4.0.0.1) (67.7.2)
Requirement already satisfied: cython>=0.27.3 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy==4.0.0.1) (0.29.34)
Collecting zipp>=3.1.0 (from importlib-resources->hydra-core==1.2.0->nvidia-tao-deploy==4.0.0.1)
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime->nvidia-tao-deploy==4.0.0.1)
  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 14.0 MB/s eta 0:00:00
Collecting pytz>=2020.1 (from pandas->seaborn==0.7.1->nvidia-tao-deploy==4.0.0.1)
  Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.3/502.3 kB 54.8 MB/s eta 0:00:00
Collecting tzdata>=2022.1 (from pandas->seaborn==0.7.1->nvidia-tao-deploy==4.0.0.1)
  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 kB 39.1 MB/s eta 0:00:00
Collecting mpmath>=0.19 (from sympy->onnxruntime->nvidia-tao-deploy==4.0.0.1)
  Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 58.6 MB/s eta 0:00:00
Building wheels for collected packages: seaborn, antlr4-python3-runtime, mpi4py, pycocotools-fix
  Building wheel for seaborn (setup.py) ... done
  Created wheel for seaborn: filename=seaborn-0.7.1-py3-none-any.whl size=165931 sha256=d988941a50b36de022e6ed42b90c981bdcc23bdf38530762c41e6dbdae022f55
  Stored in directory: /root/.cache/pip/wheels/97/14/28/123fdaafb903da8a74ad19826a2e800903d5ad8c5fd3be68e1
  Building wheel for antlr4-python3-runtime (setup.py) ... done
  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144553 sha256=36cffb9882ea98115440aed0e7bea3cfca8ec5882dc09782b80607d35a8f50be
  Stored in directory: /root/.cache/pip/wheels/b1/a3/c2/6df046c09459b73cc9bb6c4401b0be6c47048baf9a1617c485
  Building wheel for mpi4py (pyproject.toml) ... done
  Created wheel for mpi4py: filename=mpi4py-3.1.4-cp38-cp38-linux_x86_64.whl size=5997259 sha256=dc5114c3111c192608dfc372bcae138bf82e82a75070dfbce4f7e1b99447cda4
  Stored in directory: /root/.cache/pip/wheels/f3/35/48/0b9a7076995eea5ea64a7e4bc3f0f342f453080795276264e7
  Building wheel for pycocotools-fix (setup.py) ... done
  Created wheel for pycocotools-fix: filename=pycocotools_fix-2.0.0.9-cp38-cp38-linux_x86_64.whl size=418798 sha256=c597ec0538761ada0334f9b62fd01b018b203b8e3cbb92d254beededd7ca9115
  Stored in directory: /root/.cache/pip/wheels/11/44/fb/0194f0a86b9f927546ae5e6e4c964cda0e7ec1504f3e21f21e
Successfully built seaborn antlr4-python3-runtime mpi4py pycocotools-fix
Installing collected packages: pytz, mpmath, flatbuffers, antlr4-python3-runtime, zipp, tzdata, tqdm, tifffile, threadpoolctl, sympy, six, scipy, PyYAML, PyWavelets, pyparsing, pynvml, protobuf, Pillow, opencv-python, networkx, mpi4py, kiwisolver, joblib, humanfriendly, h5py, fonttools, cycler, contourpy, absl-py, scikit-learn, python-dateutil, onnx, omegaconf, importlib-resources, imageio, coloredlogs, pandas, onnxruntime, matplotlib, hydra-core, seaborn, scikit-image, pycocotools-fix, nvidia-tao-deploy
  Attempting uninstall: protobuf
    Found existing installation: protobuf 4.23.0
    Uninstalling protobuf-4.23.0:
      Successfully uninstalled protobuf-4.23.0
  Attempting uninstall: onnx
    Found existing installation: onnx 1.14.0
    Uninstalling onnx-1.14.0:
      Successfully uninstalled onnx-1.14.0
Successfully installed Pillow-8.4.0 PyWavelets-1.4.1 PyYAML-6.0 absl-py-1.4.0 antlr4-python3-runtime-4.9.3 coloredlogs-15.0.1 contourpy-1.0.7 cycler-0.11.0 flatbuffers-23.5.9 fonttools-4.39.4 h5py-3.7.0 humanfriendly-10.0 hydra-core-1.2.0 imageio-2.28.1 importlib-resources-5.12.0 joblib-1.2.0 kiwisolver-1.4.4 matplotlib-3.7.1 mpi4py-3.1.4 mpmath-1.3.0 networkx-3.1 nvidia-tao-deploy-4.0.0.1 omegaconf-2.2.2 onnx-1.12.0 onnxruntime-1.14.1 opencv-python-4.7.0.72 pandas-2.0.1 protobuf-3.20.1 pycocotools-fix-2.0.0.9 pynvml-11.0.0 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2023.3 scikit-image-0.17.2 scikit-learn-0.24.2 scipy-1.5.4 seaborn-0.7.1 six-1.16.0 sympy-1.12 threadpoolctl-3.1.0 tifffile-2023.4.12 tqdm-4.64.0 tzdata-2023.3 zipp-3.15.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
In [7]:
import os
if os.environ.get("LD_LIBRARY_PATH","") == "":
  os.environ["LD_LIBRARY_PATH"] = ""
trt_lib_path = f':{os.environ.get("trt_untar_folder_path")}/TensorRT-{os.environ.get("trt_version")}/lib'
os.environ["LD_LIBRARY_PATH"]+=trt_lib_path

6. Download the etlt model of the ptm chosen ¶

In [8]:
# Installing NGC CLI on the local machine.
## Download and install
%env LOCAL_PROJECT_DIR=/ngc_content/
%env CLI=ngccli_cat_linux.zip
!sudo mkdir -p $LOCAL_PROJECT_DIR/ngccli && chmod -R 777 $LOCAL_PROJECT_DIR

# Remove any previously existing CLI installations
!sudo rm -rf $LOCAL_PROJECT_DIR/ngccli/*
!wget "https://ngc.nvidia.com/downloads/$CLI" -P $LOCAL_PROJECT_DIR/ngccli
!unzip -u -q "$LOCAL_PROJECT_DIR/ngccli/$CLI" -d $LOCAL_PROJECT_DIR/ngccli/
!rm $LOCAL_PROJECT_DIR/ngccli/*.zip 
os.environ["PATH"]="{}/ngccli/ngc-cli:{}".format(os.getenv("LOCAL_PROJECT_DIR", ""), os.getenv("PATH", ""))
!cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6 $LOCAL_PROJECT_DIR/ngccli/ngc-cli/libstdc++.so.6

env: LOCAL_PROJECT_DIR=/ngc_content/
env: CLI=ngccli_cat_linux.zip
--2023-05-13 18:56:26--  https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip
Resolving ngc.nvidia.com (ngc.nvidia.com)... 18.66.112.75, 18.66.112.122, 18.66.112.117, ...
Connecting to ngc.nvidia.com (ngc.nvidia.com)|18.66.112.75|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42746642 (41M) [application/zip]
Saving to: ‘/ngc_content//ngccli/ngccli_cat_linux.zip’

ngccli_cat_linux.zi 100%[===================>]  40.77M   218MB/s    in 0.2s    

2023-05-13 18:56:26 (218 MB/s) - ‘/ngc_content//ngccli/ngccli_cat_linux.zip’ saved [42746642/42746642]
In [9]:
#List the available versions of the PTM model chosen

modelname_to_available_ptm_map = {
  "PeopleNet" : "nvidia/tao/peoplenet:pruned_v*",
  "PeopleSemSegNet" : "nvidia/tao/peoplesemsegnet:deployable*",
  "TrafficCamNet" : "nvidia/tao/trafficcamnet:pruned_v*",
  "DashCamNet" : "nvidia/tao/dashcamnet:pruned_v*",
  "VehicleMakeNet" : "nvidia/tao/vehiclemakenet:pruned_v*",
  "VehicleTypeNet" : "nvidia/tao/vehicletypenet:pruned_v*",
  "LicensePlateRecognition" : "nvidia/tao/lprnet:deployable*",
  "LicensePlateDetection" : "nvidia/tao/lpdnet:pruned_v*",
  "FaceDetect" : "nvidia/tao/facenet:pruned_v*",
  "FaceDetectIR" : "nvidia/tao/facedetectir:pruned_v*"
}

model_to_view_regex = modelname_to_available_ptm_map[ptm_model_name]

# You can see different version of this PTM model
!ngc registry model list $model_to_view_regex

+-------------+----------+--------+------------+-----------+------------------+-----------+-----------------+--------------+
| Version     | Accuracy | Epochs | Batch Size | GPU Model | Memory Footprint | File Size | Status          | Created Date |
+-------------+----------+--------+------------+-----------+------------------+-----------+-----------------+--------------+
| pruned_v2.3 | 86.0     | 120    | 1          | V100      | 8.5              | 8.53 MB   | UPLOAD_COMPLETE | Nov 23, 2021 |
| pruned_v2.1 | 86.0     | 120    | 1          | V100      | 7.8              | 7.82 MB   | UPLOAD_COMPLETE | Aug 24, 2021 |
| pruned_v2.0 | 84.3     | 120    | 1          | V100      | 20.9             | 20.9 MB   | UPLOAD_COMPLETE | Aug 24, 2021 |
| pruned_v1.0 | 83.0     | 120    | 1          | V100      | 11.3             | 11.3 MB   | UPLOAD_COMPLETE | Aug 24, 2021 |
+-------------+----------+--------+------------+-----------+------------------+-----------+-----------------+--------------+
In [10]:
# FIXME 6 - ptm_download_folder: set the folder path where you want to download the file into
%env ptm_download_folder=/content/drive/MyDrive/ptm_models/

!sudo rm -rf $ptm_download_folder
!sudo mkdir -p $ptm_download_folder && chmod -R 777 $ptm_download_folder

# Optional FIXME 7
# model_to_download_map dictionary_value for chosen PTM key - change the version tag of the PTM model if you want a version different from the below defaults
# Use only the versions which has a etlt file:
    # Go to the link of the PTM you chose
    # Click 'File Browser tab' to fiew the files for each version
    # Choose a version you want to override which contains a etlt file
# 1. PeopleNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet
# 2. PeopleSemSegNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesemsegnet
# 3. TrafficCamNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet
# 4. DashCamNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet
# 5. FaceDetectIR - https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir
# 6. FaceDetect - https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetect
# 7. VehicleMakeNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet
# 8. VehicleTypeNet - https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet
# 9. LicensePlateDetection - https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet
# 10. LicensePlateRecognition - https://ngc.nvidia.com/catalog/models/nvidia:tao:lprnet


model_to_download_map = {
  "PeopleNet" : "nvidia/tao/peoplenet:pruned_v2.3",
  "PeopleSemSegNet" : "nvidia/tao/peoplesemsegnet:deployable_vanilla_unet_v2.0.1",
  "TrafficCamNet" : "nvidia/tao/trafficcamnet:pruned_v1.0.2",
  "DashCamNet" : "nvidia/tao/dashcamnet:pruned_v1.0.2",
  "VehicleMakeNet" : "nvidia/tao/vehiclemakenet:pruned_v1.0.1",
  "VehicleTypeNet" : "nvidia/tao/vehicletypenet:pruned_v1.0.1",
  "LicensePlateRecognition" : "nvidia/tao/lprnet:deployable_v1.0",
  "LicensePlateDetection" : "nvidia/tao/lpdnet:pruned_v1.0",
  "FaceDetect" : "nvidia/tao/facenet:pruned_v2.0",
  "FaceDetectIR" : "nvidia/tao/facedetectir:pruned_v1.0.1"
}
model_to_download = model_to_download_map[ptm_model_name]
os.environ["model_to_download"] = model_to_download

!ngc registry model download-version $model_to_download --dest $ptm_download_folder

env: ptm_download_folder=/content/drive/MyDrive/ptm_models/
Downloaded 8.54 MB in 4s, Download speed: 2.13 MB/s               
--------------------------------------------------------------------------------
   Transfer id: peoplenet_vpruned_v2.3
   Download status: Completed
   Downloaded local path: /content/drive/MyDrive/ptm_models/peoplenet_vpruned_v2.3
   Total files downloaded: 2
   Total downloaded size: 8.54 MB
   Started at: 2023-05-13 18:57:01.680219
   Completed at: 2023-05-13 18:57:05.686773
   Duration taken: 4s
--------------------------------------------------------------------------------

7. Generate TRT engine file for the ptm's etlt file ¶

In [11]:
#FIXME 8 - data_type: choose fp32 or fp16
os.environ["data_type"] = "fp32"

#FIXME 9 - trt_out_folder: choose output folder for TensorRT engine file writing
trt_out_folder = "/content/drive/MyDrive/" + ptm_model_name

!mkdir -p $trt_out_folder

import glob
input_etlt_file_list = glob.glob(os.environ.get("ptm_download_folder")+"/**/*.etlt", recursive=True)
if len(input_etlt_file_list) == 0:
  raise Exception("ETLT file was not downloaded")

os.environ["input_etlt_file"] = input_etlt_file_list[0]

if ptm_model_name in ("LicensePlateRecognition","LicensePlateDetection"):
  # FIXME: country
  # us/ccpd for LicensePlateDetection - us for United States, ch for China
  # us/ch for LicensePlateRecongition - us for United States, ch for China

  country = "us"
  for countrywise_ptm in input_etlt_file_list:
    fname = countrywise_ptm.split("/")[-1]
    if fname.startswith(country):
      os.environ["input_etlt_file"] = countrywise_ptm

action = ""
if ptm_model_name in ("PeopleNet","LicensePlateDetection","DashCamNet","TrafficCamNet","FaceDetect","FaceDetectIR"):
  action = "_trt"

os.environ["KEY"] = "tlt_encode"

if ptm_model_name in ("LicensePlateRecognition","LicensePlateDetection","FaceDetect"):
  os.environ["KEY"] = "nvidia_tlt"


os.environ["trt_experiment_spec"] = f"{os.environ.get('COLAB_NOTEBOOKS_PATH')}/tao_deploy/specs/{ptm_model_name}/{ptm_model_name}{action}.txt"
os.environ["trt_out_file_name"] = f'{trt_out_folder}/{ptm_model_name}.trt.{os.environ["data_type"]}'

if ptm_model_name in ("PeopleNet","LicensePlateDetection","DashCamNet","TrafficCamNet","FaceDetect","FaceDetectIR"):
  !detectnet_v2 gen_trt_engine \
                    -m $input_etlt_file \
                    -k $KEY  \
                    -e $trt_experiment_spec \
                    --data_type $data_type \
                    --batch_size 1 \
                    --max_batch_size 1 \
                    --engine_file $trt_out_file_name
                
elif ptm_model_name in ("VehicleMakeNet","VehicleTypeNet"):
  !classification_tf1 gen_trt_engine \
                    -m $input_etlt_file \
                    -k $KEY  \
                    -e $trt_experiment_spec \
                    --data_type $data_type \
                    --batch_size 1 \
                    --max_batch_size 1 \
                    --batches 10 \
                    --engine_file $trt_out_file_name

elif ptm_model_name == "PeopleSemSegNet":
  !unet gen_trt_engine \
                    -m $input_etlt_file \
                    -k $KEY  \
                    -e $trt_experiment_spec \
                    --data_type $data_type \
                    --batch_size 1 \
                    --max_batch_size 3 \
                    --engine_file $trt_out_file_name

elif ptm_model_name == "LicensePlateRecognition":
  !lprnet gen_trt_engine \
                    -m $input_etlt_file \
                    -k $KEY  \
                    --data_type $data_type \
                    --min_batch_size 1 \
                    --opt_batch_size 4 \
                    --max_batch_size 16 \
                    --engine_file $trt_out_file_name

/usr/local/lib/python3.8/dist-packages/pycuda/compyte/dtypes.py:120: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
  reg.get_or_register_dtype("bool", np.bool)
Traceback (most recent call last):
  File "/usr/local/bin/detectnet_v2", line 8, in <module>
    sys.exit(main())
  File "<frozen cv.detectnet_v2.entrypoint.detectnet_v2>", line 12, in main
  File "<frozen cv.common.entrypoint.entrypoint_proto>", line 196, in launch_job
  File "<frozen cv.common.entrypoint.entrypoint_proto>", line 50, in get_modules
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "</usr/local/lib/python3.8/dist-packages/nvidia_tao_deploy/cv/detectnet_v2/scripts/evaluate.py>", line 3, in <module>
  File "<frozen cv.detectnet_v2.scripts.evaluate>", line 26, in <module>
  File "</usr/local/lib/python3.8/dist-packages/nvidia_tao_deploy/cv/detectnet_v2/inferencer.py>", line 1, in <module>
  File "<frozen cv.detectnet_v2.inferencer>", line 50, in <module>
  File "</usr/local/lib/python3.8/dist-packages/nvidia_tao_deploy/inferencer/utils.py>", line 1, in <module>
  File "<frozen inferencer.utils>", line 15, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pycuda/autoinit.py", line 7, in <module>
    from pycuda.tools import make_default_context
  File "/usr/local/lib/python3.8/dist-packages/pycuda/tools.py", line 49, in <module>
    _fill_dtype_registry(respect_windows=True)
  File "/usr/local/lib/python3.8/dist-packages/pycuda/compyte/dtypes.py", line 221, in _fill_dtype_registry
    fill_registry_with_c_types(
  File "/usr/local/lib/python3.8/dist-packages/pycuda/compyte/dtypes.py", line 120, in fill_registry_with_c_types
    reg.get_or_register_dtype("bool", np.bool)
  File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'bool'.
`np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

8. Run inference on TRT engine file ¶

In [ ]:
if ptm_model_name in ("PeopleNet","LicensePlateDetection","DashCamNet","TrafficCamNet","FaceDetect","FaceDetectIR"):
  action = "_infer"

os.environ["inference_experiment_spec"] = f"{os.environ.get('COLAB_NOTEBOOKS_PATH')}/tao_deploy/specs/{ptm_model_name}/{ptm_model_name}{action}.txt"

# FIXME 10 - inference_out_folder: Folder path to write the inference results to
os.environ["inference_out_folder"] = "/content/drive/MyDrive/tao_ptm_inference/"
!rm -rf $inference_out_folder

# FIXME 11 - inference_input_images_folder: Folder path containing images to run the inference on
os.environ["inference_input_images_folder"] = f"/content/drive/MyDrive/tao_deploy_input_images/{ptm_model_name}"

print(f"Running inference for {ptm_model_name} on {os.environ['trt_out_file_name']}")
if ptm_model_name in ("PeopleNet","LicensePlateDetection","DashCamNet","TrafficCamNet","FaceDetect","FaceDetectIR"):
  !detectnet_v2 inference -e $inference_experiment_spec \
                                   -m $trt_out_file_name \
                                   -r $inference_out_folder \
                                   -i $inference_input_images_folder

elif ptm_model_name in ("VehicleMakeNet","VehicleTypeNet"):
  os.environ["inference_classmap"] = f"{os.environ.get('COLAB_NOTEBOOKS_PATH')}/tao_deploy/specs/{ptm_model_name}/classmap.json"
  !classification_tf1 inference -e $inference_experiment_spec \
                                -m $trt_out_file_name \
                                -r $inference_out_folder \
                                -c $inference_classmap \
                                -i $inference_input_images_folder

elif ptm_model_name == "PeopleSemSegNet":
  # Write path of images to a file - it is required for PeopleSemSegNet which is based on unet 
  with open("/content/PeopleSemSegNet_inference.txt","w") as file_ptr:
    for image_name in os.listdir(os.environ["inference_input_images_folder"]):
      if image_name.endswith(".jpg") or image_name.endswith(".png"):
        file_ptr.write(os.environ["inference_input_images_folder"]+"/"+image_name+"\n")
        file_ptr.flush()

  !unet inference -e $inference_experiment_spec \
                  -m $trt_out_file_name \
                  -r $inference_out_folder
      
elif ptm_model_name == "LicensePlateRecognition":
  character_file_link = "https://api.ngc.nvidia.com/v2/models/nvidia/tao/lprnet/versions/trainable_v1.0/files/{}_lp_characters.txt".format(country)
  !wget -q -O /content/characters.txt $character_file_link

  !lprnet inference -i $inference_input_images_folder \
                      -e $inference_experiment_spec \
                      -m $trt_out_file_name

9. Visualize inference ¶

In [ ]:
if ptm_model_name in ("VehicleMakeNet", "VehicleTypeNet"):
  import pandas as pd
  dataframe = pd.read_csv(os.environ["inference_out_folder"]+ "/result.csv", header=None)
  with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.max_colwidth', None):
    print(dataframe)

elif ptm_model_name != "LicensePlateRecognition":
  from IPython.display import Image, display
  import glob

  subfolder = ""
  if ptm_model_name == "PeopleSemSegNet":
    subfolder = "vis_overlay"

  inference_out_images_png = glob.glob(f'{os.environ["inference_out_folder"]}/{subfolder}/**/*.png', recursive=True)
  inference_out_images_png = glob.glob(f'{os.environ["inference_out_folder"]}/{subfolder}/**/*.jpeg', recursive=True)
  inference_out_images_jpg = glob.glob(f'{os.environ["inference_out_folder"]}/{subfolder}/**/*.jpg', recursive=True)
  inference_out_images = inference_out_images_png + inference_out_images_jpg

  if len(inference_out_images) == 0:
    raise Exception("Run Inference before visualization")

  display(Image(inference_out_images[0]))