TensorRT build issues

[jackatparallel]

Hello,

We are building a poky image using meta-tegra and have encountered a few issues with packaging for tensorrt. Here's an excerpt from our image's .bb file:

IMAGE_INSTALL += "deepstream-6.0"
IMAGE_INSTALL += "deepstream-6.0-sources"
...
IMAGE_INSTALL += "tensorrt-plugins"
IMAGE_INSTALL += "tensorrt-core"

When trying to compile a DeepStream application on a Jetson AGX Xavier, the following line in our CMakeLists.txt fails to find libnvinfer_plugin.so:

find_file(NVINFER_LIB_PATH libnvinfer_plugin.so PATHS /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/aarch64-linux-gnu)

/usr/lib/ does in fact have libnvinfer_plugin.so.8 and libnvinfer_plugin.so.8.2.1, but it doesn't have the unversioned symlink. Looking in our poky/build/tmp directory for the unversioned file gives:

$ find poky/build/tmp/* -name *libnvinfer_plugin.so*
poky/build/tmp/sysroots-components/armv8a_tegra/tensorrt-plugins-prebuilt/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/sysroots-components/armv8a_tegra/tensorrt-plugins-prebuilt/usr/lib/libnvinfer_plugin.so
poky/build/tmp/sysroots-components/armv8a_tegra/tensorrt-plugins-prebuilt/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/jetson_agx_xavier_devkit-poky-linux/parallel-image-dev/1.0-r0/rootfs/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/jetson_agx_xavier_devkit-poky-linux/parallel-image-dev/1.0-r0/rootfs/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/tensorrt/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/tensorrt/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/tensorrt/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/packages-split/tensorrt-plugins-prebuilt/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/packages-split/tensorrt-plugins-prebuilt/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/packages-split/tensorrt-plugins-prebuilt-dev/usr/lib/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/sysroot-destdir/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/sysroot-destdir/usr/lib/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/sysroot-destdir/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/image/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/image/usr/lib/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/image/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/package/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/package/usr/lib/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/package/usr/lib/libnvinfer_plugin.so.8.2.1
poky/build/tmp/work/armv8a_tegra-poky-linux/deepstream-6.0/6.0.1-1-r0/recipe-sysroot/usr/lib/libnvinfer_plugin.so.8
poky/build/tmp/work/armv8a_tegra-poky-linux/deepstream-6.0/6.0.1-1-r0/recipe-sysroot/usr/lib/libnvinfer_plugin.so
poky/build/tmp/work/armv8a_tegra-poky-linux/deepstream-6.0/6.0.1-1-r0/recipe-sysroot/usr/lib/libnvinfer_plugin.so.8.2.1

So the unversioned symlink is being constructed, but never makes it into the image. If I bypass this issue on the Jetson by manually creating this symlink, I encounter another similar packaging issue with TensorRT header files:

...
In file included from application/src/yolo/nvdsinfer_yolo_engine.cpp:24:
/opt/nvidia/deepstream/deepstream-6.0/sources/includes/nvdsinfer_custom_impl.h:126:10: fatal error: NvCaffeParser.h: No such file or directory
 #include "NvCaffeParser.h"
          ^~~~~~~~~~~~~~~~~
compilation terminated.
In file included from application/src/yolo/nvdsparsebbox_Yolo.cpp:23:
/opt/nvidia/deepstream/deepstream-6.0/sources/includes/nvdsinfer_custom_impl.h:126:10: fatal error: NvCaffeParser.h: No such file or directory
 #include "NvCaffeParser.h"
          ^~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/yolo/CMakeFiles/yolo.dir/build.make:104: src/yolo/CMakeFiles/yolo.dir/nvdsparsebbox_Yolo.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [src/yolo/CMakeFiles/yolo.dir/build.make:90: src/yolo/CMakeFiles/yolo.dir/nvdsinfer_yolo_engine.cpp.o] Error 1
[ 26%] Building C object app/CMakeFiles/app_lib.dir/deepstream_app.c.o
In file included from application/src/yolo/trt_utils.cpp:23:
application/src/yolo/trt_utils.h:34:10: fatal error: NvInfer.h: No such file or directory
 #include "NvInfer.h"
          ^~~~~~~~~~~
compilation terminated.
...

Again searching our build directory for one of these files gives:

$ find poky/build/tmp/* -name *NvCaffeParser.h*
poky/build/tmp/sysroots-components/armv8a_tegra/tensorrt-core/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-core/8.2.1-1-r0/tensorrt/usr/include/aarch64-linux-gnu/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-core/8.2.1-1-r0/packages-split/tensorrt-core-dev/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-core/8.2.1-1-r0/sysroot-destdir/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-core/8.2.1-1-r0/image/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-core/8.2.1-1-r0/package/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/tensorrt-plugins-prebuilt/8.2.1-1-r0/recipe-sysroot/usr/include/NvCaffeParser.h
poky/build/tmp/work/armv8a_tegra-poky-linux/deepstream-6.0/6.0.1-1-r0/recipe-sysroot/usr/include/NvCaffeParser.h

and searching /usr/include/ on our system shows that these headers aren't present. Seems like the issue is with how TensorRT is packaged. Any thoughts on how to debug?

[ichergui]

Hello @jackatparallel
Are you using tegra-demo-distro ?
Could you please share the branch name if meta-tegra ?
I quit sure that I tested some stuff similar as you are doing and it works fine to me

[jackatparallel]

Ah sorry, forgot to mention that we are using kirkstone. No, we're using our own repo

[madisongh]

The non-SONAMEd libraries go into the -dev packages. Sounds like you didn't include tensorrt-core-dev and tensorrt-plugins-dev in your image?

BTW, a handy tool for tracking down this kind of thing is oe-pkgdata-util:

$ oe-pkgdata-util find-path "*/libnvinfer_plugin.so"
tensorrt-plugins-prebuilt-dev: /usr/lib/libnvinfer_plugin.so

[jackatparallel]

Yeah, looks like I forgot to include the -dev packages after tensorrt got split, thanks @madisongh