CUDA CXX CMAKE: nvlink错误:未定义的引用

Question

所以我已经花了将近一天的时间来研究这个问题，我自己也想不出来。对于一个大型的CUDA和CXX项目，我使用CMake进行依赖处理/编译设置。我正在使用MSVC2019在Windows 11上工作。我的想法是，我有多个模块，即带有主机和设备代码的库，它们链接到应用程序文件夹中的可执行文件。然而，当我试图编译可执行文件时，有一个来自CUDA的链接错误，即使(我认为)我已经链接了所有所需的repos。另一方面，库(它们都相互依赖)使用相同的设置编译得很好。

完整的生成和错误日志如下：

[main] Configuring folder: fix_cuda_linker 
[driver] Removing c:/artemis/artemis_tests/fix_cuda_linker/build/CMakeCache.txt
[driver] Removing c:\artemis\artemis_tests\fix_cuda_linker\build\CMakeFiles
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --no-warn-unused-cli -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -Sc:/artemis/artemis_tests/fix_cuda_linker -Bc:/artemis/artemis_tests/fix_cuda_linker/build -G "Visual Studio 16 2019" -T host=x64 -A x64
[cmake] Not searching for unused variables given on the command line.
[cmake] -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22000.
[cmake] -- The CUDA compiler identification is NVIDIA 11.5.50
[cmake] -- The CXX compiler identification is MSVC 19.29.30143.0
[cmake] -- The C compiler identification is MSVC 19.29.30143.0
[cmake] -- Detecting CUDA compiler ABI info
[cmake] -- Detecting CUDA compiler ABI info - done
[cmake] -- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.5/bin/nvcc.exe - skipped
[cmake] -- Detecting CUDA compile features
[cmake] -- Detecting CUDA compile features - done
[cmake] -- Detecting CXX compiler ABI info
[cmake] -- Detecting CXX compiler ABI info - done
[cmake] -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
[cmake] -- Detecting CXX compile features
[cmake] -- Detecting CXX compile features - done
[cmake] -- Detecting C compiler ABI info
[cmake] -- Detecting C compiler ABI info - done
[cmake] -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
[cmake] -- Detecting C compile features
[cmake] -- Detecting C compile features - done
[cmake] -- Configuring done
[cmake] -- Generating done
[cmake] -- Build files have been written to: C:/artemis/artemis_tests/fix_cuda_linker/build

和

[main] Building folder: fix_cuda_linker clean
[build] Starting build
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --build c:/artemis/artemis_tests/fix_cuda_linker/build --config Debug --target clean -j 18 --
[build] Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
[build] Copyright (C) Microsoft Corporation. All rights reserved.
[build] 
[build] Build finished with exit code 0
[main] Building folder: fix_cuda_linker 
[build] Starting build
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --build c:/artemis/artemis_tests/fix_cuda_linker/build --config Debug --target ALL_BUILD -j 18 --
[build] Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
[build] Copyright (C) Microsoft Corporation. All rights reserved.
[build] 
[build]   Checking Build System
[build]   Building Custom Rule C:/artemis/artemis_tests/fix_cuda_linker/modules/func1/CMakeLists.txt
[build]   Compiling CUDA source file ..\..\..\modules\func1\func1.cu...
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func1>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -gencode=arch=compute_52,code=\"compute_52,compute_52\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64" -x cu -rdc=true   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\include"     --keep-dir x64\Debug  -maxrregcount=0  --machine 64 --compile -cudart static -std=c++14 -Xcompiler="/EHsc -Zi -Ob0" -g  -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -D"CMAKE_INTDIR=\"Debug\"" -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func1.pdb /FS /Zi /RTC1 /MDd " -o func1.dir\Debug\func1.obj "C:\artemis\artemis_tests\fix_cuda_linker\modules\func1\func1.cu" 
[build]   func1.cu
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func1>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -dlink -o func1.dir\Debug\func1.device-link.obj -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func1.pdb /Zi /RTC1 /MDd " -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin/crt" -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64" cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib cudart.lib -forward-unknown-to-host-compiler -Wno-deprecated-gpu-targets  -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52  --machine 64 func1.dir\Debug\func1.obj 
[build]   cudadevrt.lib
[build]   cudart_static.lib
[build]   kernel32.lib
[build]   user32.lib
[build]   gdi32.lib
[build]   winspool.lib
[build]   comdlg32.lib
[build]   advapi32.lib
[build]   shell32.lib
[build]   ole32.lib
[build]   oleaut32.lib
[build]   uuid.lib
[build]   odbc32.lib
[build]   odbccp32.lib
[build]   cudart.lib
[build]   func1.obj
[build]   func1.vcxproj -> C:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func1.lib
[build]   Building Custom Rule C:/artemis/artemis_tests/fix_cuda_linker/modules/func2/CMakeLists.txt
[build]   Compiling CUDA source file ..\..\..\modules\func2\func2.cu...
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func2>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -gencode=arch=compute_52,code=\"compute_52,compute_52\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64" -x cu -rdc=true  -IC:\artemis\artemis_tests\fix_cuda_linker\modules -IC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\include"     --keep-dir x64\Debug  -maxrregcount=0  --machine 64 --compile -cudart static -std=c++14 -Xcompiler="/EHsc -Zi -Ob0" -g  -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -D"CMAKE_INTDIR=\"Debug\"" -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func2.pdb /FS /Zi /RTC1 /MDd " -o func2.dir\Debug\func2.obj "C:\artemis\artemis_tests\fix_cuda_linker\modules\func2\func2.cu" 
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func1/func1.cuh(11): warning #821-D: extern inline function "test::func1::AddOne" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func2\func2.vcxproj]
[build]   
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func1/func1.cuh(11): warning #821-D: extern inline function "test::func1::AddOne" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func2\func2.vcxproj]
[build]   
[build]   func2.cu
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func2>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -dlink -o func2.dir\Debug\func2.device-link.obj -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func2.pdb /Zi /RTC1 /MDd " -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin/crt" -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64" cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib cudart.lib -forward-unknown-to-host-compiler -Wno-deprecated-gpu-targets  -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52  --machine 64 func2.dir\Debug\func2.obj 
[build]   cudadevrt.lib
[build]   cudart_static.lib
[build]   kernel32.lib
[build]   user32.lib
[build]   gdi32.lib
[build]   winspool.lib
[build]   comdlg32.lib
[build]   advapi32.lib
[build]   shell32.lib
[build]   ole32.lib
[build]   oleaut32.lib
[build]   uuid.lib
[build]   odbc32.lib
[build]   odbccp32.lib
[build]   cudart.lib
[build]   func2.obj
[build]   func2.vcxproj -> C:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func2.lib
[build]   Building Custom Rule C:/artemis/artemis_tests/fix_cuda_linker/modules/func3/CMakeLists.txt
[build]   Compiling CUDA source file ..\..\..\modules\func3\func3.cu...
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -gencode=arch=compute_52,code=\"compute_52,compute_52\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64" -x cu -rdc=true  -IC:\artemis\artemis_tests\fix_cuda_linker\modules -IC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\include"     --keep-dir x64\Debug  -maxrregcount=0  --machine 64 --compile -cudart static -std=c++14 -Xcompiler="/EHsc -Zi -Ob0" -g  -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -D"CMAKE_INTDIR=\"Debug\"" -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func3.pdb /FS /Zi /RTC1 /MDd " -o func3.dir\Debug\func3.obj "C:\artemis\artemis_tests\fix_cuda_linker\modules\func3\func3.cu" 
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func1/func1.cuh(11): warning #821-D: extern inline function "test::func1::AddOne" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3\func3.vcxproj]
[build]   
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func2/func2.cuh(11): warning #821-D: extern inline function "test::func2::AddTwo" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3\func3.vcxproj]
[build]   
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func1/func1.cuh(11): warning #821-D: extern inline function "test::func1::AddOne" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3\func3.vcxproj]
[build]   
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func2/func2.cuh(11): warning #821-D: extern inline function "test::func2::AddTwo" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3\func3.vcxproj]
[build]   
[build]   func3.cu
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\modules\func3>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -dlink -o func3.dir\Debug\func3.device-link.obj -Xcompiler "/EHsc /W1 /nologo /Od /FdC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func3.pdb /Zi /RTC1 /MDd " -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin/crt" -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64" cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib cudart.lib -forward-unknown-to-host-compiler -Wno-deprecated-gpu-targets  -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52  --machine 64 func3.dir\Debug\func3.obj 
[build]   cudadevrt.lib
[build]   cudart_static.lib
[build]   kernel32.lib
[build]   user32.lib
[build]   gdi32.lib
[build]   winspool.lib
[build]   comdlg32.lib
[build]   advapi32.lib
[build]   shell32.lib
[build]   ole32.lib
[build]   oleaut32.lib
[build]   uuid.lib
[build]   odbc32.lib
[build]   odbccp32.lib
[build]   cudart.lib
[build]   func3.obj
[build]   func3.vcxproj -> C:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug\func3.lib
[build]   Building Custom Rule C:/artemis/artemis_tests/fix_cuda_linker/apps/CMakeLists.txt
[build]   Compiling CUDA source file ..\..\apps\apps1.cu...
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\apps>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -gencode=arch=compute_52,code=\"compute_52,compute_52\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64" -x cu -rdc=true  -IC:\artemis\artemis_tests\fix_cuda_linker\modules -IC:\artemis\artemis_tests\fix_cuda_linker\build\bin\Debug -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\include"     --keep-dir x64\Debug  -maxrregcount=0  --machine 64 --compile -cudart static -std=c++14 -Xcompiler="/EHsc -Zi -Ob0" -g  -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -D"CMAKE_INTDIR=\"Debug\"" -Xcompiler "/EHsc /W1 /nologo /Od /Fdapp1.dir\Debug\vc142.pdb /FS /Zi /RTC1 /MDd " -o app1.dir\Debug\apps1.obj "C:\artemis\artemis_tests\fix_cuda_linker\apps\apps1.cu" 
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func3/func3.cuh(11): warning #821-D: extern inline function "test::func3::AddThree" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\apps\app1.vcxproj]
[build]   
[build] C:\artemis\artemis_tests\fix_cuda_linker\modules\func3/func3.cuh(11): warning #821-D: extern inline function "test::func3::AddThree" was referenced but not defined [C:\artemis\artemis_tests\fix_cuda_linker\build\apps\app1.vcxproj]
[build]   
[build]   apps1.cu
[build]   
[build]   C:\artemis\artemis_tests\fix_cuda_linker\build\apps>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -dlink -o app1.dir\Debug\app1.device-link.obj -Xcompiler "/EHsc /W1 /nologo /Od /Fdapp1.dir\Debug\vc142.pdb /Zi /RTC1 /MDd " -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin/crt" -LC:/artemis/artemis_tests/fix_cuda_linker/build/bin -LC:/artemis/artemis_tests/fix_cuda_linker/build/bin/Debug -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64" ..\bin\Debug\func1.lib ..\bin\Debug\func2.lib ..\bin\Debug\func3.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib -forward-unknown-to-host-compiler -Wno-deprecated-gpu-targets  -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52  --machine 64 app1.dir\Debug\apps1.obj 
[build] CUDALINK : nvlink error : Undefined reference to '_ZN4test5func38AddThreeEf' in 'app1.dir/Debug/apps1.obj' [C:\artemis\artemis_tests\fix_cuda_linker\build\apps\app1.vcxproj]
[build]   func1.lib
[build]   func2.lib
[build]   func3.lib
[build]   cudadevrt.lib
[build]   cudart_static.lib
[build]   kernel32.lib
[build]   user32.lib
[build]   gdi32.lib
[build]   winspool.lib
[build]   shell32.lib
[build]   ole32.lib
[build]   oleaut32.lib
[build]   uuid.lib
[build]   comdlg32.lib
[build]   advapi32.lib
[build]   apps1.obj
[build] C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\BuildCustomizations\CUDA 11.5.targets(874,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" -dlink -o app1.dir\Debug\app1.device-link.obj -Xcompiler "/EHsc /W1 /nologo /Od /Fdapp1.dir\Debug\vc142.pdb /Zi /RTC1 /MDd " -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin/crt" -LC:/artemis/artemis_tests/fix_cuda_linker/build/bin -LC:/artemis/artemis_tests/fix_cuda_linker/build/bin/Debug -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64" ..\bin\Debug\func1.lib ..\bin\Debug\func2.lib ..\bin\Debug\func3.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib -forward-unknown-to-host-compiler -Wno-deprecated-gpu-targets  -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52  --machine 64 app1.dir\Debug\apps1.obj" exited with code 255. [C:\artemis\artemis_tests\fix_cuda_linker\build\apps\app1.vcxproj]
[build] Build finished with exit code 1

文件夹/文件结构如下所示：

|   .gitignore
|   CMakeLists.txt
|
+---apps
|       apps1.cu (NOTE: uses func1 func2 func3)
|       CMakeLists.txt
|
+---build (NOTE: I REMOVED LINES HERE)
|
\---modules
    |   CMakeLists.txt
    |
    +---func1
    |       CMakeLists.txt
    |       func1.cu
    |       func1.cuh
    |
    +---func2 (NOTE: uses func1)
    |       CMakeLists.txt
    |       func2.cu
    |       func2.cuh
    |
    \---func3 (NOTE: uses func1 func2)
            CMakeLists.txt
            func3.cu
            func3.cuh

我编写的最小失败示例可以在这里找到论吉乌布。

在试图修复此错误时，我发现了许多关于CUDA CMake链接错误的老问题(>5y)，大多数问题都是通过支持设备和主机代码的单独编译(将CUDA_SEPARABLE_COMPILATION设置为ON)来解决的。有时还需要解析设备符号(将CUDA_RESOLVE_DEVICE_SYMBOLS设置为ON)，因为这会强制设备和主机在开始时单独编译。然而，对我来说，可执行文件和库的组合都不起作用。

这个例子是如何编译的？谢谢你的帮助！

Answer 1

因此，我终于成功地编译了这个虚拟项目。MSVC2019的工作解决方案可以找到这里和原始问题这里。

解决方案是为所有具有__device__函数的文件启用__device__，并且只对具有__global__函数的文件启用CUDA_RESOLVE_DEVICE_SYMBOLS。此外，向项目中添加dummy.cu文件有助于CMake识别CUDA代码。

除其他外，这两个联系帮助我找到了答案：

https://forums.developer.nvidia.com/t/linking-multiple-static-cuda-libs/148964 https://gitlab.kitware.com/cmake/cmake/-/issues/17520