Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hook for debugging shared libraries in Visual Studio #121

Merged
merged 12 commits into from
May 8, 2024
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@ Makes a ZIP with all the executables and runtimes

Makes a ZIP with all the licenses from the graph

### HOOKS

#### [PDBs](extensions/hooks/_hook_copy_pdbs_to_package.py)

Post pacakge hook that copies PDBs to the package folder.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

package - typo


### Testing

To validate a new extension, it's possible to write a test that exercises its usage.
Expand Down
60 changes: 60 additions & 0 deletions extensions/hooks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
## PDBs hook
This hook copies the PDBs from their original location in the build folder to the package folder.
This is required for debugging libraries with Visual Studio when the original source files aren't present.
For more information on how to debug using the hook check the [documentation](https://docs.conan.io/2/examples/dev_flow/debug/debugging_visual.html)

### PDBs

A PDB has the information to link the source code of a debuggable object to the Visual Studio debugger. Each PDB is linked to a
specific file (executable or library) and contains the source file name and line numbers to display in the IDE.

When compiling shared libraries in Debug mode the created binary will contain the information of where the PDB will be
generated, which by default is the same path where the file is being compiled. The PDBs are created by the ``cl.exe``
compiler with the ``/Zi`` flag, or by the ``link.exe`` when linking a DLL or executable.

PDBs are created when compiling a library or executable in Debug mode and are created by default in the same directory
as the file it is associated with. This means that when using Conan they will be created in the build directory in the
same path as the DLLs.

When using the Visual Studio debugger, it will look for PDBs to load in the following paths:

- The project folder.
- The original path where the associated file was compiled.
- The path where Visual is currently finding the compiled file, in our case the DLL in the package folder.

### Locating PDBs

To locate the PDB of a DLL we can use the ``dumpbin.exe`` tool, which comes with Visual by default and can be located
using the ``vswhere`` tool. PDBs will usually have the same name as it's
DLL, but it's not always the case, so checking with the ``dumpbin \PDBPATH`` command makes sure we are getting the PDB
corresponding to each DLL.

When a DLL is created it contains the information of the path where its corresponding PDB was generated. This can be
manually checked by running the following commands:
```
$ "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -find "**\dumpbin.exe"
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\dumpbin.exe

# Use the path for the dumpbin.exe that you got from the previous command
$ "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\dumpbin.exe" /PDBPATH <dll_path>
...
Dump of file .\bin\zlib1.dll

File Type: DLL
PDB file found at 'C:\Users\{user}\.conan2\p\b\zlib78326f0099328\p\bin\zlib1.pdb'
...
```

### Source files

It is important to note that the PDB only contains the debug information, to debug through the actual file the source
files are needed. These files have to be the exact same files as when the library was compiled and the PDB generated,
as Visual Studio does a checksum check. In case where the build folder was deleted the source files from the source
folder can be used instead by telling Visual where to find them, as it is explained in the documentation.

### Static libraries

PDBs can sometimes be generated for LIB files, but for now the feature only focuses on shared libraries and
will only work with PDBs generated for DLLs. This is because the linking of PDBs and static libraries works differently
than with shared libraries and the PDBs are generated differently, which doesn't allow us to get the name and path
of a PDB through the ``dumpbin`` tool and will require different methods.
40 changes: 40 additions & 0 deletions extensions/hooks/_hook_copy_pdbs_to_package.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
from conan.tools.files import copy
import glob
import json
import os
import re
from io import StringIO
from conans.errors import ConanException
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from conan.errors import ConanException will be more future-proof



def post_package(conanfile):
if conanfile.settings.get_safe("os") != "Windows" or conanfile.settings.get_safe("compiler") != "msvc":
return
conanfile.output.info("PDBs post package hook running")
search_package_dll = os.path.join(conanfile.package_folder, "**/*.dll")
package_dll = glob.glob(search_package_dll, recursive=True)
if len(package_dll) == 0:
return
# Find dumpbin path
output = StringIO()
try:
conanfile.run(
r'"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -find "**\dumpbin.exe" -format json',
stdout=output, scope="")
except ConanException:
raise ConanException(
"Failed to locate dumpbin.exe which is needed to locate the PDBs and copy them to package folder.")
dumpbin_path = json.loads(str(output.getvalue()))[0]

for dll_path in package_dll:
# Use dumpbin to get the pdb path from each dll
dumpbin_output = StringIO()
conanfile.run(rf'"{dumpbin_path}" /PDBPATH {dll_path}', stdout=dumpbin_output)
dumpbin = str(dumpbin_output.getvalue())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually, what paths are returned here? I just realized that for example most CCI recipes remove the pdbs in the package method, and if tyhis happens after the package, maybe the pdbs do not exist anymore for the interested recipes - I'm saying that maybe we shoukd have a way to avoid those pdb deletions in the first place?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give me an example of a recipe that does this so I can check how it works? As I understand it dumpbin gets the path of were the PDB is originally located, this is usually (always?) where the library is created which should be the build folder not the package one.

Copy link
Member

@AbrilRBS AbrilRBS Apr 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libxml2 is a good example.
The line you're looking for is something that starts with
rm(self, "*.pdb", in the package() method - let me know if you need any help reproing it, but you might be right that this might point to the build folder so we're ok :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this and everything is working fine as expected!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be fine. Having a look at https://learn.microsoft.com/en-us/cpp/build/reference/pdbpath?view=msvc-170, it should return the path embedded in the binary, that should be the path in the build folder where the binary was originated. As this is a post_package hook, it should work even if the package() method deletes the pdbs because the hook is executed after the method 😄

pdb_path = re.search(r"'.*\.pdb'", dumpbin)
if pdb_path:
pdb_path = pdb_path.group()[1:-1]
# Copy the corresponding pdb file from the build to the package folder
conanfile.output.info(
f"copying {os.path.basename(pdb_path)} from {os.path.dirname(pdb_path)} to {os.path.dirname(dll_path)}")
copy(conanfile, os.path.basename(pdb_path), os.path.dirname(pdb_path), os.path.dirname(dll_path))
41 changes: 40 additions & 1 deletion tests/test_pdb_hook.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,46 @@
import os
import tempfile
import json
import platform

import pytest

from tools import run


@pytest.fixture(autouse=True)
def conan_test():
old_env = dict(os.environ)
env_vars = {'CONAN_HOME': tempfile.mkdtemp(suffix='conans')}
os.environ.update(env_vars)
current = tempfile.mkdtemp(suffix='conans')
cwd = os.getcwd()
os.chdir(current)
try:
yield
finally:
os.chdir(cwd)
os.environ.clear()
os.environ.update(old_env)


@pytest.mark.win32
def test_copy_pdb_hook():
print("This is a placeholder for future tests")
repo = os.path.join(os.path.dirname(__file__), "..")
run(f'conan config install {repo}')
conan_home = run('conan config home').strip()
hooks_path = os.path.join(conan_home, 'extensions', 'hooks')
old_file = os.path.join(hooks_path, '_hook_copy_pdbs_to_package.py')
new_file = os.path.join(hooks_path, 'hook_copy_pdbs_to_package.py')
os.rename(old_file, new_file)
run('conan profile detect')
run('conan new cmake_lib -d name=lib -d version=1.0')
out = run('conan create . -s build_type=Debug -o "*:shared=True" -tf=""')
assert "PDBs post package hook running" in out
list_output = run('conan list lib/1.0:* --format=json')
list_json = json.loads(list_output)
revision = list_json['Local Cache']['lib/1.0']['revisions'].values()
revision_info = next(iter(revision))
package_id = next(iter(revision_info['packages']))
path = run(fr'conan cache path lib/1.0:{package_id}').strip()
assert os.path.isfile(os.path.join(path, 'bin', 'lib.pdb'))
Loading