Skips breakpoints with debugger - possibly because it sets tracing

[oliversheridanmethven]

I like the interface of the of the line profiler, and often use it as a decorator when I want to profile applications, and have code similar to that shown below. However, when I come to use stepping through the application with a debugger, having such decorated functions somehow disables the debugger and it skips over all my breakpoints and continues to the end of the program (or until a crash). Based on some info in this stack overflow answer I have a suspicion this is because it sets its own tracing functions, as can be seen in the enable() and disable() methods in the file line_profiler/_line_profiler.pyx.

This means I have to comment out all my decorations if I want to use the line profiler (or change the decorator to a null-op), which is really a huge annoyance.

Is it possible for someone to confirm if this is or isn't the problem, and comment if there is a reasonable fix if so.

I'm using the PyCharm 2023.2.3 (Professional Edition) IDE and Python 3.12.4.

from line_profiler import LineProfiler
from functools import wraps

time_profiler = LineProfiler()


def profile_time(func):
    @wraps(func)
    def profiled_function(*args, **kwargs):
        time_profiler.add_function(func)
        time_profiler.enable_by_count()
        return func(*args, **kwargs)

    return profiled_function

if __name__ == "__main__":

    @profile_time
    def some_example():
        import time
        for i in range(5):  # <- If I put a breakpoint here the IDE's debugger skips right over it when this function is decorated. 
            time.sleep(0.1)


    some_example()
    time_profiler.print_stats(output_unit=1e-3)

[oliversheridanmethven]

I have been able to get the debugger to stop using cProfile based solutions, such as e.g. https://stackoverflow.com/a/5376616/5134817. However, the output is not as nice as line profiler, so I am not as happy with this solution.

[oliversheridanmethven]

For anyone interested in how to disable this when debugging:

import inspect 

def is_in_debugger():
    # Taken from https://stackoverflow.com/a/338391/5134817
    for frame in inspect.stack():
        if frame[1].endswith("pydevd.py"):
            return True
    return False

def profile_time(func):
    @wraps(func)
    def profiled_function(*args, **kwargs):
        time_profiler.add_function(func)
        time_profiler.enable_by_count()
        return func(*args, **kwargs)

    if not is_in_debugger():
        return profiled_function
    else:
        # The debugger used in e.g. Pycharm requires sys.settrace (cf. https://github.com/pyutils/line_profiler/issues/276), but that's what LineProfiler uses and it upsets everything, so we disable this in debug mode.
        logging.warning(f"We are not profiling the time of the function {func.__name__} ({inspect.getfile(func)}:{inspect.getsourcelines(func)[-1]}).")
        return func

[Erotemic]

Try using the new-style line_profiler.profile decorator that is explicitly available.

"""
FileName: mwe.py


Examples:

    python mwe.py # runs without special cases
    python mwe.py break  # breaks correctly
    python mwe.py profile  # profiles correctly
    python mwe.py profile break  # profiles correctly

"""
from line_profiler import profile
import sys
DO_BREAKPOINT = 'break' in sys.argv
DO_PROFILE = 'profile' in sys.argv

if DO_PROFILE:
    profile.enable()


if __name__ == "__main__":

    @profile
    def some_example():
        import time
        for i in range(5):
            if DO_BREAKPOINT:
                print('About to break')
                breakpoint()  # <- If I put a breakpoint here the IDE's debugger skips right over it when this function is decorated.
            else:
                print('Not trying to break')
            time.sleep(0.1)

    some_example()

That seems to work for me.

[oliversheridanmethven]

I had looked into the new style profile decorator, although wasn't overly keen on having to use an environment variable and then additionally having to pass .lprof files or change the invocation of my script (having to add $ python -m kernprof -lvr ./my_script.py in contrast to $ ./my_script.py).

[Erotemic]

You don't have to. There are several ways you can enable it. You can call line_profiler.profile.enable(), and set config options to have it print out.

from line_profiler import profile

profile.enable()
profile.show_config.update({
    'sort': True,
    'stripzeros': True,
    'rich': True,
    'details': True,
    'summarize': True,
})

profile.write_config.update({
    'lprof': False,
    'text': False,
    'timestamped_text': False,
    'stdout': True,
})


@profile
def some_example():
    import time
    for i in range(5):
        time.sleep(0.1)

some_example()

if profile.enabled:
    profile.show()  # FIXME: future versions should not break if profile is disabled when you call show.

You can also use profile.write_config to control if it dumps files to disk at all.

EDIT: edited script so it shows all config options, and gives a setting where information is only written to stdout. If you don't have rich installed, set rich to False, but it does look better with rich enabled.

[oliversheridanmethven]

So the proposed solution is almost what I am after. The biggest blocker is that if I set profile.enable(), which I want to do, then it still skips over breakpoints set with my IDE, albeit manual breakpoints hard written into the code using breakpoint() are still hit.

[Erotemic]

I see.

I'm not sure what PyCharm is doing to insert breakpoints. Is it possible to translate whatever they are doing into something that can be reproduced without needing to install and interact with PyCharm itself? I'm thinking something like:

python --special-argument-to-set-breakpoints-like-pycharm=line32 mwe.py

[oliversheridanmethven]

I'm also not sure what the JetBrains PyCharm IDE is doing under the hood to insert break points (and don't particularly want to deep dive it to find out), and if they do anything different from most other IDEs (Eclipse, VSC, XCode, etc.). Having a quick play around with VSC, it also does something funky, namely: if there is no hardcoded breakpoint() then it appears to suffer from the same issue as PyCharm, whereas if I do include the breakpoint(), then it does seem to hit the IDE's "virtual" breakpoints. This suggests that while PyCharm and VSC handle debugging and catching breakpoints slightly differently, they both don't entirely like the side effects of profile.enable().

[Erotemic]

It looks like your original suspicion is correct and this is a problem. The sys.settrace is likely the mechanism PyCharm is using.

I don't know if there is a reasonable fix or not.