Can Pyright check in CI only committed files?

[innerout]

I have been using pyright for quite some time in my projects using the pre-commit configuration from the documentation. I wanted to ask if it is possible to make pyright check only python files included in the current commit instead of checking all python files that exist in the project?

[erictraut]

I suppose you could create a script that dynamically writes a new pyrightconfig.json file and specifies only the newly-committed Python files in the "include" section. However, I wouldn't recommend that approach for CI because it will skip type checking for any modules that depend on (i.e., directly or indirectly import from) the newly-committed files.

Pyright also has a "watch mode" where it will run continuously and analyze files that have changed on disk. In this mode, it's able to track dependencies between files, so it doesn't suffer from the problem I mentioned above. However, this mode isn't generally used for CI because CI doesn't run continuously and typically relies on exit codes to determine whether the analysis succeeded or failed.

[innerout]

What will be needed in pyright to implement the file checking using the command line arguments? Pre-commit can provide the changed files since other hooks already implement this behavior. Would this be included in the mainline if someone fixed it?

[erictraut]

You can pass multiple files and/or directories to pyright's command-line interface, so I think it already has the support you're looking for. But as I said above, I wouldn't recommend doing this in CI because it will not type check any files that depend on the modified files, so it will leave you vulnerable to new type errors that would otherwise be caught by pyright. That kind of defeats the purpose of CI.

[innerout]

OK thanks for the clarification I initially misunderstood what you said in the first message. It seems I will leave it as is at the moment.

[hmc-cs-mdrissi]

The way I handle this is just remove pass_filenames: false for pre-commit config, but in CI you can do pre-commit run --all-files. That way developers only type check files they touched locally and keep commits fast, but CI still makes sure entire codebase is good type wise.

My personal experience I ended up making a wrapper script for mypy/pyright to do relative checks. Enforcing 0 type errors in a codebase that came to me untyped with 10s of thousands of errors (mostly missing types) wasn't doable so I opted to have CI measure the change in type errors and required each PR to decrease (relative to master) the number of type errors by 5 (number arbitrarily picked). Even file level wise it'd have been hard as some files had 100s of type errors back then. It still got some push back, but people were persuaded to accept it and months later codebase has fixed most type errors.

[simon376]

Enforcing 0 type errors in a codebase that came to me untyped with 10s of thousands of errors (mostly missing types) wasn't doable so I opted to have CI measure the change in type errors and required each PR to decrease (relative to master) the number of type errors by 5 (number arbitrarily picked). Even file level wise it'd have been hard as some files had 100s of type errors back then. It still got some push back, but people were persuaded to accept it and months later codebase has fixed most type errors.

Do you have a gist you could share on how to measure relative increase/decrease of type errors in the CI? I'd greatly appreciate it. Going from a non-typed to a properly typed repo is quite hard

[hmc-cs-mdrissi]

def _get_args():
  parser = argparse.ArgumentParser()
  parser.add_argument("files", nargs="*", help="Files to type check. Should all be python files.")
  parser.add_argument("--error-threshold", default=5, type=int)
  args = parser.parse_args()

if __name__ == '__main__':
    files = extract_files(args.files)
    if len(files) == 0:
        return

    # Use subprocess.run(["pyright"] + files, capture_output=True) to get pyright output, then extract error count from last line. 
    # Or maybe use json output.
    new_error_count, new_error_message = _analyze_type_errors(files) 
    logger.info(f"Error count for changes is {new_error_count}.")
    if new_error_count == 0:
        logger.info(SUCCESS_MESSAGE)
        return

    # checkout_branch to deal with unstaged files 
    # I re-used same code pre-commit tool uses and copied over logic.
    with checkout_branch(get_baseline_branch()): 
        old_error_count, old_error_message = _analyze_type_errors(files)
        logger.info(f"Error count for main is {old_error_count}.")

    error_change = old_error_count - new_error_count
    if error_change > args.error_threshold:
       # Fail and print both old/new error messages.

This code is simplified from actual script, but covers the main ideas. This same idea applies for any quality/testing tool. pyright/pylint/documentation checks/etc can all use similar technique of compare current commit vs baseline commit. I made similar rules with slight variations for improving documentation quality/test coverage. For baseline commit git merge-base HEAD refs/remotes/origin/master is what I used. git merge-base X Y is nearest common parent of two commits so see where current commit diverged off master as a baseline.