[Oct 15-22, 2021] Sparse Index & FSMonitor Bug Bash

[vdye]

🚨 The Bug Bash will run through Oct 22, 2021! 🚨

Pre-release of microsoft/git is ready for download!

Interested in participating? Subscribe to this discussion, download the release, and follow the instructions below on how to test & provide feedback!

Bug Bash: Sparse Index & FSMonitor

Overview

Welcome to the Sparse Index/FSMonitor Git Bug Bash! The goal of this is to find bugs affecting the Sparse Index and FSMonitor features in microsoft/git. This bug bash will help us (the maintainers of microsoft/git) ensure these features are as stable as possible when they're released to users and, ultimately, when they're integrated into the core Git project.

What is Sparse Index?

The background of the sparse index starts with git sparse-checkout; this blog post provides a great summary. The short version: git sparse-checkout allows a user to restrict which parts of their local repository show up on disk, resulting in some performance improvement in large repositories.

The sparse index takes that idea a step further. If you restrict the structure of your "sparse checkout definition" (that is, the glob patterns that tell git what is supposed to be tracked in git sparse-checkout) by requiring cone mode, you can identify entire directories as being outside of the sparse checkout definition. By condensing the internal representation of those directories, git won't even need to look at those directories' contents, resulting in substantial performance gains. This condensed repo state is the sparse index!

However, because git makes assumptions around the uncondensed version of the index, commands need to have the sparse index individually integrated and enabled. In this bug bash, the commands that can (at least partially) utilize the sparse index are:

• git status
• git commit
• git add
• git checkout
• git reset
• git diff
• git sparse-checkout
• git merge
• git cherry-pick
• git rebase
• git stash
• git blame
• git clean
• git update-index
• git checkout-index
• git read-tree

What is FSMonitor?

FSMonitor is a background process that feeds filesystem information ("this file was modified"; "that file was deleted") to git in order to save time when reporting on the state of a repository. This bug bash is testing V4 of FSMonitor; unlike previous versions, it is completely homegrown and does not require any third-party tools installed alongside it.

FSMonitor is currently only implemented for Windows and MacOS. For more background on its history and implementation, see this discussion in the Git for Windows repository.

Getting Started

Things you'll need

1. A general understanding of how to use git from the command line
   • New to command line git? The official git documentation site has links to visual guides, cheat sheets, and detailed formal documentation for each command.
2. A computer to do your testing
   • We have official installers for Windows, MacOS, and Ubuntu; any others will require building and installing the release from source.
3. A GitHub account
   • You'll need to be logged in to report any bugs you find!
4. Some time to test
   • Set aside as little or as much time as you'd like - verifying even one scenario will help ensure the quality of these features. Full disclosure: you will want to set aside anywhere between 5-15 minutes for setup - anything after that is up to you!

Installation

This bug bash release can be installed the same way you would install standard git releases. The installers for Windows, Mac, and Ubuntu can be found here:

Download Git v2.33.0.vfs.0.2

If you plan to test on a platform other than three listed above, you will need to install from source using the source tarball linked in the release above.

Enabling features

1. [Windows/Mac only] To enable FSMonitor, open your command line of choice and run the command

   git config --global core.useBuiltinFSMonitor true
   

2. Cone-mode sparse checkouts will have the sparse index enabled by default (specific to the microsoft/git fork of git), but the availability of that feature can be manually controlled with the git sparse-checkout init --[no-]sparse-index --cone.

Testing

Now that setup is out of the way, it's time to test! You want to start with an initialized repository - any will work for this bug bash, but you'll get extra insight into performance if you test on particularly large repositories.

Once you have a repository, you'll need to set up a "sparse-checkout cone" - basically, a subset of directories in your repository that are actually "active" (and, as a result, populated with files on disk). To do this, you can follow the instructions in the git sparse-checkout command documentation.

Example

Here is a series of commands that sets you up with a sparse checkout cone in the microsoft/Git-Credential-Manager-Core repository:

git clone --sparse https://github.com/microsoft/Git-Credential-Manager-Core.git test-repo
cd test-repo
git sparse-checkout init --cone
git sparse-checkout set src

This creates a cone-mode sparse checkout with all toplevel files populated, plus the contents of the src directory.

With that, you're ready to start testing! In general, experiment with ways you can imagine disrupting file system monitoring or a sparse checkout cone. Although it's difficult to suggest tests with more specificity (if we knew what to look for, we wouldn't need to bug bash!), we can provide some more specific tips on what to try & look for:

1. If you're stuck on where to start, clone an example repo specifically meant for sparse-checkout testing. From there, treat the repo like something you'd normally work on: making local changes, committing, switching branches, merging, etc.
2. Try different platforms! Windows, MacOS, Linux!
3. Change your sparse-checkout cone a lot.
4. Try creating files outside of the sparse-checkout cone.
5. Keep an eye on performance: slow commands are broken commands. Consider utilizing trace2 logs to search for potential performance issues.
   • To print performance-related logs to the terminal, run commands prefixed with GIT_TRACE2_PERF=1 (e.g., GIT_TRACE2_PERF=1 git status). Look for ensure_full_index() in particular - while ensure_full_index() does not always indicate a bug, encountering it in a command that uses the sparse index (see "What is Sparse Index?") and only operates within the sparse checkout definition may be worth noting.
   • Alternatively, you can set GIT_TRACE2_EVENT to a absolute directory path (e.g., GIT_TRACE2_EVENT=$(pwd)/logs/ git status) to have it print the command's trace2 log to a file.
6. Look for any confusing messages or warnings.

Submitting Feedback

To submit a bug report, you will need to create a new top-level comment on this discussion thread. Failure to create the bug report in the right place may prevent your bug report from being read/counted!

Below are templates for bug reports involving FSMonitor and Sparse Index, respectively. You will need to add your answers in the < > brackets, then delete the brackets (a reference on formatting text can be found here).

FSMonitor

## Affected Feature
- [X] FSMonitor
- [ ] Sparse Index

## Operating System & Version
_Instructions (run from command line):_
1. _On Windows, run `cmd.exe /c ver`_
2. _On MacOS, run `sw_vers` - your version is the "ProductVersion"_
3. _On any Linux distro, run `lsb_release -a` - your version is the "Release"_

**Operating System:** <your OS, e.g. Windows/MacOS/Ubuntu>

**OS Version:** <your OS version, following the instructions above>

## Description
_Describe the bug: what happened? why would you consider it broken?_

<your bug description>

## How to Recreate
_Add step-by-step instructions on how to start from a "clean" repository (including any system state changes, e.g. "Run immediately after waking laptop from sleep")._

<your steps to recreate>

Sparse Index

## Affected Feature
- [ ] FSMonitor
- [X] Sparse Index

## Operating System & Version
_Instructions (run from command line):_
1. _On Windows, run `cmd.exe /c ver`_
2. _On MacOS, run `sw_vers` - your version is the "ProductVersion"_
3. _On any Linux distro, run `lsb_release -a` - your version is the "Release"_

**Operating System:** <your OS, e.g. Windows/MacOS/Ubuntu>

**OS Version:** <your OS version, following the instructions above>

## Description
_Describe the bug: what happened? why would you consider it broken?_

<your bug description>

## How to Recreate
_Add step-by-step instructions on how to start from a "clean" repository (including any system state changes, e.g. "Run immediately after waking laptop from sleep")._

<your steps to recreate>

## Does the bug also occur with `index.sparse` disabled?
_To verify, run `git sparse-checkout init --no-sparse-index --cone` (to disable the sparse index), then re-run your "How to Recreate" instructions. Then, re-enable sparse index with `git sparse-checkout init --sparse-index --cone`._

<Yes/No>

Cleanup

If you want to switch back over to a stable git release for your day-to-day use, simply re-install git from your preferred source (the Git website, Homebrew, etc.). You can also remove the modified config settings with:

git config --global --unset core.useBuiltinFSMonitor

🎉 Thank you for participating in this bug bash! 🎉

Prizes

To show our gratitude for the hard work you're putting in to test our code, we're planning on awarding prizes of GitHub merch & gift cards to the participants that provide:

• Most unique bugs reported
• Best written bug report(s)
• Strangest bug and/or weirdest edge case

Stay subscribed to this thread after the bug bash to find out whether you've won!

We will also be calling out the participants in the upcoming microsoft/git 2.34.0 release notes to acknowledge your valuable contributions in this bug bash. We appreciate the time you've graciously decided to set aside for testing this - thanks to you, we can all look forward to an always-improving Git!

[vdye]

The Bug Bash has started!

The test version of microsoft/git downloaded here.

🚨 NOTE: We ask that you only create top-level comments for new bug reports - for miscellaneous questions, please reply to this comment. This will make it much easier for us to aggregate bug reports after the bug bash (which, in turn, means we can determine our prize winners sooner), and will provide a nice, centralized location for Q&As that other testers to reference.

We'll be actively monitoring this discussion as feedback is submitted to answer questions, provide context on strange behavior, etc.

Thank you again for your hard work in making our software better! 🎉

[marcelais]

Do you want to hear about bugs in the installer? I got two unexpected error messages thrown during install.

[vdye]

@marcelais Yes! You can use the reporting template for "FSMonitor" above, but change the "Affected Feature" section to:

## Affected Feature
- [ ] FSMonitor
- [ ] Sparse Index
- [X] Other

[marcelais]

Affected Feature

• FSMonitor
• Sparse Index
• Other

Operating System: Windows

OS Version: Microsoft Windows [Version 10.0.19043.1288]

Description

Describe the bug: what happened? why would you consider it broken?

Ran the Windows git installer with git 2.32.0.vfs.0.2 already installed. During the install, hit two popup messages (see below). Installation appeared to continue and complete successfully.

Unable to set system config "init.defaultBranch":="master": exit code 4
stderr:
error: could not write config file C:/Program Files/Git/etc/gitconfig: Permission denied

Unclear why there would be a permissions error here -- the config file does exist and is not read-only. Like the rest of the program files directory, only admins can write to it but the installer was elevated.

2. Line 3351: Count not reconfigure Scalar enlsitments (output:, errors: warning: could not switch to 'D:/git/EnlistmentTest': No such file or directory).

d:\git\enlistmenttest was the former location of a scalar repo that has been previously deleted. Scalar apparently still has some record of it recorded somewhere. I would expect orphaned repo registrations to just be cleaned up instead of generating an error.

[dscho]

Unable to set system config "init.defaultBranch":="master": exit code 4

Are you installing 2.32.0.vfs.0.2 or 2.33.0.vfs.0.2?

1. Line 3351: Count not reconfigure Scalar enlsitments (output:, errors: warning: could not switch to 'D:/git/EnlistmentTest': No such file or directory).

You probably typed this (instead of copy/pasting it): I did not find the message with that typo, but I found it without that typo.

d:\git\enlistmenttest was the former location of a scalar repo that has been previously deleted. Scalar apparently still has some record of it recorded somewhere.

The location is stored in your user-wide Git config. Call git config -e --global to see it (there should be two entries, one in the [scalar] section and one in the [maintenance] section).

I would expect orphaned repo registrations to just be cleaned up instead of generating an error.

That is a good point. We should probably teach Scalar to do exactly that.

[marcelais]

Yes, I did copy that by hand since retyping the text was better than a screen-shot of the message box.

I was installing 2.33 over an already installed 2.32.

[dscho]

Hmm. I opened #449 so that we can investigate this issue in a more focused manner.

[vdye]

The Bug Bash has ended!

Thank you again for your participation - we're looking forward to seeing the impact of FSMonitor/sparse index on our upcoming releases!

As always, if you encounter bugs in the future (or want to suggest a feature), you can open an issue on the repository's Issues page.