[enhancement]: Support Sparse Checkout

[bdovaz]

Describe your feature request here

For monorepo repositories this is a must to be able to speed up builds as much as possible.

I see that this very thing has been recently merged by @dscho into the GitHub checkout action: actions/checkout#1369

Would this same functionality be possible in azure pipelines?

Thanks!

[KonstantinTyukalov]

Hi @bdovaz thanks for the reporting! We're currently working on higher priority issues, but we'll get back this this one soon.

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[bdovaz]

No stale

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

Note that partial clones are now supported by Azure Repos!

[bdovaz]

Although it is an improvement, it has nothing to do with the purpose of this issue.

The big improvement would be that at a high level through the checkout task we can make a sparse checkout indicating the directories to include as you can do in Github from your PR from half a year ago.

[sanghel-orbyta]

any way to bump priority on this one?

monorepos consume agent time uselessly doing full checkouts...

[ToreDemant]

Seems to be related to these:
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
https://developercommunity.visualstudio.com/t/Support-Gits-sparse-checkout-in-the-Azu/10472337

[Khayoann2]

I proposed this one
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
So i’m also waiting this feature …
Please 🙏

[mggrand]

If anyone is interested, I wrote the sparse checkout as a template, which can then be used as follows:

- template: sparse-checkout.yml
    parameters:
      folders:
        - frontend
        - resources

with the template sparse-checkout.yml

parameters:
  - name: folders
    type: object
    default:

  - name: fetchLfs
    type: boolean
    default: true

  - name: persistCredentials
    type: boolean
    default: false

steps:
  - checkout: none
  - pwsh: |
      function Start-Group($name){ Write-Host "##[group]$name"}
      function End-Group { Write-Host "##[endgroup]"}

      function Run-Cmd($command){
          Write-Host "##[command]$command"
          Invoke-Expression $command
      }


      $folders = "${{ join(' ',parameters.folders) }}"
      $repositoryUri = "$(Build.Repository.Uri)"
      Write-Host "Sync repository folders '$folders' to '$pwd'"
            
      Start-Group "Getting git version info"
      Run-Cmd "git version"
      Run-Cmd "git lfs version"
      End-Group

      if(Test-Path ".git"){
        Start-Group "Cleanup existing repository"
        # Existing directory
        $null = Remove-Item ".git/index.lock" -Force -ErrorAction SilentlyContinue
        $null = Remove-Item ".git/shallow.lock" -Force -ErrorAction SilentlyContinue

        $currentRemote = Run-Cmd "git remote get-url origin"
        if($currentRemote -ne $repositoryUri){
            Write-Host "Current fetch url: $currentRemote"
            Write-Host "New repository fetch url: $repositoryUri"
            Write-Host "##[warning]Fetch urls do not match. Clean the whole directory"
            $null = Remove-Item ./* -Force -Recurse
        }
        else{
            Run-Cmd "git remote remove origin"
            Run-Cmd "git clean -ffxd"
            Run-Cmd "git reset --hard"
        }
        End-Group
      }

      Start-Group "Initialize the repository"
      Run-Cmd "git init"
      Run-Cmd "git sparse-checkout init --cone"
      if($folders.Length -gt 1){
        Run-Cmd "git sparse-checkout add $folders"
      } else {
        Write-Host "No folders were provided for the sparse checkout"
      }
      Run-Cmd "git lfs install --local"
      End-Group

      Start-Group "Configure the repository"
      Run-Cmd "git remote add origin $repositoryUri"
      Run-Cmd "git config core.sparsecheckout true"
      Run-Cmd "git config gc.auto 0"
      Run-Cmd "git config --get-all http.$($repositoryUri).extraheader"
      Run-Cmd "git config --get-all http.proxy"
      Run-Cmd "git config http.version HTTP/1.1"
      End-Group


      Start-Group "Fetching the repository"
      Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader fetch --force --no-tags --progress --prune --no-recurse-submodules --verbose --depth=1 origin +$(Build.SourceVersion):refs/remotes/origin/$(Build.SourceVersion)"
      if("${{ parameters.fetchLfs }}" -eq "true"){
        Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader lfs fetch origin refs/remotes/origin/$(Build.SourceVersion)"
      } else {
        Write-Host "Skipping lfs fetch"
      }
      End-Group


      Start-Group "Checkout the repository"
      Run-Cmd "git checkout --force --progress refs/remotes/origin/$(Build.SourceVersion)"
      End-Group

      if("${{ parameters.persistCredentials }}" -eq "true"){
        Start-Group "Persist credentials in the repository"
        Run-Cmd "git config http.$($repositoryUri).extraheader `"$($env:env_var_http_extraheader)`""
        End-Group
      }

      if("$($env:env_var_vsts_debug)" -eq "true"){ 
        Start-Group "Print directory outputs"
        Run-Cmd "Get-ChildItem -Recurse"
        End-Group
      }

    displayName: "Clone partial git repo"
    env:
      env_var_http_extraheader: "AUTHORIZATION: bearer $(System.AccessToken)"
      env_var_vsts_debug: $[eq(variables['System.Debug'], true)]

[bdovaz]

https://developercommunity.visualstudio.com/t/Support-Gits-sparse-checkout-in-the-Azu/10472337#T-N10776291

[bdovaz]

Added in #5015 and will be in release v3.247.1

Closing...

[bdovaz]

There is no documentation for this feature yet?

https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines

cc @jnilau @merlynomsft

[PaulVrugt]

docs would be really helpful

[PaulVrugt]

so, trying to use this in azure devops gives me:

(Line: 91, Col: 9): Unexpected value 'sparseCheckoutDirectories'

and another Object reference not set to an instance of an object. error

could it be that azure devops still needs an update before this is supported?

[ChristopherHX]

ADO has exactly two syntaxes, the less familiar one is working right know in dev.azure.com, the short notation needs server side updates to the schema and eventually more.

6d15af64-176c-496d-b583-fd2ae21d4df4 is the taskid of the agents checkout task, version 1 the only available version.

variables:
- name: AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK
  value: true
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4@1
  inputs:
    sparseCheckoutDirectories: .vscode
    repository: self
- script: ls -la

If you need AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK or not has not been investigated

You would see the git sparse commands in the shell.

 * branch            6ccd4af5a724d5b17713a6280806877f23a8fabe -> FETCH_HEAD
git sparse-checkout init --cone
git sparse-checkout set .vscode
git checkout --progress --force refs/remotes/origin/6ccd4af5a724d5b17713a6280806877f23a8fabe