Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.4.0] Output reuse #23674

Merged
merged 4 commits into from
Sep 19, 2024
Merged

[7.4.0] Output reuse #23674

merged 4 commits into from
Sep 19, 2024

Commits on Sep 19, 2024

  1. Reapply "[7.3.0] Deduplicate locally executed path mapped spawns (baz… 

    …elbuild#23069)"

This reverts commit 371fbf6.
    @fmeum
    fmeum committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    ef928d9 View commit details
    Browse the repository at this point in the history

  2. Fix local execution deduplication to work with optional outputs 

    This fixes failures such as the following when a spawn, e.g. a reduced Java compilation spawn, doesn't create an optional output:

```
java.io.FileNotFoundException: /private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/execroot/_main/bazel-out/_tmp/actions/remote/175.tmp (No such file or directory)
	at com.google.devtools.build.lib.unix.NativePosixFiles.lstat(Native Method)
	at com.google.devtools.build.lib.unix.UnixFileSystem.statInternal(UnixFileSystem.java:212)
	at com.google.devtools.build.lib.unix.UnixFileSystem.stat(UnixFileSystem.java:201)
	at com.google.devtools.build.lib.vfs.Path.stat(Path.java:290)
	at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:456)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.moveOutputsToFinalLocation(RemoteExecutionService.java:878)
        ...
```

Also clean up temporary files in case of an exception.

Work towards bazelbuild#23288

Closes bazelbuild#23296.

PiperOrigin-RevId: 665744936
Change-Id: I89a409c7a6b28b2a5fa532bdb233dca9bc5bde73
    @fmeum
    fmeum committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    654fe31 View commit details
    Browse the repository at this point in the history

  3. Force synchronous upload and reuse of possibly modified spawn outputs 

    When an action may modify a spawn's outputs after execution, the upload of outputs to the cache and reuse for deduplicated actions need to happen synchronously directly after spawn execution to avoid a race.

This commit implements this for cache uploads by marking all actions with this property and simply disabling async upload for all spawns executed by such actions.

For output reuse, all executions deduplicated against the first one register atomically upon deduplication and cause the cache upload to wait for all of them to complete reuse.

Fixes bazelbuild#22501
Fixes bazelbuild#23288
Work towards bazelbuild#21578
Closes bazelbuild#23307 (no longer needed)

Closes bazelbuild#23382.

PiperOrigin-RevId: 668101364
Change-Id: Ice75dbe14a7dd46e02ecb096d2b2a30940216356
    @fmeum
    fmeum committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5d68e1f View commit details
    Browse the repository at this point in the history

  4. Add support for in-memory outputs to output reuse 

    Closes bazelbuild#23422.

PiperOrigin-RevId: 669296792
Change-Id: If3a2519cbf57cc82f1f733900b93705065b37e63
    @fmeum
    fmeum committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    00be245 View commit details
    Browse the repository at this point in the history