Merge branch 'master' into 451-enable-streaming-of-build-logs

seqeralabs · Jul 22, 2024 · 7dedb5f · 7dedb5f
2 parents fd9ea32 + 760f217
commit 7dedb5f
Show file tree

Hide file tree

Showing 92 changed files with 1,624 additions and 267 deletions.
diff --git a/.codespellignore b/.codespellignore
@@ -1 +1,2 @@
 carrer
+ser
diff --git a/.github/workflows/typespec.yml b/.github/workflows/typespec.yml
@@ -0,0 +1,40 @@
+---
+name: Typespec_Validation
+
+on:
+  push:
+    branches:
+      - '**'
+    paths :
+      - 'typespec/**'
+  pull_request:
+    types: [opened, reopened, synchronize]
+    paths:
+      - 'typespec/**'
+
+permissions:
+  contents: read
+
+jobs:
+  typespec_validation:
+    name: validate typespec files
+    runs-on: ubuntu-latest
+
+    steps:
+      - name : Checkout
+        uses : actions/checkout@v4
+
+      - name : Setup Node.js environment
+        uses : actions/setup-node@v4
+        with :
+          node-version : '20.9.0'
+
+      - name : Install tsp
+        run : npm install -g @typespec/compiler
+
+      - name : Validate tsp files
+        run : |
+          cd typespec
+          tsp install
+          tsp compile .
+
diff --git a/.gitignore b/.gitignore
@@ -32,3 +32,8 @@ scan-workspace/
 .cache
 site/
 deployment-url.txt
+
+#typespec
+tsp-output/
+node_modules/
+package-lock.json
diff --git a/README.md b/README.md
@@ -14,7 +14,7 @@ images.
 * Augment container images i.e. dynamically add one or more container layers to existing images;
 * Build container images on-demand for a given container file (aka Dockerfile);
 * Build container images on-demand based on one or more Conda packages;
-* Build container images on-demand based on one or more Spack packages;
+* Build container images on-demand based on one or more Spack packages, Spack support will be removed in future releases;
 * Build container images for a specified target platform (currently linux/amd64 and linux/arm64);
 * Push and cache built containers to a user-provided container repository;
 * Build Singularity native containers both using a Singularity spec file, Conda package(s) and Spack package(s);
@@ -108,6 +108,17 @@ container registry where the image is stored, while the instrumented layers are
     '-Djdk.httpclient.HttpClient.log=requests,headers'
     ```
 
+## TypeSpec API Specifications
+
+- You can find the API specifications using (typespec)[https://github.com/microsoft/typespec] in typespec directory. Use following command to generate the API specifications.
+
+    ```bash
+    'cd typespec'
+    'tsp install'
+    'tsp compile .'
+    ```
+
+- Check `typespec/tsp-output` directory for the generated API specifications.
 
 ## Related links
 * [Wave command line tool](https://github.com/seqeralabs/wave-cli)

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-1.8.0
+1.9.0
diff --git a/build.gradle b/build.gradle
@@ -35,7 +35,7 @@ dependencies {
     implementation("jakarta.persistence:jakarta.persistence-api:3.0.0")
     api 'io.seqera:lib-mail:1.0.0'
     api 'io.seqera:wave-api:0.10.0'
-    api 'io.seqera:wave-utils:0.12.0'
+    api 'io.seqera:wave-utils:0.13.1'
 
     implementation("io.micronaut:micronaut-http-client")
     implementation("io.micronaut:micronaut-jackson-databind")

diff --git a/changelog.txt b/changelog.txt
@@ -1,4 +1,38 @@
 # Wave changelog
+1.9.1 - 11 Jul 2024
+- Prevent hard error when launch credentials cannot be accessed [a318a855]
+
+1.9.0 - 11 Jul 2024
+- Add Typespec API definitions (#537) [32f7dd16]
+- Add cache record-stats (#534) [229926e2]
+- Add http 429 error to auth service retry condition [8282a492]
+- Check and delete corrupted blobs cache uploads (#533) [b0c775a3]
+- Deprecate the support for Spack and remove the support for it in the codebase (#550) [85a05196]
+- Enable ECR authentication via AWS compute env credentials (#303) [ec895222]
+- Fix Blob cache config for local dev [0d5413bf]
+- Fix client cache deadlock (#547) [cc6012ff]
+- Fix multiple s3clients in wave (#554) [45689500]
+- Minor change in mail notification [0aba6997]
+- Refactored metrics service (#549) [5e0d32ac]
+- Remove buildlogs aws config to not initialize AwsS3Operations (#558) [d50e5b7a]
+- Update metrics response (#536) [b6b36a97]
+- Bump buildkit 0.14.0 (#528) [c54172d1]
+- Bump buildkit 0.14.1 (#548) [37afb782]
+- Bump trivy 0.53.0 [f27ff527]
+- Bump version 1.8.3 [55b473e1]
+
+1.8.3 - 29 Jun 2024
+- Fix client cache deadlock (#547) [cc6012ff]
+- Fix failing test [263b44d3]
+- Add cache record-stats (#534) [229926e2]
+
+1.8.2 - 14 Jun 2024
+- Fix get image auth digest (#530) [df8ec041]
+
+1.8.1 - 13 Jun 2024
+- Fix digest warning (#526) [6e34ff0a]
+- Fix image name strategy (#513) [3bd8ffb5]
+
 1.8.0 - 10 Jun 2024
 - Add breakdown per org on stats metrics when a date is specified (#517) [b24894a4]
 - Replace Kaniko with Buildkit (#503) [a26f5dfe]

diff --git a/configuration.md b/configuration.md
@@ -109,6 +109,8 @@ Below are the standard format for known registries, but you can change registry
 
 ### Spack configuration for wave build process
 
+**Note**: Spack support will be removed in future releases.
+
 Spack configuration consists of the path of its secret file, the mount path for the secret file in the spack container, and the optional S3 bucket name for the spack binary cache.
 
 **Note**: these configuration are mandatory to support Spack in a wave installation.

diff --git a/docs/_images/wave_container_build_failure_details.png b/docs/_images/wave_container_build_failure_details.png
diff --git a/docs/api.mdx b/docs/api.mdx
@@ -67,7 +67,7 @@ This API endpoint is deprecated in current versions of Wave.
 | `containerConfig.layers.gzipSize`   | The size in bytes of the the provided layer tar gzip file.                                                                                         |
 | `containerFile`                     | Dockerfile used for building a new container encoded in base64 (optional). When provided, the attribute `containerImage` must be omitted.        |
 | `condaFile`                         | Conda environment file encoded as base64 string.                                                                                                   |
-| `spackFile`                         | Spack recipe file encoded as base64 string.                                                                                                        |
+| `spackFile`                         | `Deprecated` Spack recipe file encoded as base64 string. Spack support will be removed in future releases.                                                                                                        |
 | `containerPlatform`                 | Target container architecture of the built container, e.g., `linux/amd64` (optional). Currently only supporting amd64 and arm64.                     |
 | `buildRepository`                   | Container repository where container builds should be pushed, e.g., `docker.io/user/my-image` (optional).                                            |
 | `cacheRepository`                   | Container repository used to cache build layers `docker.io/user/my-cache` (optional).                                                              |
@@ -168,7 +168,7 @@ The endpoint returns the name of the container request made available by Wave.
 | `containerConfig.layers.gzipSize`   | The size in bytes of the the provided layer tar gzip file.                                                                                         |
 | `containerFile`                     | Dockerfile used for building a new container encoded in base64 (optional). When provided, the attribute `containerImage` must be omitted.        |
 | `condaFile`                         | Conda environment file encoded as base64 string.                                                                                                   |
-| `spackFile`                         | Spack recipe file encoded as base64 string.                                                                                                        |
+| `spackFile`                         | `Deprecated` Spack recipe file encoded as base64 string. Spack support will be removed in future releases.                                                                                                           |
 | `containerPlatform`                 | Target container architecture of the built container, e.g., `linux/amd64` (optional). Currently only supporting amd64 and arm64.                     |
 | `buildRepository`                   | Container repository where container builds should be pushed, e.g., `docker.io/user/my-image` (optional).                                            |
 | `cacheRepository`                   | Container repository used to cache build layers `docker.io/user/my-cache` (optional).                                                              |
@@ -178,9 +178,9 @@ The endpoint returns the name of the container request made available by Wave.
 | `towerEndpoint`                     | Seqera Platform service endpoint from where container registry credentials are retrieved (optional). Default `https://api.cloud.seqera.io`.                       |
 | `towerAccessToken`                  | Access token of the user account granting access to the Seqera Platform service specified via `towerEndpoint` (optional).                      |
 | `towerWorkspaceId`                  | ID of the Seqera Platform workspace from where the container registry credentials are retrieved (optional). When omitted the personal workspace is used. |
-| `packages`                          | This object specifies Conda or Spack packages environment information.                                                                                         |
+| `packages`                          | This object specifies Conda packages environment information.                                                                                         |
 | `environment`                       | The package environment file encoded as a base64 string.                                                                                                       |
-| `type`                              | This represents the type of package builder. Use `SPACK` or `CONDA`.                                                                                   |
+| `type`                              | This represents the type of package builder. Use `CONDA`.                                                                                   |
 | `entries`                           | List of the packages names.                                                                                                                                    |
 | `channels`                          | List of Conda channels, which will be used to download packages.                                                                                               |
 | `mambaImage`                        | Name of the docker image used to build Conda containers.                                                                                              |
@@ -285,13 +285,11 @@ Provides status of build against buildId passed as path variable
 
 ```json
 {
-    serviceInfo: {
-        id: string,
-        status: string,
-        startTime: string,
-        duration: string,
-        succeeded: boolean
-    }
+    id: string,
+    status: string,
+    startTime: string,
+    duration: string,
+    succeeded: boolean
 }
 ```
 
@@ -508,7 +506,7 @@ curl --location 'http://localhost:9090/v1alpha1/inspect' \
 These APIs provide usage (builds and pulls) metrics of Wave for a specific date and/or a specific organization.
 These APIs require basic authentication, so you must provide a username and password while calling these APIs.
 
-All Metrics API endpoints use these query parameters (at least one is required):
+All Metrics API endpoints use these query parameters:
 
 | Name | Description                                                           | sample Value |
 |------|-----------------------------------------------------------------------|--------------|
@@ -521,63 +519,125 @@ These APIs are used to retrieve metrics about container builds performed by Wave
 
 ### GET `/v1alpha2/metrics/builds`
 
-This endpoint is used to retrieve the total number of builds performed by Wave.
+This endpoint is used to retrieve the builds performed by Wave.
 
 ### Response
 
 ```json
 {
-    count: integer
+    metric: "builds",
+    count: integer,
+    orgs: {
+        String: integer,
+        String: integer,
+        ...
+    }
 }
 ```
 
 #### Examples
 
+```shell
+curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/builds"
+{
+    "metric": "builds",
+    "count": 18,
+    "orgs": {
+        "seqera.io": 13,
+        "gmail.com": 5
+    }
+}
+```
+
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/builds?date=2024-04-08&org=seqera.io"
 {"count":4}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/builds?date=2024-04-08"
-{"count":6}
+{
+    "metric": "builds",
+    "count": 8,
+    "orgs": {
+        "gmail.com": 4,
+        "seqera.io": 4
+    }
+}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/builds?org=seqera.io"
-{"count":4}
+{
+    "metric": "builds",
+    "count": 13,
+    "orgs": {
+        "seqera.io": 13
+    }
+}
 ```
 ### Pull Metrics API
 
 These APIs are used to get the metrics about the container pulls through Wave.
 
 ### GET `/v1alpha2/metrics/pulls`
 
-This endpoint is used to get the total numbers of pulls performed through Wave.
+This endpoint is used to get the pulls performed through Wave.
 
 ### Response
 
 ```json
 {
-    count: integer
+    metric: "pulls",
+    count: integer,
+    orgs: {
+        String: integer,
+        String: integer,
+        ...
+    }
 }
 ```
 
 #### Examples
 
+```shell
+curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/pulls"
+{
+    "metric": "pulls",
+    "count": 11,
+    "orgs": {
+        "seqera.io": 7,
+        "gmail.com": 4
+    }
+}
+```
+
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/pulls?date=2024-04-08&org=seqera.io"
 {"count":5}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/pulls?date=2024-04-08"
-{"count":7}
+{
+    "metric": "pulls",
+    "count": 4,
+    "orgs": {
+        "seqera.io": 1,
+        "gmail.com": 3
+    }
+}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/pulls?org=seqera.io"
-{"count":5}
+{
+    "metric": "pulls",
+    "count": 7,
+    "orgs": {
+        "seqera.io": 7
+    }
+}
 ```
 
 ### Fusion Pull Metrics API
@@ -586,29 +646,59 @@ These APIs are used to get the metrics about the Fusion-based container pulls th
 
 ### GET `/v1alpha2/metrics/fusion/pulls`
 
-This endpoint is used to get the total numbers of pulls of Fusion-based containers performed through Wave.
+This endpoint is used to get the pulls of Fusion-based containers performed through Wave.
 
 ### Response
 
 ```json
 {
-    count: integer
+    metric: "fusion",
+    count: integer,
+    orgs: {
+        String: integer,
+        String: integer,
+        ...
+    }
 }
 ```
 
 #### Examples
 
+```shell
+curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/fusion/pulls"
+{
+    "metric": "fusion",
+    "count": 2,
+    "orgs": {
+        "seqera.io": 1,
+        "gmail.com": 1
+    }
+}
+```
+
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/fusion/pulls?date=2024-04-08&org=seqera.io"
 {"count":2}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/fusion/pulls?date=2024-04-08"
-{"count":2}%
+{
+    "metric": "fusion",
+    "count": 1,
+    "orgs": {
+        "gmail.com": 1
+    }
+}
 ```
 
 ```shell
 curl -u foo:bar "http://localhost:9090/v1alpha2/metrics/fusion/pulls?org=seqera.io"
-{"count":2}
+{
+    "metric": "fusion",
+    "count": 1,
+    "orgs": {
+        "seqera.io": 1
+    }
+}
 ```