Skip to content

Releases: thoth-station/thoth-application

Release v2022.05.30

31 May 19:01
@harshad16 harshad16
v2022.05.31
d17fc9c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release v2022.05.30

we have completed the release for v2022.05.30 🎉 🎊 🥳

Features

There are two main themes for this sprint:

  • Continue with the integration test improvements
  • Increase advise manager usage

Memory and CPU resource allocation
The process of the memory and CPU resource allocation used by the adviser component for guidance stack resolution is documented in video format.

Include Tensorflow 2.9.0 APIs symbols to resolver
With the release of the TensorFlow 2.9.0, we have included the API symbols to the resolver, so it could use them for lookup while trying to resolve a stack requesting TensorFlow 2.9.0.

Handle too many requests error (429) in gh- handlers*
Handle 429 HTTP errors occur when too many requests are made to the GitHub API, leading to prescriptions being wrongly deleted as the URL is recognized as invalid or non-existent.
This change will allow the containers running-related jobs to exit and restart properly when the request quota for the GitHub API token in use is reached.

Use the new format of the OSSF security cards
Consume things computed by security scorecards in Thoth recommendations, Update the Thoth's implementation to use the v3 scorecards implementation instead of the old scorecards API.

Component Updates

Thanks for the amazing work everyone. 💯

What's Changed

Full Changelog: v2022.05.10...v2022.05.31

Contributors

fridex, harshad16, and 2 other contributors
Assets 2
Loading

Release 2022.05.09

10 May 10:47
@harshad16 harshad16
v2022.05.10
6572957
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.05.09

we have completed the release for v2022.05.09 🎉 🎊 🥳

Features

Github action that integrates Thoth

GitHub action on github repository that integrates with Thoth. The GitHub action can use Thamos CLI to contact Thoth backend to get results. For example, if there is a security vulnerability, the GitHub action can turn pull-request to red state, eventually blocking merge.

This feature is an addition to Kebechet for teams that would like to consume Thoth recommendations but do not want to install kebechet on their own. The GitHub action could be designed in a way so that users do not need to keep configuration in their repositories at all (ex. detect where requirements are stated and such).

Organize stack trace / justifications
Expect data displayed in a readable way. In the Advice details tab, in the justifications card, all justifications are displayed in a list that is hard to read. Organize the justifications into either their own separate cards or into some other organized fashion.

Component Updates

Thanks for the amazing work everyone. 💯

What's Changed

Full Changelog: v2022.04.18...v2022.05.10

Contributors

codificat, tumido, and 2 other contributors
Assets 2
Loading

Release 2022.04.18

19 Apr 07:38
@harshad16 harshad16
v2022.04.18
558dd20
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.04.18

we have completed the release for v2022.04.18 🎉 🎊 🥳

Features

User API drops count and limit parameters on advice endpoint

Starting this release, user-API no longer provides count and limit as parameters on the advised endpoints. These parameters were not used. Note this is a breaking change on API endpoints.

Fix scoring of the user stack supplied

Users could be confronted with issues when adding a new package to requirements and submitting the lock file to the backend. The backend did not invalidate the lock file sent as it no longer corresponded to the adjusted requirements.

Thoth Search UI moves to Typescript and adds a advise document compare screen
Users can now compare multiple advice documents to see the differences and similarities between them. They can access old advice documents through a local history feature added. The home page now has a button for viewing past runs.

Component Updates

What's Changed

Full Changelog: v2022.03.28...v2022.04.18

Contributors

goern, fridex, and 3 other contributors
Assets 2
Loading

Release 2022.03.28

28 Mar 17:05
@harshad16 harshad16
v2022.03.28
16236aa
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.03.28

we have completed the release for v2022.03.28 🎉 🎊 🥳

Announcements

  • Team would be switching to 3 weeks sprint cycles.
  • GitHub Project would be used for managing SIG Workload
  • Subscribe to the Thoth-Station calendar for the change of meeting timings.

Features

Ingesting data for Fedora 35 running Python 3.10
we plan to support Fedora 35 running Python 3.10 as a new runtime environment

Integration tests for prod and smaug deployment
Updates are made to establish integration tests against the prod and smaug(semi-prod) deployments

Use of GitHub projects for each SIG
For each Special interest group, the team has created GitHub projects respectively to track the work more efficiently.

Extend issue body when opening pull requests with configuration
Extended PR body for information on how to configure Kebechet on my repository. The body of PRs could be extended to guide users on how to properly configure Kebechet.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release 2022.03.14

14 Mar 13:21
@harshad16 harshad16
v2022.03.14
149eb93
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.03.14

we have completed the release for v2022.03.14 🎉 🎊 🥳

Features

Notify users when they use PyTorch index

Newly, the resolver notifies users if they consume releases from the PyTorch index.

Memory optimizer in adviser

If Thoth's adviser consumes too much memory and is killed on OOM in a deployment, a memory optimizer can be turned on. The memory optimizer removes data from some of the internal data structures to optimize memory consumption. This can slow down finding a resolved set of dependencies or making the search not that optimal. Users, however, can get results instead of OOM.

Created new handler that uses image analysis results and generate new prescriptions

Automatically propagated from the container image analyses - the container image analysis would provide how Pipfile looked like on image build and that information will be part of the image analyses information derived by package-extract (similarly as we propagate information about RPM packages). In this case, we implemented a handler in prescriptions-refresh-job that:

  • checks what ps images are hosted on quay
  • ask what is the container image analysis result and how the Pipfile looked like during the build
  • automatically create prescriptions based on direct dependencies in Pipfile

References:

Automatically bump base image versions to latest available on Quay

A new script integrated in the aicoe-ci pipeline allows to automatically update the base image versions present in configuration files such as .aicoe-ci.yaml to the latest available on Quay. The pipeline is triggered by an action on a repository such as the opening of a pull request or an issue. If the base image versions are not up-to-date compared on available versions on Quay, a pull request is automatically opened for the corresponding update.
For an example, see: thoth-station/package-releases-job#637 which was triggered by thoth-station/package-releases-job#636 on the package-releases-job repository.

Fix method to iterate on thoth files

slo-reporter was not able to provide an analysis of adviser results. advise-reporter logic was not able to read adviser files, due to a method in thoth-storages. Iterating over files in a bucket gives empty lists even if files exist because the prefix used to identify them is using a default value that is not updated. The value RESULT_TYPE is fixed.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release v2022.02.28

28 Feb 19:02
@harshad16 harshad16
v2022.02.28
024b778
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release v2022.02.28

we have completed the release for v2022.02.28 🎉 🎊 🥳

Features

Add sorting to User API query requests

Responses that include sortable data will have a new param that allows sorting the data before paginating it.

Added feature to initiate GitHub repo with thoth requirement via API

Users can submit their GitHub repositories to get initiated with thoth setup for that repository.

Fixed response of the image inspection via skopeo

Modify the ImageMetadataResponse OpenAPI schema to correspond to the values returned by the skopeo-inspect section of package-extract documents.

Fixed issue due to broken links in package extraction

Syncing the data from the package extraction phases caused abnormality in the python interpreter table, the issue was fixed by resolving the broken links read up.

Expose metrics about the last run of solver

This metric can also help with information when the last database sync was done as solvers are components that are supposed to be running constantly in a deployment that does ingestion.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release 2021.02.14

14 Feb 21:23
@harshad16 harshad16
v2022.02.14
3b98093
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2021.02.14

we have completed the release for v2022.02.14 🎉 🎊 🥳

Features

Optimized syncs across deployments in document-sync-job

Our last release introduced a new component called document-sync-job which can sync documents created in one Thoth deployment to another. This release comes with an optimized implementation of document-sync-job which can sync documents concurrently. This optimization was required as the number of documents we need to handle is too large.

Pedantic thamos run feature

Our command-line interface, Thamos, now performs pedantic application runs. As thamos run acts as a wrapper for Python interpreter for spawning Python applications (thamos manages virtual environments automatically for users if configured so), it can now also check that the configuration supplied by the user matches the configuration of the runtime environment used. An example can be a requirement on CUDA, eventually in a specific version. If CUDA in the specified version is not present in the runtime environment, thamos run refuses to run the application (by default). This way users can ensure that the requirements on the runtime environment they use match expectations recorded in Thoth's configuration file. Moreover, thamos run newly also checks that the lockfile respects direct dependencies (no re-locking is needed).

These checks can be suppressed by using --no-pedantic option or by providing THAMOS_RUN_NO_PEDANTIC=1 environment variable to the runtime environment.

thamos run allows specifying environment variables to the Python process run

Starting this release, users can declare their environment variables in .env files configured per overlay. This configuration follows Pipenv's .env file configuration, except Thoth users can declare different environment variables to be supplied to the process based on overlay/runtime environment used.

Pulp Grafana Dashboard

A new Grafana Dashboard for Pulp deployed on Operate First is available at Grafana Operate First: https://grafana.operate-first.cloud/d/B7DLT7anz/pulp-metrics?orgId=1

graph-backup-job
New metrics were added to monitor the creation of pg dumps and if any issue appears creating pg dumps.

graph-metrics-exporter
Introduce a new task for monitoring pg_dumps on Ceph. In this way, we can verify if all backups are created correctly.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release 2022.01.31

01 Feb 00:57
@harshad16 harshad16
v2022.02.01
120af32
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.01.31

we have completed the release for v2022.01.31 🎉 🎊 🥳

Features

document-sync-job

A new component called document-sync-job is responsible for syncing documents across deployments. It helps to manage automatic syncs of data that are computed in one deployment (ex. OCP4 stage environment) to another deployment (ex. prod deployment). This way, we can make sure only one deployment computes data (and thus requires more resources) that are automatically propagated to deployments that need it.

pulp-metrics-exporter

To monitor Pulp instance on Operate First, we have created pulp-metrics-exporter component. This component is following design principles used in Thoth's metrics-exporter and provides metrics about Pulp instance and its python_plugin as configured. As Pulp team does not have any mechanism to expose metrics as of now, we provide pulp-metrics-exporter to monitor the Operate First Pulp instance until they come up with a solution they want to maintain and use. Eventually, pulp-metrics-exporter will be offered to the Pulp team as a project they could use, redesign, experiment or start with.

Query containerized environments based on content

It is now possible to query analyzed containerized environments based on more provided content (symbols, RPM packages and Python packages) via Thamos running thamos images and via the user API with new parameters added to the API endpoints.

Example for retrieving an image with the GLIBC_FOO symbol using Thamos:

thamos images --symbol GLIBC_FOO

NEW dependency metric card in Thoth Search package overview

When searching for a package, you will now be presented with a metric card with all required and extra dependencies. This also serves as a navigation between other packages by selecting a version from a dropdown menu.

Availability of a public Thoth database dump for the community

A minimal dump of Thoth's database is now available on the Operate First public bucket opf-datacatalog at s3://opf-datacatalog/thoth/datasets/thoth_public_database.sql for external contributors to develop components of Thoth. A dump can be automatically generated using a simple script that removes sensitive information from the database.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release 2022.01.17

17 Jan 18:45
@harshad16 harshad16
v2022.01.17
e33fd34
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2022.01.17

we have completed the 1st release of 2022.
v2022.01.17 🎉 🎊 🥳

Features

TensorFlow 2.8 symbols

Thoth's database of TensorFlow symbols has been updated. It now keeps also symbols for the upcoming TensorFlow~=2.8.0 releases. This way, users of Thoth can get recommendations based on TF API used.

API endpoint for listing available Python package versions for environments

Starting this release, consumers of Thoth's user-api endpoints can list Python versions solved based on environments. An example can be an HTTP GET request to obtain all the releases of flask solved by Thoth for UBI8 Python 3.8 environment.

API endpoint for obtaining Python package metadata and dependency information

Yet another new endpoint exposes information about Python package metadata computed and kept up to date by the system. This endpoint shows metadata for versions of Python packages, but also includes dependency information that cannot be found on PyPI - this dependency information is specific for environments used to solve packages and does not include just "latest" as in case of deps.dev. The dependency information is kept up to date by the system (based on new releases monitored on indexes, analyzed, and solved by Thoth). This new endpoint replaces /python/package/metadata which is now obsolete and will be removed in one of the upcoming releases

Proper paginating in User API REST endpoints

Consumers of Thoth User API REST endpoints are now able to browse exposed content respecting pagination. The pagination information is available in HTTP response headers on endpoints that provide paginated entries. Notably, HTTP headers provide:

  • next - for the next page in the paginated response, if any
  • prev - for the previous page in the paginated response, if any
  • page - page number to show, allows also negative indexing which loops the pagination
  • per_page - number of entries shown per single page
  • entries_count - number of entries in total
  • page_count - number of pages available

Related:

thamos verify command to check lockfile hash

A new sub-command verifies the correctness of requirements based on their hash. This way, the newly introduced sub-command thamos verify shows if requirements for the configured runtime environments correspond to the lockfile created. By doing so, users can make sure there are no changes made that would require re-triggering new advise requests to the backend to generate a new lock file. This sub-command can be also used in tooling that uses Thamos to verify requirements were not adjusted and do not require a new lockfile.

Prescription refresh workflow has a new handler for image analysis

This handler uses information collected from image analysis to create prescriptions about container images. In this way, users can identify secure, maintained containers to run their applications directly from Thoth recommendations.

Polished and updated OpenAPI specification for Thoth User API

This release is shipped with polished and updated OpenAPI specification which documents endpoints, inputs, responses:

Added detailed Python package routing in Thoth Search

This release uses the new User API metadata endpoint that breaks down metadata into OS name, version, and Python version. Selectors were added to accommodate this change .

Prioritizing Thoth data and changing the name to search

In this release, Thoth Search is renamed to search and no longer uses PyPi data as a source. If Thoth has no info on a package, then it is given null values.

Managing Vulnerabilities with Thoth Tutorial

The Managing Vulnerabilities with Thoth tutorial based on cli-examples publicly accessible on Red Hat Scholars provides a reproducible example of vulnerability management for the stack of a simple application via the Thamos CLI, and contains further references to other functionalities available with Thoth.

Component Updates

Thanks for the amazing work everyone. 💯

Assets 2
Loading

Release 2021.12.20

20 Dec 22:15
@harshad16 harshad16
v2021.12.20
7b9326c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2021.12.20

we have completed the release of 2021.12.20 🎉 🎊 🥳

Features

List available environments

Starting this release, users can issue thamos environments to list available environments for the resolution. This allows users to ask which environments are available and can be used in Thoth's configuration file:

Warn if users use too lax versions in their requirements

Resolver warns if users use too lax versions in their requirements. It is a good practice to specify desired versions in the requirements file that are tested and expected to work, not to introduce overpinning issues.

Links to Search UI

Thanks to Thoth Search project, people can browse the resolver results in a web browser. The link to Thoth Search can be found in each resolver result (might not immediately work with the current release as Thoth Search is to be deployed).

The resolver also creates a link for each resolved package to the corresponding Thoth Search UI package entry:

Create an example application in cli-examples to demonstrate Thamos dependency resolution

The game_of_life.py simple application allows users to understand how Thoth detects vulnerabilities in an application software stack and prevents the import of vulnerable packages when put on the security setting.

Warn if a package has no recent releases on PyPI

We have added prescriptions that warn users if they use a package that has no recent releases on PyPI (180+ days). This might indicate that the project is unmaintained -- especially when combined together with other prescriptions that we have about projects (such as repo activity). Prescriptions created are automatically updated by the prescriptions-refresh-job run periodically in deployment.

Warnings produced based on PyPI maintainers

Newly, resolver warns about using projects that can have suspicious behaviour based on statistics computed from PyPI maintainers info:

  • resolver warns if a package has not enough maintainers on PyPI
  • resolver warns if a package is maintained by maintainers that maintain a small number of projects on PyPI
  • resolver warns if all package maintainers for a package hosted on PyPI have joined PyPI just recently

Related:

Prescriptions produced to inform users on the number of downloads for a given package

New prescriptions inform users on the number of downloads for a given package on PyPI and on the number of downloads for each package version in the last 180 days according to data from the PyPI downloads BigQuery dataset. Prescriptions for a project include the most downloaded project version, and prescriptions for each project version score the popularity of the given version.

Installed Python artifact size

Yet another new set of prescriptions show to users the downloaded artifact size that will be eventually installed. Note that this is just an estimate (upper limit) as the actual artifact installed depends on decisions done by pip during the resolution for the target runtime environment. Reports show information only for artifacts that have 3MiB+.

Size of container images used

Similar to Python artifact size, users get notified about container image size if Thoth containerized environments are used. The container image size is automatically recomputed in prescriptions-refresh-job each time there is a versioned release done by AICoE-CI.

Migrated thoth-search repo over to Thoth Station

The Thoth Search app has been transferred over to Thoth Station. Bots have been added and pre-commits have been configured. Also there has been changes in the CI/CD GitHub workflow that now check for ESlint errors and prettier formatting issues. The app is deployed at https://thoth-station.ninja/thoth-search/

Filtering Python packages used during the resolution based on index labels supplied to the resolver

Users can now use labels to distinguish Python package indexes that should be used during the resolution process. This is an addition to the strict index configuration.

An example can be an advise for a Python application that should not use packages hosted on PyPI nor PyTorch CUDA 11.1 index but use all the others monitored by Thoth:

thamos advise --labels "pypi-index=disabled,pytorch-cu111-index=disabled"

or use solely packages hosted on Pulp indexes on the Operate First instance:

thamos advise --labels "opf-pulp-indexes=solely"

The full listing of labels with values available:

- pytorch-cpu-index: {disabled,solely}
- pytorch-cu111-index: {disabled,solely}
- aicoe-avx2-tf-index: {disabled,solely}
- pypi-index: {disabled,solely}
- opf-pulp-indexes: {disabled,solely}

Labels in Thoth's configuration file

To simplify label handling and to give users the ability to specify persistent labels, Thamos CLI accepts labels section in each runtime environment entry in .thoth.yaml file. Labels stated there are specific to the runtime environment used and can additionally configure how the resolution process should look like for the desired runtime environment. Labels are used to additionally select pipeline units that adjust the resolution process.

Available since thamos>=1.22.0

Python package information stored in each container image produced by AICoE-CI

AICoE-CI now propagates information about packages used in a form of Pipfile, Pipfile.lock stored in /opt/aicoe-ci. This enabled propagating this information in container image analyses done in Thoth's package-extract which enables to introspect what Python packages from which indexes are installed during the application build.

Component Updates

Thanks for the amazing work everyone. 💯
⛄ 🎄 🎅 🎁 ...

Read more
Assets 2
Loading