Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSAF ingestion performance #1037

Open
ctron opened this issue Nov 22, 2024 · 2 comments
Open

CSAF ingestion performance #1037

ctron opened this issue Nov 22, 2024 · 2 comments
Labels
DB Database aspects. Cross cutting concerns. Ownership of the entire data model. Ensuring it's design w performance

Comments

@ctron
Copy link
Contributor

ctron commented Nov 22, 2024

I think that the CSAF ingestion performance is problematic:

image

@ctron ctron added performance DB Database aspects. Cross cutting concerns. Ownership of the entire data model. Ensuring it's design w labels Nov 22, 2024
@ctron ctron added this to Trustify Nov 22, 2024
@ctron ctron moved this to Ready in Trustify Nov 22, 2024
@helio-frota
Copy link
Collaborator

I think we can't discard the latency involved in the process

with this script to fake internet connection for tests , the Time remaining: 3 hours jumped to Time remaining: 5 hours

➜  bin cat f_my_internet
#!/bin/bash
sudo tc qdisc add dev **the-interface-here** root tbf rate 1000bps burst 50kbit latency 300ms

@helio-frota
Copy link
Collaborator

I found a warning on csaf-walker

  --> common/src/fetcher/mod.rs:75:5
   |
75 | /     pub async fn new(options: FetcherOptions) -> anyhow::Result<Self> {
76 | |         let client = ClientBuilder::new().timeout(options.timeout);
77 | |
78 | |         Ok(Self::with_client(client.build()?, options))
79 | |     }
   | |_____^
   |
   = help: consider removing the `async` from this function
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unused_async
   = note: requested on the command line with `-W clippy::unused-async`

warning: unused `async` for function with no await statements
  --> common/src/fetcher/mod.rs:89:5
   |
89 | /     async fn new_request(
90 | |         &self,
91 | |         method: Method,
92 | |         url: Url,
93 | |     ) -> Result<reqwest::RequestBuilder, reqwest::Error> {
94 | |         Ok(self.client.request(method, url))
95 | |     }
   | |_____^

This is being used in the app:

trustify git:(main) rg "Fetcher\:\:new"
modules/importer/src/runner/sbom/mod.rs
62:            Fetcher::new(FetcherOptions::new().retries(fetch_retries.unwrap_or_default())).await?,

modules/importer/src/runner/csaf/mod.rs
57:            Fetcher::new(FetcherOptions::new().retries(fetch_retries.unwrap_or_default())).await?;

Although I don't think the new function is causing problems as apparently it is being called just one time in the app
inside of this functioin: pub async fn run_once_csaf(

  • I'm not saying csaf-walker is causing problems
  • I'm investigating... because apparently tokio is going wild and I'm looking for sync functions in different places...

2024-12-11_12-35

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DB Database aspects. Cross cutting concerns. Ownership of the entire data model. Ensuring it's design w performance
Projects
Status: Ready
Development

No branches or pull requests

2 participants