Day Zero bulk data-loading

[bobmcwhirter]

This also may apply for dev-cycle...

When we examine stuff like "ingest all CVEs" or "ingest all RHSAs" etc, those are not necessarily quick.

I'm wondering if it'd be worthwhile to have CI (or some pipeline) perform a task of doing those ingestions, and squeezing out a bulk-loadable .sql file matching our schema.

Then from zero-to-basically-full-system could be much quicker than waiting for every importer to execute.

Once up and running, a system would only need to perform the marginal change-loading from that initial .sql checkpoint.

Not a task for this week, but wanted to open a discussion.

[ctron]

There are already some instructions for this: https://github.com/trustification/trustify/blob/main/importer/README.md

That should do the trick, and can be "cached" to speed things up when re-executing this.

I don't think this is suitable however for production deployments. For that I would hope we can have an API, to get something like this:

http POST localhost:8080/api/sources/csaf/redhat url=https://www.redhat.com schedule=daily

[bobmcwhirter]

Yep, that works also. Just thinking an .sql load would be faster than deserializing a ton of individual .jsons.

[ctron]

True. But that would also mean we would need to create such data dumps. And be opinionated about the data we import.

Maybe can do that if we can't get out any more performance on the ingestion side, and it's still too slow.

[bobmcwhirter]

Yah, I'm only thinking in terms of things where we solve for exactly one opinion (e.g., how we handle CVE vulnerabilities) and if we all end up at the same SQL state after ingesting. Maybe even an optional .sql per $thing, so folks can pick and choose their initial start state.

[bobmcwhirter]

(And I acknowledge this whole idea might be stupid)