Slow ingestion of large SBOMs

[dejanb]

I was trying locally to ingest some of the larger sboms and noticed a couple of things we need to take a look at:

1. Current performance tests don't cover the whole process of sbom ingestion. For example there are no coverage of analysis graph during ingestion which seems to have a big impact on the performance. @JimFuller-RedHat

The test shown here
6a29f8f

with and without commented out graph show the difference

without graph: INFO fundamental::sbom::spdx::perf: ingest: 1m 15s 769ms 403us 459ns
with graph: INFO fundamental::sbom::spdx::perf: ingest: 13m 7s 669ms 187us 167ns  

2. The current examples of large files don't work without fixing package relationships. Is this something we need to consider for the regular code paths as well? I'm currently using the latest openshift and rhel sboms, which work fine. @ctron

[JimFuller-RedHat]

ya, there is optimisations we can do for ingestion - I would rather do that after we do any further changes to analysis graph (which is solidifying fast).

[ctron]

I'd assume that one bigger improvement we could do is the way the graph is populated. Right now, it's reading nodes by manifesting relationships. Meaning, a lot of joins. We could however split this up into reading nodes and relationships independently. Doing the join in-memory.