Correctness of identified components

[maaretp]

The tool has a lot of checks for various kinds of things, but it looks like it is missing one which comes across very quickly when you compare sbom-generating tools: not all tools identify all the components. It looks to me like there is quite substantial differences, and I am sure you have already run into this.

Scoring different tools outputs against expected components isn't a straightforward check to implement, so I was mostly curious on your take of caring (or not) for this aspect.

[riteshnoronha]

Currently, sbomqs is designed to check an SBOM for compliance with CycloneDX, SPDX, NTIA, and BSI specifications. While it does not verify the accuracy of the data, it ensures adherence to industry standards, such as validating the format of CPEs, although the CPE itself could be non-existent.

We have developed a website, https://sbombenchmark.dev/, to assist in evaluating open source SBOM generators by comparing their outputs for the same repository. This site provides a rough estimate of the quality of the SBOM generated by each tool.

Ensuring accurate dependency data in your SBOM is a challenging problem to solve. One potential approach is to prepare golden-master lists of components and versions for repositories. These lists can be used by sbomqs during scoring to validate how many components were accurately identified.

Golden master files need not be a complete authoritative set of components and versions belonging to a project. There could be multiple such files e.g one could have only direct dependencies, one could have transitive of a single component. etc. These could then be used to rank OSS SBOM generators based on certain criteria's that would be useful.

We welcome any suggestions for better approaches and are open to discussions on this topic.

@surendrapathak any more ideas that we discussed.

[maaretp]

Thanks, I was aware of this. For a few days I went into the comparison and upstream approach, found your benchmark site and noted it had a possibility to show extra comparison (tool vs. tool) but the trouble would still be that there's both false positives and negatives. Coming to this from a perspective of a user company, the question becomes that being aware of quality increases liability in cases where you chose the poorer option. Nice work with the tool, very much appreciated.

[surendrapathak]

@maaretp Thanks for calling this out. A "completeness" check remains an open question - specifically because as Ritesh called out "completeness" can depend on the context.

From a useability point of view, would that be useful if sbomqs had a mode to compare a 'component-list' to an SBOM component-list? How would you go about generating such list?

sbomqs compare sbom.cdx.json components-list.csv

sbomqs compare sbom.cdx.json complete-sbom.cdx.json

sbomqs compare --direct-dep sbom.cdx.json complete-sbom-direct-dep.cdx.json