Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: CBOM support #933

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Conversation

san-zrl
Copy link

@san-zrl san-zrl commented Sep 27, 2024

Description

Enhances DT to read, persist, serve and export CBOM 1.6 data. See additional_details section for more information on what has been changed in particular. Note that there is a corresponding PR for hyades-frontend that enhances the UI to render CBOM data.

Addressed Issue

Issue #1538

Additional Details

  • Added model classes and metric for CBOM data to org.depencytrack.model
  • Enhanced ModelConverter to populate the additional model classes
  • ModelConverter became quite massive so I split it into ModelConverter (cyclonedx -> internal model) and ModelExporter (internal model -> cyclonedx
  • Added org.depencytrack.persistence.v1.CryptoAssetsResource with endpoints that serve the UI
  • Enhanced exporter to generate CBOM data in download BOM
  • CBOM data are components of type Classifier.CRYPTOGRAPHIC_ASSET with CryptoProperties and Occurrence attributes

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have updated the migration changelog accordingly
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@nscuro nscuro added the enhancement New feature or request label Sep 27, 2024
@VinodAnandan
Copy link
Contributor

Hi @san-zrl Thank you for the Pull Request! I believe this will be a great feature. It seems the test pipeline action is currently
failing, would you mind taking a look at it?

https://github.com/DependencyTrack/hyades-apiserver/actions/runs/11067343603/job/30750563096?pr=933

"Error:    BomResourceTest.exportProjectAsCycloneDxInventoryTest:266 JSON documents are different:
Array "dependencies" has different length, expected: <4> but was: <2>.
Array "dependencies" has different content. Missing values: [{"ref":"${json-unit.matches:componentWithVulnUuid}","dependsOn":[]}, {"ref":"${json-unit.matches:componentWithVulnAndAnalysisUuid}","dependsOn":[]}], extra values: [], expected: <[{"ref":"${json-unit.matches:projectUuid}","dependsOn":["${json-unit.matches:componentWithoutVulnUuid}","${json-unit.matches:componentWithVulnAndAnalysisUuid}"]},{"ref":"${json-unit.matches:componentWithoutVulnUuid}","dependsOn":["${json-unit.matches:componentWithVulnUuid}"]},{"ref":"${json-unit.matches:componentWithVulnUuid}","dependsOn":[]},{"ref":"${json-unit.matches:componentWithVulnAndAnalysisUuid}","dependsOn":[]}]> but was: <[{"ref":"a2e6e2b6-9352-49e7-a189-140edfda8b13","dependsOn":["dbce1c1b-e1d1-40d6-8d3e-8bb0b27a3ec1","e119913b-4ced-4582-a1de-f31f49c7d27f"]},{"ref":"dbce1c1b-e1d1-40d6-8d3e-8bb0b27a3ec1","dependsOn":["f8ccf117-7d61-47f6-b417-bb74927184ca"]}]>
Different keys found in node "metadata", missing: "metadata.manufacture", expected: <{"authors":[{"name":"bomAuthor"}],"component":{"author":"SampleAuthor","bom-ref":"${json-unit.matches:projectUuid}","name":"acme-app","supplier":{"name":"projectSupplier"},"type":"application","version":"SNAPSHOT"},"manufacture":{"name":"projectManufacturer"},"supplier":{"name":"bomSupplier"},"timestamp":"${json-unit.any-string}","tools":[{"name":"Dependency-Track","vendor":"OWASP","version":"${json-unit.any-string}"}]}> but was: <{"authors":[{"name":"bomAuthor"}],"component":{"author":"SampleAuthor","bom-ref":"a2e6e2b6-9352-49e7-a189-140edfda8b13","manufacturer":{"name":"projectManufacturer"},"name":"acme-app","supplier":{"name":"projectSupplier"},"type":"application","version":"SNAPSHOT"},"supplier":{"name":"bomSupplier"},"timestamp":"2024-09-27T08:57:00Z","tools":[{"name":"Dependency-Track","vendor":"OWASP","version":"5.6.0-SNAPSHOT"}]}>
Different keys found in node "metadata.component", extra: "metadata.component.manufacturer", expected: <{"author":"SampleAuthor","bom-ref":"${json-unit.matches:projectUuid}","name":"acme-app","supplier":{"name":"projectSupplier"},"type":"application","version":"SNAPSHOT"}> but was: <{"author":"SampleAuthor","bom-ref":"a2e6e2b6-9352-49e7-a189-140edfda8b13","manufacturer":{"name":"projectManufacturer"},"name":"acme-app","supplier":{"name":"projectSupplier"},"type":"application","version":"SNAPSHOT"}>
Different value found in node "specVersion", expected: <"1.5"> but was: <"1.6">.

Error:    BomResourceTest.exportProjectAsCycloneDxInventoryWithVulnerabilitiesTest:505 
expected: 200
 but was: 500
Error:    BomResourceTest.exportProjectAsCycloneDxLicenseTest:391 JSON documents are different:
Array "dependencies" has different length, expected: <2> but was: <0>.
Array "dependencies" has different content. Missing values: [{"ref":"${json-unit.matches:projectUuid}","dependsOn":[]}, {"ref":"${json-unit.matches:component}","dependsOn":[]}], expected: <[{"ref":"${json-unit.matches:projectUuid}","dependsOn":[]},{"ref":"${json-unit.matches:component}","dependsOn":[]}]> but was: <[]>
Different value found in node "specVersion", expected: <"1.5"> but was: <"1.6">.

Error:    BomResourceTest.exportProjectAsCycloneDxVdrTest:700 
expected: 200
 but was: 500
Error:    VexResourceTest.exportProjectAsCycloneDxTest:145 
expected: 200
 but was: 500
Error:    VexResourceTest.exportVexWithDifferentVulnAnalysisValidJsonTest:485 
expected: 200
 but was: 500
Error:    VexResourceTest.exportVexWithSameVulnAnalysisValidJsonTest:387 
expected: 200
 but was: 500
Error:    BomUploadProcessingTaskTest.informWithExistingDuplicateComponentPropertiesAndBomWithDuplicateComponentProperties:1618 
Expecting code not to raise a throwable but caught
  "javax.jdo.JDOObjectNotFoundException: Object with id "org.dependencytrack.model.ComponentProperty:5188" not found !
	at org.datanucleus.api.jdo.JDOAdapter.getJDOExceptionForNucleusException(JDOAdapter.java:637)
	at org.datanucleus.api.jdo.JDOPersistenceManager.jdoRefresh(JDOPersistenceManager.java:495)
	at org.datanucleus.api.jdo.JDOPersistenceManager.refresh(JDOPersistenceManager.java:507)
	at org.dependencytrack.tasks.BomUploadProcessingTaskTest.lambda$informWithExistingDuplicateComponentPropertiesAndBomWithDuplicateComponentProperties$116(BomUploadProcessingTaskTest.java:1618)
	at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
	at org.assertj.core.api.NotThrownAssert.isThrownBy(NotThrownAssert.java:43)
	at org.dependencytrack.tasks.BomUploadProcessingTaskTest.informWithExistingDuplicateComponentPropertiesAndBomWithDuplicateComponentProperties(BomUploadProcessingTaskTest.java:1618)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
NestedThrowablesStackTrace:
Object with id "org.dependencytrack.model.ComponentProperty:5188" not found !
org.datanucleus.exceptions.NucleusObjectNotFoundException: Object with id "org.dependencytrack.model.ComponentProperty:5188" not found !
	at org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:492)
	at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:427)
	at org.datanucleus.state.StateManagerImpl.loadFieldsFromDatastore(StateManagerImpl.java:1632)
	at org.datanucleus.state.StateManagerImpl.refreshFieldsInFetchPlan(StateManagerImpl.java:4034)
	at org.datanucleus.api.jdo.state.Hollow.transitionRefresh(Hollow.java:169)
	at org.datanucleus.state.StateManagerImpl.refresh(StateManagerImpl.java:1031)
	at org.datanucleus.ExecutionContextImpl.refreshObject(ExecutionContextImpl.java:1664)
	at org.datanucleus.api.jdo.JDOPersistenceManager.jdoRefresh(JDOPersistenceManager.java:490)
	at org.datanucleus.api.jdo.JDOPersistenceManager.refresh(JDOPersistenceManager.java:507)
	at org.dependencytrack.tasks.BomUploadProcessingTaskTest.lambda$informWithExistingDuplicateComponentPropertiesAndBomWithDuplicateComponentProperties$116(BomUploadProcessingTaskTest.java:1618)
	at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
	at org.assertj.core.api.NotThrownAssert.isThrownBy(NotThrownAssert.java:43)
	at org.dependencytrack.tasks.BomUploadProcessingTaskTest.informWithExistingDuplicateComponentPropertiesAndBomWithDuplicateComponentProperties(BomUploadProcessingTaskTest.java:1618)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
"
Error:  Errors: 
Error:    ModelConverterTest.testConvertCycloneDX1:91 » NullPointer Cannot invoke "org.cyclonedx.model.component.crypto.enums.AssetType.ordinal()" because the return value of "org.dependencytrack.model.CryptoAssetProperties.getAssetType()" is null
Error:    ModelConverterTest.testGenerateDependencies:203 » JDOUser One or more instances could not be made persistent
Error:    BomUploadProcessingTaskTest.informTest:138->lambda$informTest$8:139 NullPointer Cannot invoke "org.dependencytrack.model.OrganizationalEntity.getName()" because "manufacturer" is null
Error:    BomUploadProcessingTaskTest.informWithExistingComponentPropertiesAndBomWithComponentProperties:1399 » JDOObjectNotFound Object with id "org.dependencytrack.model.Component:17394" not found !
Error:    BomUploadProcessingTaskTest.informWithExistingComponentPropertiesAndBomWithoutComponentProperties:1371 » JDOObjectNotFound Object with id "org.dependencytrack.model.Component:2852" not found !
[INFO] 
Error:  Tests run: 1708, Failures: 8, Errors: 5, Skipped: 2
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  32:34 min
[INFO] Finished at: 2024-09-27T09:10:35Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.5.0:test (default-test) on project dependency-track: There are test failures.
Error:  
Error:  Please refer to /home/runner/work/hyades-apiserver/hyades-apiserver/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Error: Process completed with exit code 1.
"

Signed-off-by: san-zrl <[email protected]>

fix: added CryptoAssetsResource

Signed-off-by: san-zrl <[email protected]>

added getAllCryptoAssets() perr project and globally

Signed-off-by: san-zrl <[email protected]>
Signed-off-by: san-zrl <[email protected]>
@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

Hi @VinodAnandan - I looked into the tests and fixed the main problems that are related to the migration to cyclonedx 1..6. Not sure if the entire test set works because I never managed to run it successfully on my local system. I'm on a Mac M1 and getting testcontainers to run was a challenge. Issues:

  1. There are some tests that rely on the jdo refresh feature to sync java objects with background changes in the database. One such example is informWithExistingComponentPropertiesAndBomWithoutComponentProperties. I couldn't get those to run and commented them out for the time being.
  2. MigrationInitializerTest creates a temporary postgres container as the migration target. Some of the 4 tests in this class always fail. Behavior is pretty random.

@nscuro
Copy link
Member

nscuro commented Oct 1, 2024

Not sure if the entire test set works because I never managed to run it successfully on my local system. I'm on a Mac M1 and getting testcontainers to run was a challenge

Can you elaborate what about the test containers was problematic? The team so far has been working predominantly with M1 macs, so that should not be a problem.

There are some tests that rely on the jdo refresh feature to sync java objects with background changes in the database. [...] I couldn't get those to run and commented them out for the time being.

Can you share the errors you were getting here? Really all the refresh is doing is reloading the object from the database, so it's not much different from doing a SELECT query.

MigrationInitializerTest creates a temporary postgres container as the migration target. Some of the 4 tests in this class always fail. Behavior is pretty random.

Same as above, can you share the errors you're getting?

@nscuro
Copy link
Member

nscuro commented Oct 1, 2024

Also, have you tried if launching the API server with Dev Services works for you? https://dependencytrack.github.io/hyades/0.6.0-SNAPSHOT/development/testing/#api-server

@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

  1. BomUploadProcessingTaskTest#informWithExistingComponentPropertiesAndBomWithComponentProperties (test report: org.dependencytrack.tasks.BomUploadProcessingTaskTest.txt) fails with
javax.jdo.JDOObjectNotFoundException: Object with id "org.dependencytrack.model.Component:33590 not found
  1. Some test in MigrationInitializerTest (test report: org.dependencytrack.persistence.migration.MigrationInitializerTest.txt) always fail with the msg below. Some seem to work even though they use the same mechanism underneath.
Caused by: org.postgresql.util.PSQLException: Connection to localhost:33271 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

I ran above tests with mvn -P enhance verify -Dtest=<class>[#<method>]

@nscuro
Copy link
Member

nscuro commented Oct 1, 2024

I can reproduce the failures of the component property tests in this PR, but not in main. The tests expect a component to be modified, but it gets deleted instead. Which means DT couldn't match it's identity upon the second BOM upload.

The problem seems to be the modified buildExactComponentIdentityQuery method, where you currently have:

        if (cid.getOid() != null) {
            filterParts.add("(cryptoAssetProperties != null && cryptoAssetProperties.oid == :oid)");
            params.put("oid", cid.getOid());
        } else {
            filterParts.add("cryptoAssetProperties != null && cryptoAssetProperties.oid == null");
        }

But it should be this instead:

        if (cid.getOid() != null) {
            filterParts.add("(cryptoAssetProperties != null && cryptoAssetProperties.oid == :oid)");
            params.put("oid", cid.getOid());
        } else {
            filterParts.add("(cryptoAssetProperties == null || cryptoAssetProperties.oid == null)");
        }

The MigrationInitializerTest successfully completes for me. They also don't fail in CI. I wonder if perhaps there's something special about your local Docker setup? Are you using Docker or some other derivative such as Podman or Rancher?

@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

@nscuro: Good catch, thanks!

I'm using Rancher.

@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

@nscuro - The buildExactComponentIdentityQuery fix resolved most of the issues related to refresh(). However, there are still 3 testUpdateMetricsUnchanged tests that fail with assert violations on timestamps. Here's an example:
org.dependencytrack.tasks.metrics.ProjectMetricsUpdateTaskTest.txt. Could you have a look into this?

@nscuro
Copy link
Member

nscuro commented Oct 1, 2024

Will have a look at the failing tests.

Regarding Testcontainers, could this be relevant? https://docs.rancherdesktop.io/how-to-guides/using-testcontainers/#prerequisites

Signed-off-by: san-zrl <[email protected]>
@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

Regarding Testcontainers, could this be relevant? https://docs.rancherdesktop.io/how-to-guides/using-> testcontainers/#prerequisites

I've seen this. Rancher uses admin rights. Kubernetes is disabled, VM type is set to QEMU. My env settings are

DOCKER_HOST=unix:///Users/san/.rd/docker.sock
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
TESTCONTAINERS_HOST_OVERRIDE=localhost
TESTCONTAINERS_RYUK_DISABLED=true

@nscuro
Copy link
Member

nscuro commented Oct 1, 2024

TESTCONTAINERS_RYUK_DISABLED=true

Not entirely sure, but maybe Ryuk not being there could be a problem. Can you try enabling it?

Copy link

codacy-production bot commented Oct 1, 2024

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
-1.45% (target: -1.00%) 42.42% (target: 70.00%)
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (b0581ff) 21938 18137 82.67%
Head commit (2daae1b) 22676 (+738) 18419 (+282) 81.23% (-1.45%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#933) 1035 439 42.42%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more

@san-zrl
Copy link
Author

san-zrl commented Oct 1, 2024

TESTCONTAINERS_RYUK_DISABLED=false makes no difference for MigrationInitializerTest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants