Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker-Compose fresh install minor issue #1320

Open
2 tasks done
mikesindieiev opened this issue Jun 11, 2024 · 7 comments · Fixed by DependencyTrack/hyades-apiserver#711
Open
2 tasks done

Docker-Compose fresh install minor issue #1320

mikesindieiev opened this issue Jun 11, 2024 · 7 comments · Fixed by DependencyTrack/hyades-apiserver#711
Assignees
Labels
component/api-server defect Something isn't working p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/S Small effort

Comments

@mikesindieiev
Copy link

mikesindieiev commented Jun 11, 2024

Current Behavior

I am running hyades via docker-compose as described in the quick start guide:
docker-compose --profile demo up -d --pull always

By default it uses arm64 images which match to my mac archetecture. I did not check if this issue exists on x86_64 images.

On the first run frontend returns 503 http error during login

Screenshot 2024-06-11 at 10 53 40

But once I execute docker-compose --profile demo down and docker-compose --profile demo up -d --pull always everything works fine

Additionally, Postgres healthcheck does not work properly on arm64

It continuously echoes FATAL: role "root" does not exist. I did not check if this issue exists on x86_64 images as well.

image

Steps to Reproduce

Make a fresh installation of hyades:

  1. docker-compose --profile demo down
  2. Remove hyades-related docker volumes
  3. docker-compose --profile demo up -d --pull always
  4. Navigate to http://localhost:8081

Expected Behavior

Login should work on the first run

Hyades Version

Versions are hardcoded in the docker-compose file - https://github.com/DependencyTrack/hyades/blob/601d221dfd7e277007b75b70d12c9d1e9ea948f8/docker-compose.yml

Repository Type

Hyades services

Browser

Google Chrome

Checklist

@nscuro
Copy link
Member

nscuro commented Jun 11, 2024

On the first run frontend returns 503 http error during login

Are all containers up and healthy when this happens? There will be a slight delay at first, since the API server depends on multiple containers being healthy:

hyades/docker-compose.yml

Lines 127 to 133 in 224540c

depends_on:
postgres:
condition: service_healthy
redpanda:
condition: service_healthy
secret-init:
condition: service_completed_successfully

And those containers have healthchecks with 5s intervals:

hyades/docker-compose.yml

Lines 174 to 178 in 224540c

healthcheck:
test: [ "CMD-SHELL", "pg_isready", "-d", "dtrack" ]
interval: 5s
timeout: 3s
retries: 3

hyades/docker-compose.yml

Lines 208 to 212 in 224540c

healthcheck:
test: [ "CMD-SHELL", "rpk", "cluster", "health", "--exit-when-healthy" ]
interval: 5s
timeout: 3s
retries: 3

It continuously echoes FATAL: role "root" does not exist.

Good catch, I raised #1321 to fix it.

@mikesindieiev
Copy link
Author

mikesindieiev commented Jun 11, 2024

Yes, all containers are healthy and up.

I have waited 5 minutes after starting the containers to check the issue again. It still there

@nscuro
Copy link
Member

nscuro commented Jun 11, 2024

Ah I think I found the issue.

When you look at the API server logs, is it stuck at this line?

INFO [ProcessorInitializer] Initializing processors

I think we need to tweak how the API server is waiting for topics to be present.

@nscuro nscuro added p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/S Small effort component/api-server and removed in triage labels Jun 11, 2024
@nscuro nscuro self-assigned this Jun 11, 2024
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Jun 11, 2024
Prevents the API server from getting stuck while waiting for Kafka API responses.

Fixes DependencyTrack/hyades#1320

Signed-off-by: nscuro <[email protected]>
@mikesindieiev
Copy link
Author

Ah I think I found the issue.

When you look at the API server logs, is it stuck at this line?

INFO [ProcessorInitializer] Initializing processors

I think we need to tweak how the API server is waiting for topics to be present.

Here is the log output

2024-06-11 11:36:25 2024-06-11 09:36:25,949 INFO [EmbeddedJettyServer] alpine-executable-war v2.2.5 (c3a1a709-acdc-4ca7-96dd-5eaab858ee32) built on: 2024-02-29T20:30:01Z
2024-06-11 11:36:26 2024-06-11 09:36:26,971 INFO [Config] --------------------------------------------------------------------------------
2024-06-11 11:36:26 2024-06-11 09:36:26,972 INFO [Config] OS Name:      Linux
2024-06-11 11:36:26 2024-06-11 09:36:26,972 INFO [Config] OS Version:   6.6.26-linuxkit
2024-06-11 11:36:26 2024-06-11 09:36:26,972 INFO [Config] OS Arch:      aarch64
2024-06-11 11:36:26 2024-06-11 09:36:26,972 INFO [Config] CPU Cores:    12
2024-06-11 11:36:26 2024-06-11 09:36:26,973 INFO [Config] Max Memory:   2.0 GB (2,147,483,648.0 bytes)
2024-06-11 11:36:26 2024-06-11 09:36:26,973 INFO [Config] Java Vendor:  Eclipse Adoptium
2024-06-11 11:36:26 2024-06-11 09:36:26,973 INFO [Config] Java Version: 21.0.3+9-LTS
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] Java Home:    /opt/java/openjdk
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] Java Temp:    /tmp
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] User:         dtrack
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] User Home:    /data/
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] --------------------------------------------------------------------------------
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] Initializing Configuration
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] System property alpine.application.properties not specified
2024-06-11 11:36:26 2024-06-11 09:36:26,974 INFO [Config] Loading application.properties from classpath
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] --------------------------------------------------------------------------------
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Application:  Dependency-Track
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Version:      5.5.0-SNAPSHOT
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Built-on:     2024-06-10T16:53:18Z
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] --------------------------------------------------------------------------------
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Framework:    Alpine
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Version :     2.2.5
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] Built-on:     2024-02-29T20:30:01Z
2024-06-11 11:36:26 2024-06-11 09:36:26,977 INFO [Config] --------------------------------------------------------------------------------
2024-06-11 11:36:27 2024-06-11 09:36:27,101 INFO [MetricsInitializer] Registering system metrics
2024-06-11 11:36:27 2024-06-11 09:36:27,122 INFO [MigrationInitializer] Running migrations
2024-06-11 11:36:27 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-1::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-2::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-3::VithikaS
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-4::VithikaS
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-5::VithikaS
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-6::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-7::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-8::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-9::VithikaS
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-10::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-11::[email protected]
2024-06-11 11:36:28 2024-06-11 09:36:28,512 WARN [executor] constraint "COMPONENT_CLASSIFIER_check" of relation "COMPONENT" does not exist, skipping
2024-06-11 11:36:28 2024-06-11 09:36:28,512 WARN [executor] constraint "INTEGRITY_META_COMPONENT_STATUS_check" of relation "INTEGRITY_META_COMPONENT" does not exist, skipping
2024-06-11 11:36:28 2024-06-11 09:36:28,513 WARN [executor] constraint "PROJECT_CLASSIFIER_check" of relation "PROJECT" does not exist, skipping
2024-06-11 11:36:28 2024-06-11 09:36:28,513 WARN [executor] constraint "WORKFLOW_STATE_STEP_check" of relation "WORKFLOW_STATE" does not exist, skipping
2024-06-11 11:36:28 2024-06-11 09:36:28,514 WARN [executor] constraint "WORKFLOW_STATE_STATUS_check" of relation "WORKFLOW_STATE" does not exist, skipping
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-12::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-13::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-14::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.3.0.xml::v5.3.0-15::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.4.0.xml::v5.4.0-1::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.4.0.xml::v5.4.0-2::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-1::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-2::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-3::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-4::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-5::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-6::nscuro
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-7::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-8::nscuro
2024-06-11 11:36:28 Running Changeset: migration/changelog-v5.5.0.xml::v5.5.0-9::sahibamittal
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::function_cvssv2-to-severity::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::function_cvssv3-to-severity::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::function_calc-severity::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::function_calc-risk-score::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::procedure_update-component-metrics::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::procedure_update-project-metrics::[email protected]
2024-06-11 11:36:28 Running Changeset: migration/changelog-procedures.xml::procedure_update-portfolio-metrics::[email protected]
2024-06-11 11:36:28 
2024-06-11 11:36:28 UPDATE SUMMARY
2024-06-11 11:36:28 Run:                         33
2024-06-11 11:36:28 Previously run:               0
2024-06-11 11:36:28 Filtered out:                 0
2024-06-11 11:36:28 -------------------------------
2024-06-11 11:36:28 Total change sets:           33
2024-06-11 11:36:28 
2024-06-11 11:36:28 Liquibase: Update has been successful. Rows affected: 33
2024-06-11 11:36:28 2024-06-11 09:36:28,652 INFO [PersistenceManagerFactory] Initializing persistence framework
2024-06-11 11:36:28 2024-06-11 09:36:28,653 INFO [PersistenceManagerFactory] Creating transactional connection pool
2024-06-11 11:36:28 2024-06-11 09:36:28,666 INFO [PersistenceManagerFactory] Creating non-transactional connection pool
2024-06-11 11:36:28 2024-06-11 09:36:28,949 WARN [MetaData] Class "org.dependencytrack.model.VulnerabilityPolicy" field "conditions" is an array of (non-serialised) elements of type "java.lang.String" yet no <join> has been specified. You must have a join table to store an array of non-persistable elements, or serialise the array.
2024-06-11 11:36:29 2024-06-11 09:36:29,023 INFO [PersistenceManagerFactory] Registering DataNucleus metrics
2024-06-11 11:36:29 2024-06-11 09:36:29,024 INFO [HealthCheckInitializer] Registering health checks
2024-06-11 11:36:29 2024-06-11 09:36:29,025 INFO [HealthCheckInitializer] Registering extra process metrics
2024-06-11 11:36:29 2024-06-11 09:36:29,027 INFO [DefaultObjectGenerator] Initializing default object generator
2024-06-11 11:36:29 2024-06-11 09:36:29,037 INFO [DefaultObjectGenerator] Synchronizing permissions to datastore
2024-06-11 11:36:29 2024-06-11 09:36:29,172 INFO [DefaultObjectGenerator] Adding default users and teams to datastore
2024-06-11 11:36:30 2024-06-11 09:36:30,284 INFO [DefaultObjectGenerator] Synchronizing SPDX license definitions to datastore
2024-06-11 11:36:31 2024-06-11 09:36:31,701 INFO [DefaultObjectGenerator] Adding default license group definitions to datastore
2024-06-11 11:36:31 2024-06-11 09:36:31,782 INFO [DefaultObjectGenerator] Synchronizing default repositories to datastore
2024-06-11 11:36:31 2024-06-11 09:36:31,806 INFO [DefaultObjectGenerator] Synchronizing config properties to datastore
2024-06-11 11:36:31 2024-06-11 09:36:31,852 INFO [DefaultObjectGenerator] Synchronizing notification publishers to datastore
2024-06-11 11:36:31 2024-06-11 09:36:31,871 INFO [KafkaProducerInitializer] Initializing Kafka producer
2024-06-11 11:36:31 2024-06-11 09:36:31,909 INFO [KafkaProducerInitializer] Registering Kafka producer metrics
2024-06-11 11:36:31 2024-06-11 09:36:31,916 INFO [EventSubsystemInitializer] Initializing asynchronous event subsystem
2024-06-11 11:36:31 2024-06-11 09:36:31,925 INFO [ProcessorInitializer] Initializing processors
2024-06-11 11:39:31 2024-06-11 09:39:31,969 INFO [VulnerabilityMetricsUpdateTask] Executing metrics update on vulnerability database
2024-06-11 11:39:32 2024-06-11 09:39:32,025 INFO [VulnerabilityMetricsUpdateTask] Completed metrics update on vulnerability database in PT0.055682709S
2024-06-11 11:44:31 2024-06-11 09:44:31,991 INFO [WorkflowStateCleanupTask] No workflow steps to transition to TIMED_OUT state
2024-06-11 11:44:31 2024-06-11 09:44:31,992 INFO [WorkflowStateCleanupTask] No TIMED_OUT workflow steps to transition to FAILED status
2024-06-11 11:44:32 2024-06-11 09:44:32,049 INFO [WorkflowStateCleanupTask] No workflows to delete for retention cutoff 2024-06-08T09:44:31
2024-06-11 11:59:32 2024-06-11 09:59:32,038 INFO [WorkflowStateCleanupTask] No workflow steps to transition to TIMED_OUT state
2024-06-11 11:59:32 2024-06-11 09:59:32,040 INFO [WorkflowStateCleanupTask] No TIMED_OUT workflow steps to transition to FAILED status
2024-06-11 11:59:32 2024-06-11 09:59:32,042 INFO [WorkflowStateCleanupTask] No workflows to delete for retention cutoff 2024-06-08T09:59:32
2024-06-11 12:09:32 2024-06-11 10:09:32,071 INFO [PortfolioMetricsUpdateTask] Executing portfolio metrics update
2024-06-11 12:09:32 2024-06-11 10:09:32,072 INFO [PortfolioMetricsUpdateTask] Refreshing project metrics
2024-06-11 12:09:32 2024-06-11 10:09:32,101 INFO [PortfolioMetricsUpdateTask] Completed portfolio metrics update in PT0.029659083S
2024-06-11 12:14:32 2024-06-11 10:14:32,084 INFO [WorkflowStateCleanupTask] No workflow steps to transition to TIMED_OUT state
2024-06-11 12:14:32 2024-06-11 10:14:32,086 INFO [WorkflowStateCleanupTask] No TIMED_OUT workflow steps to transition to FAILED status
2024-06-11 12:14:32 2024-06-11 10:14:32,089 INFO [WorkflowStateCleanupTask] No workflows to delete for retention cutoff 2024-06-08T10:14:32

hyades_apiserver.log

@nscuro
Copy link
Member

nscuro commented Jun 11, 2024

Thanks for the logs. Looks like that's the issue I meant.

There should be further logs from ProcessorInitializer past the Initializing processors log, but there are none. DependencyTrack/hyades-apiserver#711 should resolve this.

@nscuro
Copy link
Member

nscuro commented Jun 11, 2024

Reopening, that didn't help. For some reason the request to describe topics of the cluster gets stuck, despite various timeouts being configured:

2024-06-11 13:12:38 2024-06-11 11:12:38,417 DEBUG [KafkaAdminClient] [AdminClient clientId=3cdb0ea7-e509-48bb-96a4-3cb3d218655f-admin-client] Queueing Call(callName=describeTopics, deadlineMs=1718104361417, tries=0, nextAllowedTryMs=0) with a timeout 3000 ms from now.
2024-06-11 13:12:38 2024-06-11 11:12:38,420 DEBUG [KafkaAdminClient] [AdminClient clientId=3cdb0ea7-e509-48bb-96a4-3cb3d218655f-admin-client] Sending MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='dtrack.vulnerability')], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to dt-redpanda:29092 (id: 0 rack: null). correlationId=3, timeoutMs=2998

Might be a bug in Kafka's AdminClient, need to investigate.

@nscuro nscuro reopened this Jun 11, 2024
@nscuro
Copy link
Member

nscuro commented Jun 11, 2024

I can only reproduce this on an arm64 MacBook, but not on a similarly specced amd64 Linux machine. Really odd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/api-server defect Something isn't working p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/S Small effort
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants