Jfrog Platform: Artifactory Failing to determine database type on startup

[Souheil-Yazji]

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:
Helm: 3.7.1
k8s 1.25 (AKS)

Which chart:
Jfrog-platform: 10.15.1

Which product license (Enterprise/Pro/oss): Enterprise

JFrog support reference (if already raised with support team): - no support issue raised.

What happened:
On start up, we see the following:
2023-09-26T04:58:09.780Z [shell] [INFO ] [] [systemYamlHelper.sh:607 ] [main] - Resolved .shared.database.type (postgresql) from /opt/jfrog/artifactory/var/etc/system.yaml

then while installerCommon.sh is running:

.federation key is misplaced or doesnt apply at this location
yaml validation failed
2023-09-26T04:58:10.688Z [shell] [WARN ] [] [installerCommon.sh:805        ] [main] - System.yaml validation failed

Database connection check failed Could not determine database type

Not sure if these two failures are related, but the database.type seems to be successfully parsed above AND the connection from the mission control service succeeds right after.

2023-09-26T04:58:21.319Z [jfmc ] [INFO ] [ ] [.MissionControlApplication:158] [Catalina-utility-3 ] - Connection to database successful

What you expected to happen:
No failure
RT successfully connects to the external pgdb

How to reproduce it (as minimally and precisely as possible):
Create a helm release with all defaults but add to the chart values
If using the jfrog-platform chart::

  database:
    initDBCreation: false
    host: <your external pgdb fqdn>
    sslMode: require
    secrets:
      adminUsername:
        name: "admin"
        key: "admin-user"
      adminPassword:
        name: "admin-password"
        key: "admin-password"

Anything else we need to know:

[Logeshwarsn]

@Souheil-Yazji Could you please share the complete values.yaml file ?

[gjlam95]

@Souheil-Yazji Could you please share the complete values.yaml file ?

Hi, I'm facing the same issue as well with this helm chart deployment

values,yaml

postgresql:
  enabled: false
...
database:
  type: postgresql
  driver: org.postgresql.Driver
  url: "${artifactory_database_url}"
  user: "${artifactory_database_username}"
  password: "${artifactory_database_password}"

[Souheil-Yazji]

@Souheil-Yazji Could you please share the complete values.yaml file ?

Hi, I'm facing the same issue as well with this helm chart deployment

values,yaml

postgresql:
  enabled: false
...
database:
  type: postgresql
  driver: org.postgresql.Driver
  url: "${artifactory_database_url}"
  user: "${artifactory_database_username}"
  password: "${artifactory_database_password}"

+1 to this
Using an external db provides the same results.

[johnjelinek]

I'm facing the same issue with mysql 8 in after upgrading from 107.71.16 to 107.77.3. Then all the logs say unable to find join.key and artifactory doesn't boot:

2024-01-25T11:47:14.390Z [jfac ] [ERROR] [a2d28409b6c5518e] [.s.d.u.AccessJdbcHelperImpl:73] [Catalina-utility-2  ] - Could not initialize database: 
org.flywaydb.core.api.FlywayException: Detected failed migration to version 7.82.0.1 (PermissionsV2DataMigration)
	at org.flywaydb.core.internal.info.MigrationInfoImpl.validate(MigrationInfoImpl.java:218)
	at org.flywaydb.core.internal.info.MigrationInfoServiceImpl.validate(MigrationInfoServiceImpl.java:327)
	at org.flywaydb.core.internal.command.DbValidate$2.call(DbValidate.java:174)
	at org.flywaydb.core.internal.command.DbValidate$2.call(DbValidate.java:164)
	at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:75)
	at org.flywaydb.core.internal.command.DbValidate.validate(DbValidate.java:164)
	at org.flywaydb.core.Flyway.doValidate(Flyway.java:1017)
	at org.flywaydb.core.Flyway.access$100(Flyway.java:72)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:934)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:930)
	at org.flywaydb.core.Flyway.execute(Flyway.java:1413)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:930)
	at org.jfrog.storage.util.migration.MigrationUtil.initSingleTenantSchema(MigrationUtil.java:159)
	at org.jfrog.storage.util.migration.MigrationUtil.initSchemas(MigrationUtil.java:98)
	at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.init(AccessJdbcHelperImpl.java:71)
	at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.<init>(AccessJdbcHelperImpl.java:56)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

[PerGon]

I don't think this is an issue with the helm chart - I believe it's an issue with the Artifactory version.

Was running Artifactory version 7.63.12 and tried to upgrade to 7.77.5 - got into the exact same issue as described above.

Then I tried to run Artifactory version 7.71.16 and it worked. So I'm afraid the issue is with the Artifactory application and not the helm chart.

My helm chart version is 107.63.12

[gitta-jfrog]

Indeed it's an issue with Artifactory application 7.77 - More details and resolution can be found here.

[bewinsnw]

@gitta-jfrog we're seeing the exact same issue on 7.84.20 after upgrading from 7.84.11 and the link you posted above says absolutely nothing about the details and resolution. I even checked in archive.org - unless that info was removed before April 12 there's nothing about database type errors on that page.

[gitta-jfrog]

Hi @bewinsnw

The link I shared is related to the issue with instances running on MySQL database as described "Detected failed migration to version 7.82.0.1 (PermissionsV2DataMigration)". It is not related to "Database connection check failed Could not determine database type" which might cause for several reasons, not necessarily failing Artifactory from start.

If you are having support as part of your subscription - please open support ticket and our support engineers will work with you up to full resolution.

if you are not having support - I'll ask you to open a new Github issue with all the details (version before, version after upgrade, database type and values yaml) and I'll try to assist as much as I can.

Thanks

[KlimDos]

@bewinsnw We're enterprise customers and are experiencing the same issue. I've submitted a support request and will keep you updated on what JFrog says.

@gitta-jfrog it might be a good idea to reopen this issue. Is your solution just to wait?

[KlimDos]

DELETE FROM <dbname>.access_schema_version WHERE version = '7.82.0.1';
SET GLOBAL log_bin_trust_function_creators = 1;

Restoring and restarting the migration helped resolve the issue. This solution is actually described in the link posted by @gitta-jfrog.

I'm hosting on AWS RDS MySQL, and even my super admin didn't have the SYSTEM_VARIABLES_ADMIN privilege enabled.

[artm]

is it possible to enable more logging with a helm chart based installation to see what exactly goes wrong?

[lowflying]

We have run in to remarkably similar, but not identical, issues with artifactory v7.90.10 (plus a few other minor versions). While we are still grappling with finding an RCA one commonality we noted was the presence of dynatrace operators running on the same node. Anyone else have monitoring workloads on the same node? No idea how this would cause the symptoms described here but it's the only common denominator we can find so far.