Skip to content

Config JSON

Jump to bottom
thoniTUB edited this page Sep 2, 2024 · 17 revisions

Config JSON

This is an automatically created documentation. It is not 100% accurate since the generator does not handle every edge case.

Instead of a list ConQuery also always accepts a single element.

The config.json is required for every type of execution. Its root element is a ConqueryConfig object.

Base AuthorizationConfig

An AuthorizationConfig defines the initial users that are created on application startup and other permission related options.

Different types of AuthorizationConfig can be used by setting type to one of the following values:

DEFAULT ✎

Details

Java Type: com.bakdata.conquery.models.config.auth.DefaultAuthorizationConfig

Supported Fields:

Field Type Default Example Description
✎ initialRoles list of ProtoRole
✎ initialUsers list of ProtoUser []
✎ overviewScope list of String ["datasets","admin","*"]

DEVELOPMENT ✎

Details

Java Type: com.bakdata.conquery.models.config.auth.DevelopmentAuthorizationConfig

No fields can be set for this type.

Base AuthenticationRealmFactory

An AuthenticationConfig is used to define how specific realms for authentication are configured.

Different types of AuthenticationRealmFactory can be used by setting type to one of the following values:

DEVELOPMENT ✎

Default configuration for the auth system. Sets up all other default components. This configuration causes that every request is handled as invoked by the super user.

Details

Java Type: com.bakdata.conquery.models.auth.develop.DevAuthConfig

No fields can be set for this type.

JWT_PKCE_REALM ✎

A realm that verifies oauth tokens using PKCE. Since the adminEnd-UI mainly works with direct links that do not support setting of the Authorization-header to transport an access token and it also does not support the oauth code flow, this realm proxies the flow. 1. The user/client is redirected to the IDP for authentication and redirected back to the adminEnd 2. The this realm then picks up the code the transported authorization code from the query and redeems it for an access token and refresh token. 3. Both are converted to cookies, which are saved on the client upon redirection to the page the user initially wanted to visit. 4. With the redirection the client sends back the cookies from which the adminEnd extracts the access token and verifies it. (5.) If the previous step failed. The refresh token is extracted and exchanged for a fresh access token and refresh token and continues with step 3.

Details

Java Type: com.bakdata.conquery.models.config.auth.JwtPkceVerifyingRealmFactory

Supported Fields:

Field Type Default Example Description
✎ additionalTokenChecks list of String []
✎ alternativeIdClaims list of String [] Which claims hold alternative Ids of the user in case the user name does not match a user. Pay attention, that the user must not be able to alter the value of any of these claims.
✎ client String null The client id is also used as the expected audience in the validated token. Ensure that the IDP is configured accordingly.
✎ idpConfiguration IdpConfiguration null See wellKnownEndpoint.
✎ tokenLeeway @jakarta.validation.constraints.Min(0) int 60 A leeway for token's expiration in seconds, this should be a short time.

One Minute is the default.
✎ wellKnownEndpoint URI null Either the wellKnownEndpoint from which an idpConfiguration can be obtained or the idpConfiguration must be supplied. If the idpConfiguration is given, the wellKnownEndpoint is ignored.

LOCAL_AUTHENTICATION ✎

Details

Java Type: com.bakdata.conquery.models.config.auth.LocalAuthenticationConfig

Supported Fields:

Field Type Default Example Description
✎ directory File "./storage"
✎ jwtDuration @MinDuration(value=1, unit=TimeUnit.MINUTES) Duration "12 hours"
✎ passwordStoreConfig XodusConfig Configuration for the password store. An encryption for the store itself might be set here.
✎ storeName String "authenticationStore" The name of the folder the store lives in.

OIDC_AUTHORIZATION_CODE_FLOW ✎

Factory for a simple realm that just forwards tokens to the IDP for verification.

Details

Java Type: com.bakdata.conquery.models.config.auth.OIDCAuthorizationCodeFlowRealmFactory

Supported Fields:

Field Type Default Example Description
✎ client IntrospectionDelegatingRealmFactory null

OIDC_RESOURCE_OWNER_PASSWORD_CREDENTIAL_AUTHENTICATION ✎

Realm that supports the Open ID Connect Resource-Owner-Password-Credential-Flow with a Keycloak IdP.

Details

Java Type: com.bakdata.conquery.models.config.auth.OIDCResourceOwnerPasswordCredentialRealmFactory

Supported Fields:

Field Type Default Example Description
✎ client IntrospectionDelegatingRealmFactory null

Base PluginConfig

A PluginConfig is used to define settings for Conquery plugins.

Different types of PluginConfig can be used by setting type to one of the following values:

FORM_BACKEND ✎

{@link PluginConfig} for an external form backend. The external form backend must implement the OpenAPI spec for external form backend.

Details

Java Type: com.bakdata.conquery.models.config.FormBackendConfig

Supported Fields:

Field Type Default Example Description
✎ authentication @Valid AuthenticationClientFilterProvider null
✎ baseURI @NotNull URI null
✎ cancelTaskPath String
✎ conqueryApiUrl @NotNull URL null
✎ formConfigPath String "form-config"
✎ healthCheckPath String "health"
✎ id String null
✎ postFormPath String "task"
✎ statusTemplatePath String
✎ versionPath String "version"

MANUAL ✎

Configuration of manual links for {@link org.checkerframework.checker.signature.qual.InternalForm} and overriding of other {@link com.bakdata.conquery.apiv1.forms.Form}s.

Details

Java Type: com.bakdata.conquery.models.config.ManualConfig

Supported Fields:

Field Type Default Example Description
✎ forms map from String to @ManualURI URI Maps a {@link CPSType} of a {@link com.bakdata.conquery.apiv1.forms.Form} to a manual URL. E.g.:

Other Types

Type APIConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.APIConfig

Supported Fields:

Field Type Default Example Description
✎ allowCORSRequests boolean false
✎ caching boolean true

Type CSVConfig ✎

Holds the necessary information to configure CSV parsers and writers.

Details

Java Type: com.bakdata.conquery.models.config.CSVConfig

Supported Fields:

Field Type Default Example Description
✎ comment char "\u0000"
✎ delimeter char ","
✎ encoding @NotNull Charset "UTF-8"
✎ escape char "\\"
✎ lineSeparator String "\n"
✎ maxColumns int 1000000
✎ parseHeaders boolean true
✎ quote char "\""
✎ skipHeader boolean false

Type ClusterConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.ClusterConfig

Supported Fields:

Field Type Default Example Description
✎ backpressure @jakarta.validation.constraints.Min(0) int 1500 Amount of backpressure before jobs can volunteer to block to send messages to their shards.

Mostly {@link com.bakdata.conquery.models.jobs.ImportJob} is interested in this. Note that an average import should create more than #Entities / {@linkplain #entityBucketSize} jobs (via {@link com.bakdata.conquery.models.jobs.CalculateCBlocksJob}) in short succession, which will cause it to sleep. This field helps alleviate memory pressure on the Shards by slowing down the Manager, should it be sending too fast.
✎ connectRetryTimeout Duration "30 seconds"
✎ entityBucketSize @jakarta.validation.constraints.Min(1) int 1000
✎ heartbeatTimeout Duration "1 minute"
✎ idleTimeOut Duration "5 minutes"
✎ managerURL @Valid @NotNull InetAddress "localhost"
✎ mina MinaConfig
✎ port @io.dropwizard.validation.PortRange int 16170

Type ConqueryConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.ConqueryConfig

Supported Fields:

Field Type Default Example Description
✎ api APIConfig
✎ authentication @Valid @NotNull AuthenticationConfig
✎ authenticationRealms list of AuthenticationRealmFactory
✎ authorizationRealms @Valid @NotNull AuthorizationConfig
✎ cluster ClusterConfig
✎ csv CSVConfig
✎ debugMode boolean or null null null means here that we try to deduce from an attached agent
✎ failOnError boolean false
✎ frontend FrontendConfig
✎ idColumns @Valid @NotNull IdColumnConfig
✎ index @NotNull @Valid IndexConfig
✎ jerseyClient @Valid @NotNull JerseyClientConfiguration
✎ locale LocaleConfig
✎ metricsConfig ConqueryMetricsConfig
✎ plugins list of PluginConfig []
✎ preprocessor PreprocessingConfig
✎ queries QueryConfig
✎ resultProviders list of ResultRendererProvider The order of this list determines the order of the generated result urls in a query status.
✎ sqlConnectorConfig SqlConnectorConfig
✎ standalone StandaloneConfig
✎ storage @Valid @NotNull StoreFactory

Type CurrencyConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.FrontendConfig$CurrencyConfig

Supported Fields:

Field Type Default Example Description
✎ decimalScale int 2
✎ decimalSeparator String ","
✎ thousandSeparator String "."
✎ unit String "€"

Type DatabaseConfig ✎

Connection properties for a SQL database.

Currently supported are HANA and Prostgres databases, see {@link DatabaseConfig#dialect}.
Details

Java Type: com.bakdata.conquery.models.config.DatabaseConfig

Supported Fields:

Field Type Default Example Description
✎ databasePassword String null Password used to connect to the database.
✎ databaseUsername String null Username used to connect to the database.
✎ dialect Dialect null SQL vendor specific dialect used to transform queries to SQL
✎ jdbcConnectionUrl String null Connections url in JDBC notation.
✎ primaryColumn String "pid" Name of the column which is shared among the table and all aggregations are grouped by.

Type Dialect ✎

The dialect sets SQL vendor specific query transformation rules.

There is no fallback dialect, so the dialect must fit the targeted database. Values:

  • POSTGRESQL: Dialect for PostgreSQL database

  • HANA: Dialect for SAP HANA database

Details

Java Type: com.bakdata.conquery.models.config.Dialect

No fields can be set for this type.

Type FrontendConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.FrontendConfig

Supported Fields:

Field Type Default Example Description
✎ alwaysAllowCreateValue boolean false If true, users are always allowed to add custom values into SelectFilter input fields.
✎ contactEmail String null
✎ currency CurrencyConfig
✎ manualUrl URL null The url that points a manual. This is also used by the {@link FormScanner} as the base url for forms that specify a relative url. Internally {@link URI#resolve(URI)} is used to concatenate this base url and the manual url from the form. An final slash ('{@code /}') on the base url has the following effect:
✎ observationPeriodYears @jakarta.validation.constraints.Min(0) int 6 Years to include in entity preview.
✎ visualisationPercentiles Range<Integer>
✎ visualisationsHistogramLimit @jakarta.validation.constraints.Min(0) int 10 Limit to number of histogram entries. Note, that zero and out of bounds values are tracked in separate bins, so you can have three additional bins.

Type LocaleConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.LocaleConfig

Supported Fields:

Field Type Default Example Description
✎ dateFormatMapping map from Locale to String Mappings from user provided locale to date format which is used in the generation of result tables. The formats are also available for parsing dates using the {@link DateReader}. However, the locale is neglected there and the formats are tried until one that fits is found.
✎ frontend @NotNull Locale "und"
✎ listFormats list of ListFormat List formats that are available for parsing inputs and (the first one) for rendering result tables. Spaces at the ends of the separator are only relevant for the output of results. For the input (parsing) the separator string can be surrounded by an arbitrary number of spaces.
✎ localeRangeStartEndSeparators map from Locale to String Mappings from user provided locale to date range format which is used in the generation of result tables. The formats are also available for parsing dates ranges using the {@link DateReader}. However, the locale is neglected there and the formats are tried until one that fits is found.
✎ parsingDateFormats @NotNull Set<String> ["yyyyMMdd"] Additional date formats that are available only for parsing.
✎ parsingRangeStartEndSeparators Set<String> ["/"] Additional date range formats that are available only for parsing.

Type MinaConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.MinaConfig

Supported Fields:

Field Type Default Example Description
✎ bothIdleTime int 300 The delay in seconds before we notify a session that it has been idle on read and write. Default to infinite
✎ maxReadBufferSize int 500000000 The maximum size of the buffer used to read incoming data
✎ minReadBufferSize int 64 The minimum size of the buffer used to read incoming data
✎ readBufferSize int 8192 The default size of the buffer used to read incoming data
✎ readerIdleTime int 300 The delay in seconds before we notify a session that it has been idle on read. Default to infinite
✎ throughputCalculationInterval int 3
✎ useReadOperation boolean false A flag set to true when we allow the application to do a session.read(). Default to false
✎ writeTimeout int 0 The delay to wait for a write operation to complete before bailing out
✎ writerIdleTime int 300 The delay in seconds before we notify a session that it has been idle on write. Default to infinite

Type PreprocessingConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.PreprocessingConfig

Supported Fields:

Field Type Default Example Description
✎ faultyLineThreshold @jakarta.validation.constraints.Min(0) @jakarta.validation.constraints.Max(1) double 0.01
✎ maximumPrintedErrors @jakarta.validation.constraints.Min(0) int 10
✎ nThreads @jakarta.validation.constraints.Min(1) int ␀
✎ parsers ParserConfig

Type QueryConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.QueryConfig

Supported Fields:

Field Type Default Example Description
✎ executionPool ThreadPoolDefinition
✎ oldQueriesTime Duration "30 days"
✎ secondaryIdSubPlanRetention int 15 Limits how many subQuery-Plans should be cached between executions: This number limits how many sub-plans are cached per core so that outliers do not cause massive memory overhead. TODO Implement global limit of active secondaryId sub plans

Type SqlConnectorConfig ✎

Configuration for SQL databases to send dataset queries to.

Multiple databases can be configured for different datasets.
Details

Java Type: com.bakdata.conquery.models.config.SqlConnectorConfig

Supported Fields:

Field Type Default Example Description
✎ connectivityCheckTimeout Duration null Timeout duration after which a database connection is considered unhealthy (defaults to connection timeout)
✎ databaseConfigs map from String to DatabaseConfig null Keys must match the name of existing {@link Dataset}s.
✎ enabled boolean false
✎ withPrettyPrinting boolean false Determines if generated SQL should be formatted.

Type StandaloneConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.StandaloneConfig

Supported Fields:

Field Type Default Example Description
✎ numberOfShardNodes int 2

Type XodusConfig ✎

Details

Java Type: com.bakdata.conquery.models.config.XodusConfig

Supported Fields:

Field Type Default Example Description
✎ cipherBasicIV long or null null
✎ cipherId String null
✎ cipherKey String null
✎ envCloseForcedly boolean or null null
✎ envGatherStatistics boolean or null null
✎ envIsReadonly boolean or null null
✎ envMaxParallelReadonlyTxns int or null null
✎ envMaxParallelTxns int or null null
✎ envMonitorTxnsCheckFreq int or null null
✎ envMonitorTxnsTimeout @NotNull @Valid Duration "10 minutes"
✎ envReadonlyEmptyStores boolean or null null
✎ envStoreGetCacheSize int or null null
✎ envTxnDowngradeAfterFlush boolean or null null
✎ envTxnReplayMaxCount int or null null
✎ envTxnReplayTimeout long or null null
✎ fullFileReadonly boolean or null null
✎ gcEnabled boolean or null null
✎ gcFileMinAge int or null null
✎ gcFilesDeletionDelay int or null null
✎ gcFilesInterval int or null null
✎ gcMinUtilization int or null null
✎ gcRenameFiles boolean or null null
✎ gcRunPeriod int or null null
✎ gcStartIn int or null null
✎ gcTransactionAcquireTimeout int or null null
✎ gcTransactionTimeout int or null null
✎ gcUseExclusiveTransaction boolean or null null
✎ gcUtilizationFromFile String null
✎ gcUtilizationFromScratch boolean or null null
✎ logCacheFreePhysicalMemoryThreshold long or null null
✎ logCacheNonBlocking boolean or null null
✎ logCacheOpenFilesCount int or null null
✎ logCachePageSize DataSize null
✎ logCacheShared boolean or null null
✎ logCacheUseNio boolean or null null
✎ logCleanDirectoryExpected boolean or null null
✎ logClearInvalid boolean or null null
✎ logDurableWrite boolean or null null
✎ logFileSize @MaxDataSize(value=1, unit=DataSizeUnit.GIGABYTES) DataSize "400 megabytes"
✎ logLockId String null
✎ logLockTimeout Duration "1 second"
✎ logSyncPeriod long or null null
✎ managementEnabled boolean or null null
✎ managementOperationsRestricted boolean or null null
✎ memoryUsage DataSize null
✎ memoryUsagePercentage int or null null
✎ treeMaxPageSize int or null null

Type XodusStoreFactory ✎

Details

Java Type: com.bakdata.conquery.models.config.XodusStoreFactory

Supported Fields:

Field Type Default Example Description
✎ bufferPerWorker @jakarta.validation.constraints.Min(1) int 20 How many slots of buffering to use before the IO thread is put to sleep.
✎ directory Path "file://./storage"
✎ loadEnvironmentWithMissingStores boolean false If set, an environment will not be loaded if it misses a required store. If not set, the environment is loaded and the application needs to create the store. This is useful if a new version introduces a new store, but will also alter the environment upon reading.
✎ readerWorkers @jakarta.validation.constraints.Min(1) int 4 Number of threads reading from XoduStore.
✎ removeUnreadableFromStore boolean false Flag for the {@link SerializingStore} whether to delete values from the underlying store, that cannot be mapped to an object anymore.
✎ unreadableDataDumpDirectory File null When set, all values that could not be deserialized from the persistent store, are dump into individual files.
✎ useWeakDictionaryCaching boolean false
✎ validateOnWrite boolean false
✎ weakCacheDuration @NotNull Duration "48 hours"
✎ xodus XodusConfig
Clone this wiki locally