Resources
get /vdi-datasets
List Datasets
Returns a list of datasets available to the requesting user, optionally filtered by query parameters.
Results are sorted by creation date in reverse order. This means the most recently created datasets will be first and the oldest dataset will be last in the list.
Parameters chevron_right expand_more
| Parameter | Type | Description |
|---|---|---|
| Query | ||
| project_id Project ID | string | ID of the VEuPathDB project that results should be filtered to. This means only datasets that are relevant to the project ID given will be returned. Additionally, this controls the sites on which the dataset installation status will be checked. Meaning, if this parameter is specified and set to, for example, Inherits: string |
| ownership Dataset Ownership Filter | string | Ownership status filter. Enum of:
If set to If set to If set to Default value: "any" |
curl -X GET \
+ Resources
get /vdi-datasets
List Datasets Returns a list of datasets available to the requesting user, optionally filtered by query parameters.
Results are sorted by creation date in reverse order. This means the most recently created datasets will be first and the oldest dataset will be last in the list.
Parameters chevron_right expand_more
Parameter Type Description Query project_id Project ID string ID of the VEuPathDB project that results should be filtered to.
This means only datasets that are relevant to the project ID given will be returned.
Additionally, this controls the sites on which the dataset installation status will be checked. Meaning, if this parameter is specified and set to, for example, PlasmoDB, the status block in the response objects will only include installation status details for PlasmoDB and not any other sites that the dataset may have been installed into.
Inherits: string
ownership Dataset Ownership Filter string Ownership status filter.
Enum of:
anyownedshared
If set to any the results are not filtered.
If set to owned, the results will be filtered to only results that are owned by the requesting user.
If set to shared, the results will be filtered to only results that are shared with the requesting user.
Default value: "any"
curl -X GET \
undefined/vdi-datasets?ProjectID=<value>&ownership=<value>
200 OK chevron_right expand_more
Success.
This response means that all checks passed and zero or more dataset records were found for the requesting user.
application/json
application/json
Parameter Type Description [] Dataset List Item object Short entry with basic details about a dataset.
Inherits: object
[].datasetId* Dataset ID string Unique VDI Dataset identifier string.
Pattern: ^[a-zA-Z0-9_-]{14}$
Min. length: 14
Max. length: 14
Inherits: lib.VDI-ID
[].owner* Owner Details object Details about the owner of a VDI dataset.
Inherits: lib.DatasetOwner
[].owner.userId* Owner User ID integer VEuPathDB user ID of the owner of the dataset.
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: lib.User-ID
[].owner.firstName Owner First Name string [].owner.lastName Owner Last Name string [].owner.email Owner Email string [].owner.organization Owner Organization string [].datasetType* Dataset Type object Information about a specific dataset type.
Inherits: lib.DatasetTypeInfo
[].datasetType.name* Type Name string [].datasetType.displayName Type Display Name string Display name for the type. This field is ignored in requests and will always be present in responses.
[].datasetType.version* Type Version string [].visibility* Dataset Visibility string Enum:- private
- protected
- public
Inherits: lib.DatasetVisibility
[].name* Dataset Name string User provided name for the dataset.
[].summary Dataset Summary string User provided summary of the dataset.
[].description Dataset Description string User provided description of the dataset.
[].sourceUrl Source URL string URL of the dataset data source, if the dataset was uploaded via URL.
[].origin* Dataset Origin string String representing the origin of the dataset. Examples include direct-upload, nephele, or galaxy.
[].projectIds* Project IDs array Project IDs for projects the dataset record was submitted to.
[].projectIds[] Project ID string Name or ID of a target VEuPathDB project.
Valid project IDs are:
- AmoebaDB
- ClinEpiDB
- CryptoDB
- FungiDB
- GiardiaDB
- HostDB
- MicrobiomeDB
- MicrosporidiaDB
- PiroplasmaDB
- PlasmoDB
- ToxoDB
- TrichDB
- TriTrypDB
- VectorBase
- VEuPathDB
Inherits: lib.ProjectID
[].status* Status Info object Information about the import and install status of a dataset.
Inherits: lib.DatasetStatusInfo
[].status.import* string Import status of the dataset.
Value Description queuedThe dataset has not yet been processed and is waiting in the queue. in-progressThe dataset is currently being import processed. completeThe dataset has been processed and imported for installation. invalidThe dataset failed import validation. failedThe dataset import failed due to an internal server error.
Enum:- queued
- in-progress
- complete
- invalid
- failed
Inherits: lib.DatasetImportStatus
[].status.install array [].status.install[] Dataset Install Status Entry object Entry in a list of install statuses for a dataset.
Inherits: lib.DatasetInstallStatusEntry
[].status.install[].projectId* string Name or ID of a target VEuPathDB project.
Valid project IDs are:
- AmoebaDB
- ClinEpiDB
- CryptoDB
- FungiDB
- GiardiaDB
- HostDB
- MicrobiomeDB
- MicrosporidiaDB
- PiroplasmaDB
- PlasmoDB
- ToxoDB
- TrichDB
- TriTrypDB
- VectorBase
- VEuPathDB
Inherits: lib.ProjectID
[].status.install[].metaStatus string Enum:- running
- complete
- failed-validation
- failed-installation
- ready-for-reinstall
- missing-dependency
Inherits: lib.DatasetInstallStatus
[].status.install[].metaMessage string [].status.install[].dataStatus string Enum:- running
- complete
- failed-validation
- failed-installation
- ready-for-reinstall
- missing-dependency
Inherits: lib.DatasetInstallStatus
[].status.install[].dataMessage string [].shares* Shared With array [].shares[] object Inherits: object
[].shares[].userId* integer Unique user identifier
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: lib.User-ID
[].shares[].firstName* string [].shares[].lastName* string [].shares[].organization* string [].shares[].accepted* boolean [].fileCount* File Count integer Number of files uploaded for this dataset.
[].fileSizeTotal* File Size Total integer Sum of the sizes of all the files uploaded for this dataset.
Format: int64
[].created* Creation Timestamp datetime Timestamp of the creation of this dataset.
Response Body
[
{
"datasetId": "zaZqAAGLGJhBgg",
@@ -1306,11 +1306,16 @@
]
}
}
+}
424 Failed Dependency chevron_right expand_more
Failed Dependency.
Returned when the dataset data source was a URL and the VDI service encountered a non-success HTTP status code from the target URL. This could be, for example, a 403 error from an expired AWS S3 URL, or a 404 for a file that no longer exists on the remote server.
Failed Dependency Error FailedDependencyError
application/json
Parameter Type Description status* string Enum:- bad-request
- unauthorized
- forbidden
- not-found
- bad-method
- conflict
- gone
- invalid-input
- failed-dependency
- server-error
Inherits: lib.ErrorType
message* string dependency* string
Response Body
{
+ "status": "failed-dependency",
+ "dependency": "google.com",
+ "message": "unexpected status code 403 from google.com"
}
500 Internal Server Error chevron_right expand_more
Internal Server Error.
This status is returned when an unhandled or unexpected issue arises when attempting to process the request.
Internal Server Error ServerError
application/json
Parameter Type Description status* string Enum:- bad-request
- unauthorized
- forbidden
- not-found
- bad-method
- conflict
- gone
- invalid-input
- failed-dependency
- server-error
Inherits: lib.ErrorType
message* string requestId* string
Response Body
{
"status": "server-error",
"message": "Dataset store is unreachable",
"requestId": "b296c3d9-4032-41b1-906e-c97ccfc447e3"
-}
post /admin/proxy-upload
Upload a dataset on behalf of another user.
Headers chevron_right expand_more
Parameter Type Description User-ID* VEuPathDB User ID integer ID of the target user on whose behalf a dataset is being uploaded
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: integer
curl -X POST \
+}
post /admin/reconciler
Execute Reconciler
Triggers a full reconciliation run if one is not already in progress.
curl -X POST \
+ undefined/admin/reconciler204 No Content chevron_right expand_more
Success
409 Conflict chevron_right expand_more
Reconciler already running.
post /admin/proxy-upload
Upload a dataset on behalf of another user.
Headers chevron_right expand_more
| Parameter | Type | Description |
|---|---|---|
| User-ID* VEuPathDB User ID | integer | ID of the target user on whose behalf a dataset is being uploaded Min. value: 1 Max. value: 9223372036854776000 Format: int64 Inherits: integer |
curl -X POST \
-H "User-ID: <value>" \
-H "Content-type: multipart/form-data"
-d @file \
diff --git a/lib/app-db/src/main/kotlin/vdi/component/db/app/AppDatabaseRegistry.kt b/lib/app-db/src/main/kotlin/vdi/component/db/app/AppDatabaseRegistry.kt
index ae09769f..42fb595f 100644
--- a/lib/app-db/src/main/kotlin/vdi/component/db/app/AppDatabaseRegistry.kt
+++ b/lib/app-db/src/main/kotlin/vdi/component/db/app/AppDatabaseRegistry.kt
@@ -27,13 +27,14 @@ object AppDatabaseRegistry {
operator fun get(key: String): AppDBRegistryEntry? = dataSources[key]
- operator fun iterator() =
+ operator fun iterator() = asSequence().iterator()
+
+ fun size() = dataSources.size
+
+ fun asSequence() =
dataSources.entries
.asSequence()
.map { (key, value) -> key to value }
- .iterator()
-
- fun size() = dataSources.size
fun require(key: String): AppDBRegistryEntry =
get(key) ?: throw IllegalStateException("required AppDB connection $key was not registered with AppDatabases")
@@ -80,7 +81,7 @@ object AppDatabaseRegistry {
TNS: {}
Pool Size: {}
Port: {}
- DBNamme: {}
+ DBName: {}
User/Schema: {}""",
env.name,
env.name,
diff --git a/lib/env/src/main/kotlin/vdi/component/env/EnvKey.kt b/lib/env/src/main/kotlin/vdi/component/env/EnvKey.kt
index 3c24c1d2..f49de5d5 100644
--- a/lib/env/src/main/kotlin/vdi/component/env/EnvKey.kt
+++ b/lib/env/src/main/kotlin/vdi/component/env/EnvKey.kt
@@ -656,6 +656,16 @@ object EnvKey {
* Required: no
*/
const val SlimRunInterval = "RECONCILER_SLIM_RUN_INTERVAL"
+
+ /**
+ * Whether the reconciler should perform delete operations. If set to
+ * `false`, the reconciler will only log when a delete would have taken
+ * place.
+ *
+ * Type: Boolean
+ * Required: no
+ */
+ const val DeletesEnabled = "RECONCILER_DELETES_ENABLED"
}
object ReconciliationTriggerHandler {
diff --git a/lib/reconciler/build.gradle.kts b/lib/reconciler/build.gradle.kts
new file mode 100644
index 00000000..d19d345a
--- /dev/null
+++ b/lib/reconciler/build.gradle.kts
@@ -0,0 +1,38 @@
+plugins {
+ kotlin("jvm")
+}
+
+dependencies {
+ implementation(platform(project(":platform")))
+
+
+ implementation(project(":lib:app-db"))
+ implementation(project(":lib:cache-db"))
+ implementation(project(":lib:env"))
+ implementation(project(":lib:plugin-client"))
+ implementation(project(":lib:kafka"))
+ implementation(project(":lib:metrics"))
+ implementation(project(":lib:module-core"))
+ implementation(project(":lib:plugin-mapping"))
+ implementation(project(":lib:s3"))
+
+ implementation("org.veupathdb.vdi:vdi-component-common")
+
+ implementation("org.veupathdb.lib.s3:s34k-minio")
+
+ implementation("org.apache.logging.log4j:log4j-api-kotlin")
+
+ implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core")
+
+ testImplementation(kotlin("test"))
+ testImplementation("org.junit.jupiter:junit-jupiter-api:5.9.2")
+ testImplementation("org.mockito:mockito-core:5.2.0")
+ testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:5.9.2")
+ testRuntimeOnly("org.apache.logging.log4j:log4j-slf4j-impl")
+}
+
+tasks.test {
+ useJUnitPlatform()
+
+ testLogging.showStandardStreams = true
+}
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/AppDBTarget.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/AppDBTarget.kt
similarity index 98%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/AppDBTarget.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/AppDBTarget.kt
index 045f6ee8..f03988df 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/AppDBTarget.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/AppDBTarget.kt
@@ -1,4 +1,4 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
import org.veupathdb.vdi.lib.common.model.VDIReconcilerTargetRecord
import org.veupathdb.vdi.lib.common.util.CloseableIterator
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/CacheDBTarget.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/CacheDBTarget.kt
similarity index 97%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/CacheDBTarget.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/CacheDBTarget.kt
index 4e818e2b..3ed389a3 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/CacheDBTarget.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/CacheDBTarget.kt
@@ -1,4 +1,4 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
import org.veupathdb.vdi.lib.common.model.VDIReconcilerTargetRecord
import org.veupathdb.vdi.lib.common.util.CloseableIterator
diff --git a/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/Reconciler.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/Reconciler.kt
new file mode 100644
index 00000000..d519f179
--- /dev/null
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/Reconciler.kt
@@ -0,0 +1,111 @@
+package vdi.lib.reconciler
+
+import kotlinx.coroutines.coroutineScope
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.runBlocking
+import kotlinx.coroutines.sync.Mutex
+import kotlinx.coroutines.sync.withLock
+import org.apache.logging.log4j.kotlin.CoroutineThreadContext
+import org.apache.logging.log4j.kotlin.ThreadContextData
+import org.apache.logging.log4j.kotlin.logger
+import org.veupathdb.lib.s3.s34k.S3Api
+import vdi.component.db.app.AppDatabaseRegistry
+import vdi.component.kafka.router.KafkaRouter
+import vdi.component.kafka.router.KafkaRouterFactory
+import vdi.component.metrics.Metrics
+import vdi.component.modules.AbortCB
+import vdi.component.s3.DatasetManager
+
+object Reconciler {
+ private val logger = logger().delegate
+
+ private val config = ReconcilerConfig()
+
+ private val active = Mutex()
+
+ private var initialized = false
+
+ private lateinit var datasetManager: DatasetManager
+
+ private lateinit var kafkaRouter: KafkaRouter
+
+ fun initialize(abortCB: AbortCB) {
+ // We lock while initializing to avoid double init.
+ if (active.isLocked)
+ return
+
+ // If we've already successfully initialized, nothing to do.
+ if (initialized)
+ return
+
+ // Initialize
+ runBlocking {
+ active.withLock {
+ try {
+ val s3 = S3Api.newClient(config.s3Config)
+ val bucket = s3.buckets[config.s3Bucket]
+ ?: throw IllegalStateException("S3 bucket ${config.s3Bucket} does not exist!")
+
+ datasetManager = DatasetManager(bucket)
+ } catch (e: Throwable) {
+ logger.error("failed to init dataset manager", e)
+ abortCB(e.message)
+ }
+
+ try {
+ kafkaRouter = KafkaRouterFactory(config.kafkaRouterConfig).newKafkaRouter()
+ } catch (e: Throwable) {
+ logger.error("failed to init kafka router", e)
+ abortCB(e.message)
+ }
+ }
+ }
+ }
+
+ suspend fun runFull(): Boolean {
+ if (active.isLocked)
+ return false
+
+ val targets = AppDatabaseRegistry.asSequence()
+ .map { (project, _) -> AppDBTarget(project, project) }
+ .toMutableList()
+ targets.add(CacheDBTarget())
+
+ val timer = Metrics.Reconciler.Full.reconcilerTimes.startTimer()
+
+ logger.info("running full reconciler for ${targets.size} targets")
+
+ coroutineScope {
+ targets.forEach {
+ launch(CoroutineThreadContext(ThreadContextData(mapOf("workerID" to workerName(false, it))))) {
+ ReconcilerInstance(it, datasetManager, kafkaRouter, false, config.deletesEnabled).reconcile()
+ }
+ }
+ }
+
+ timer.observeDuration()
+
+ return true
+ }
+
+ suspend fun runSlim(): Boolean {
+ if (active.isLocked)
+ return false
+
+ val timer = Metrics.Reconciler.Slim.executionTime.startTimer()
+
+ val target = CacheDBTarget()
+
+ coroutineScope {
+ launch(CoroutineThreadContext(ThreadContextData(mapOf("workerID" to workerName(true, target))))) {
+ ReconcilerInstance(target, datasetManager, kafkaRouter, true, false).reconcile()
+ }
+ }
+
+ timer.observeDuration()
+
+ return true
+ }
+
+ private fun workerName(slim: Boolean, tgt: ReconcilerTarget) = if (slim) "slim-${tgt.name}" else "full-${tgt.name}"
+}
\ No newline at end of file
diff --git a/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerConfig.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerConfig.kt
new file mode 100644
index 00000000..6d1b4913
--- /dev/null
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerConfig.kt
@@ -0,0 +1,31 @@
+package vdi.lib.reconciler
+
+import org.veupathdb.lib.s3.s34k.S3Config
+import org.veupathdb.lib.s3.s34k.fields.BucketName
+import org.veupathdb.vdi.lib.common.env.Environment
+import org.veupathdb.vdi.lib.common.env.optBool
+import org.veupathdb.vdi.lib.common.env.require
+import vdi.component.env.EnvKey
+import vdi.component.kafka.router.KafkaRouterConfig
+import vdi.component.s3.util.S3Config
+
+internal data class ReconcilerConfig(
+ val kafkaRouterConfig: KafkaRouterConfig,
+ val s3Config: S3Config,
+ val s3Bucket: BucketName,
+ val deletesEnabled: Boolean,
+) {
+ constructor() : this(System.getenv())
+
+ constructor(env: Environment) : this(
+ kafkaRouterConfig = KafkaRouterConfig(env, "reconciler"),
+ s3Config = S3Config(env),
+ s3Bucket = BucketName(env.require(EnvKey.S3.BucketName)),
+ deletesEnabled = env.optBool(EnvKey.Reconciler.DeletesEnabled) ?: DefaultDeletesEnabled,
+ )
+
+ companion object {
+ const val DefaultDeletesEnabled = true
+ }
+}
+
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerInstance.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerInstance.kt
similarity index 98%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerInstance.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerInstance.kt
index 16b31afe..17f00362 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerInstance.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerInstance.kt
@@ -1,4 +1,4 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
import org.apache.logging.log4j.kotlin.logger
import org.veupathdb.vdi.lib.common.field.DatasetID
@@ -27,7 +27,7 @@ internal class ReconcilerInstance(
private val datasetManager: DatasetManager,
private val kafkaRouter: KafkaRouter,
private val slim: Boolean,
- private val deleteDryMode: Boolean = false
+ private val deletesEnabled: Boolean
) {
private val log = logger().delegate
@@ -35,7 +35,7 @@ internal class ReconcilerInstance(
val name = targetDB.name
- suspend fun reconcile() {
+ internal suspend fun reconcile() {
try {
tryReconcile()
@@ -200,7 +200,7 @@ internal class ReconcilerInstance(
try {
Metrics.Reconciler.Full.reconcilerDatasetDeleted.labels(targetDB.name).inc()
- if (!deleteDryMode) {
+ if (deletesEnabled) {
log.info("trying to delete dataset {}/{}", record.ownerID, record.datasetID)
targetDB.deleteDataset(record)
} else {
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTarget.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTarget.kt
similarity index 96%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTarget.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTarget.kt
index 33c578df..573d72db 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTarget.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTarget.kt
@@ -1,4 +1,4 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
import org.veupathdb.vdi.lib.common.model.VDIReconcilerTargetRecord
import org.veupathdb.vdi.lib.common.util.CloseableIterator
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTargetType.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTargetType.kt
similarity index 66%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTargetType.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTargetType.kt
index e014f239..5fdb2c78 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerTargetType.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/ReconcilerTargetType.kt
@@ -1,3 +1,3 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
internal enum class ReconcilerTargetType { Cache, Install }
\ No newline at end of file
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/UnsupportedTypeException.kt b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/UnsupportedTypeException.kt
similarity index 72%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/UnsupportedTypeException.kt
rename to lib/reconciler/src/main/kotlin/vdi/lib/reconciler/UnsupportedTypeException.kt
index 77c06b7f..116f8ca7 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/UnsupportedTypeException.kt
+++ b/lib/reconciler/src/main/kotlin/vdi/lib/reconciler/UnsupportedTypeException.kt
@@ -1,3 +1,3 @@
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
internal class UnsupportedTypeException(message: String) : Exception(message)
\ No newline at end of file
diff --git a/service/daemon/reconciler/src/test/kotlin/vdi/daemon/reconciler/ReconcilerTest.kt b/lib/reconciler/src/test/kotlin/vdi/lib/reconciler/ReconcilerTest.kt
similarity index 98%
rename from service/daemon/reconciler/src/test/kotlin/vdi/daemon/reconciler/ReconcilerTest.kt
rename to lib/reconciler/src/test/kotlin/vdi/lib/reconciler/ReconcilerTest.kt
index 0d936666..869fccdc 100644
--- a/service/daemon/reconciler/src/test/kotlin/vdi/daemon/reconciler/ReconcilerTest.kt
+++ b/lib/reconciler/src/test/kotlin/vdi/lib/reconciler/ReconcilerTest.kt
@@ -1,6 +1,6 @@
@file:Suppress("NOTHING_TO_INLINE")
-package vdi.daemon.reconciler
+package vdi.lib.reconciler
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.DisplayName
@@ -31,7 +31,7 @@ class ReconcilerTest {
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, true)
`when`(cacheDb.streamSortedSyncControlRecords()).thenReturn(
closeableIterator(
@@ -64,7 +64,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, false)
`when`(cacheDb.streamSortedSyncControlRecords())
.thenReturn(closeableIterator(listOf(makeTargetRecord(111, "12345678123456781234567812345678")).iterator()))
@@ -86,7 +86,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, false)
`when`(cacheDb.streamSortedSyncControlRecords())
.thenReturn(closeableIterator(listOf(makeTargetRecord(111, "12345678123456781234567812345678")).iterator()))
@@ -104,7 +104,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, false)
`when`(cacheDb.streamSortedSyncControlRecords()).thenReturn(
closeableIterator(emptyList().iterator())
@@ -127,7 +127,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, true)
`when`(cacheDb.streamSortedSyncControlRecords()).thenReturn(
closeableIterator(listOf(
@@ -160,7 +160,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, true)
`when`(cacheDb.type).thenReturn(ReconcilerTargetType.Cache)
@@ -194,7 +194,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, false)
`when`(cacheDb.type).thenReturn(ReconcilerTargetType.Cache)
@@ -226,7 +226,7 @@ class ReconcilerTest {
`when`(cacheDb.name).thenReturn("CacheDB")
val datasetManager = mock()
val kafkaRouter = mock()
- val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false)
+ val recon = ReconcilerInstance(cacheDb, datasetManager, kafkaRouter, false, false)
`when`(cacheDb.type).thenReturn(ReconcilerTargetType.Cache)
diff --git a/service/bootstrap/src/main/kotlin/vdi/bootstrap/Main.kt b/service/bootstrap/src/main/kotlin/vdi/bootstrap/Main.kt
index a414ad96..dcc2e592 100644
--- a/service/bootstrap/src/main/kotlin/vdi/bootstrap/Main.kt
+++ b/service/bootstrap/src/main/kotlin/vdi/bootstrap/Main.kt
@@ -35,7 +35,7 @@ object Main {
ShareTriggerHandler(::fatality),
SoftDeleteTriggerHandler(::fatality),
UpdateMetaTriggerHandler(::fatality),
- Reconciler(::fatality),
+ Reconciler(abortCB = ::fatality),
ReconciliationEventHandler(::fatality),
)
diff --git a/service/daemon/reconciler/build.gradle.kts b/service/daemon/reconciler/build.gradle.kts
index 32fc83d9..514a8cb8 100644
--- a/service/daemon/reconciler/build.gradle.kts
+++ b/service/daemon/reconciler/build.gradle.kts
@@ -7,17 +7,9 @@ dependencies {
implementation("org.veupathdb.vdi:vdi-component-common")
- implementation(project(":lib:app-db"))
- implementation(project(":lib:cache-db"))
implementation(project(":lib:env"))
- implementation(project(":lib:plugin-client"))
- implementation(project(":lib:kafka"))
- implementation(project(":lib:metrics"))
implementation(project(":lib:module-core"))
- implementation(project(":lib:plugin-mapping"))
- implementation(project(":lib:s3"))
-
- implementation("org.veupathdb.lib.s3:s34k-minio")
+ implementation(project(":lib:reconciler"))
implementation("org.apache.logging.log4j:log4j-api-kotlin")
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerConfig.kt b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerDaemonConfig.kt
similarity index 56%
rename from service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerConfig.kt
rename to service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerDaemonConfig.kt
index 4b635e67..337ce2d4 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerConfig.kt
+++ b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerDaemonConfig.kt
@@ -1,32 +1,21 @@
package vdi.daemon.reconciler
-import org.veupathdb.lib.s3.s34k.S3Config
-import org.veupathdb.lib.s3.s34k.fields.BucketName
-import org.veupathdb.vdi.lib.common.env.Environment
import org.veupathdb.vdi.lib.common.env.optBool
import org.veupathdb.vdi.lib.common.env.optDuration
-import org.veupathdb.vdi.lib.common.env.require
import vdi.component.env.EnvKey
-import vdi.component.kafka.router.KafkaRouterConfig
-import vdi.component.s3.util.S3Config
+import vdi.component.env.Environment
import kotlin.time.Duration
import kotlin.time.Duration.Companion.hours
import kotlin.time.Duration.Companion.minutes
-data class ReconcilerConfig(
- val kafkaRouterConfig: KafkaRouterConfig,
- val s3Config: S3Config,
- val s3Bucket: BucketName,
+data class ReconcilerDaemonConfig(
val reconcilerEnabled: Boolean,
- val fullRunInterval: Duration,
- val slimRunInterval: Duration,
+ val fullRunInterval: Duration,
+ val slimRunInterval: Duration,
) {
constructor() : this(System.getenv())
constructor(env: Environment) : this(
- kafkaRouterConfig = KafkaRouterConfig(env, "reconciler"),
- s3Config = S3Config(env),
- s3Bucket = BucketName(env.require(EnvKey.S3.BucketName)),
reconcilerEnabled = env.optBool(EnvKey.Reconciler.Enabled) ?: DefaultEnabledValue,
fullRunInterval = env.optDuration(EnvKey.Reconciler.FullRunInterval) ?: DefaultFullRunInterval,
slimRunInterval = env.optDuration(EnvKey.Reconciler.SlimRunInterval) ?: DefaultSlimRunInterval,
@@ -38,4 +27,3 @@ data class ReconcilerConfig(
inline val DefaultSlimRunInterval get() = 5.minutes
}
}
-
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerImpl.kt b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerImpl.kt
index cd6689ec..7650a583 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerImpl.kt
+++ b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/ReconcilerImpl.kt
@@ -1,53 +1,23 @@
package vdi.daemon.reconciler
-import kotlinx.coroutines.coroutineScope
-import kotlinx.coroutines.launch
-import kotlinx.coroutines.runBlocking
-import org.apache.logging.log4j.kotlin.CoroutineThreadContext
-import org.apache.logging.log4j.kotlin.ThreadContextData
import org.apache.logging.log4j.kotlin.logger
-import vdi.component.db.app.AppDatabaseRegistry
-import vdi.component.kafka.router.KafkaRouter
-import vdi.component.kafka.router.KafkaRouterFactory
-import vdi.component.metrics.Metrics
import vdi.component.modules.AbortCB
import vdi.component.modules.AbstractJobExecutor
-import vdi.component.s3.DatasetManager
+import vdi.lib.reconciler.Reconciler as Recon
import kotlin.time.Duration.Companion.milliseconds
-internal class ReconcilerImpl(private val config: ReconcilerConfig, abortCB: AbortCB)
+internal class ReconcilerImpl(private val config: ReconcilerDaemonConfig, abortCB: AbortCB)
: Reconciler
, AbstractJobExecutor("reconciler", abortCB)
{
- private val log = logger().delegate
-
- private var datasetManager: DatasetManager
-
- private var kafkaRouter: KafkaRouter
-
- private val targets: List
+ private val logger = logger().delegate
private var lastSlimRun = 0L
private var lastFullRun = 0L
- private val cacheDBTarget: ReconcilerTarget
-
init {
- runBlocking {
- datasetManager = requireDatasetManager(config.s3Config, config.s3Bucket)
- kafkaRouter = requireKafkaRouter()
- }
-
- targets = ArrayList(AppDatabaseRegistry.size() + 1)
-
- AppDatabaseRegistry.iterator().asSequence()
- .map { (project, _) -> AppDBTarget(project, project) }
- .forEach { targets.add(it) }
-
- cacheDBTarget = CacheDBTarget()
-
- targets.add(cacheDBTarget)
+ Recon.initialize(abortCB)
}
override suspend fun runJob() {
@@ -58,52 +28,21 @@ internal class ReconcilerImpl(private val config: ReconcilerConfig, abortCB: Abo
// full reconciler run interval
(now - lastFullRun).milliseconds >= config.fullRunInterval -> {
lastFullRun = now
- runFull()
+
+ // If the reconciler thread is disabled, just log a reminder.
+ if (!config.reconcilerEnabled) {
+ logger.info("reconciler disabled by config")
+ } else {
+ Recon.runFull()
+ }
}
// delta between last slim run and now is greater than or equal to the
// slim reconciler run interval
(now - lastSlimRun).milliseconds >= config.slimRunInterval -> {
lastSlimRun = now
- runSlim()
+ Recon.runSlim()
}
}
}
-
- private suspend fun runSlim() {
- log.info("scheduling slim reconciler")
-
- val timer = Metrics.Reconciler.Slim.executionTime.startTimer()
-
- coroutineScope {
- launch(CoroutineThreadContext(ThreadContextData(mapOf("workerID" to workerName(true, cacheDBTarget))))) {
- ReconcilerInstance(cacheDBTarget, datasetManager, kafkaRouter, true).reconcile()
- }
- }
-
- timer.observeDuration()
- }
-
- private suspend fun runFull() {
- log.info("scheduling reconciler for {} targets", targets.size)
-
- val timer = Metrics.Reconciler.Full.reconcilerTimes.startTimer()
-
- // Schedule the reconciler for each target database.
- coroutineScope {
- targets.forEach {
- launch(CoroutineThreadContext(ThreadContextData(mapOf("workerID" to workerName(false, it))))) {
- ReconcilerInstance(it, datasetManager, kafkaRouter, false).reconcile()
- }
- }
- }
-
- timer.observeDuration()
- }
-
- private fun workerName(slim: Boolean, tgt: ReconcilerTarget) = if (slim) "slim-${tgt.name}" else "full-${tgt.name}"
-
- private suspend fun requireKafkaRouter() = safeExec("failed to create KafkaRouter instance") {
- KafkaRouterFactory(config.kafkaRouterConfig).newKafkaRouter()
- }
}
\ No newline at end of file
diff --git a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/index.kt b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/index.kt
index 09898338..825c6806 100644
--- a/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/index.kt
+++ b/service/daemon/reconciler/src/main/kotlin/vdi/daemon/reconciler/index.kt
@@ -1,4 +1,6 @@
package vdi.daemon.reconciler
-fun Reconciler(abortCB: (String?) -> Nothing, config: ReconcilerConfig = ReconcilerConfig()): Reconciler =
+import vdi.component.modules.AbortCB
+
+fun Reconciler(config: ReconcilerDaemonConfig = ReconcilerDaemonConfig(), abortCB: AbortCB): Reconciler =
ReconcilerImpl(config, abortCB)
\ No newline at end of file
diff --git a/service/rest-service/api.raml b/service/rest-service/api.raml
index f0201856..a97ad7ce 100644
--- a/service/rest-service/api.raml
+++ b/service/rest-service/api.raml
@@ -244,6 +244,18 @@ uses:
/admin:
displayName: Admin RPC
+ /reconciler:
+ displayName: Reconciler
+ post:
+ displayName: Execute Reconciler
+ description: |
+ Triggers a full reconciliation run if one is not already in progress.
+ responses:
+ 204:
+ description: Success
+ 409:
+ description: Reconciler already running.
+
/proxy-upload:
displayName: Proxy Upload
post:
@@ -329,7 +341,6 @@ uses:
displayName: List
description: |
Lists datasets that failed to import.
- headers:
queryParameters:
user:
type: lib.User-ID
diff --git a/service/rest-service/build.gradle.kts b/service/rest-service/build.gradle.kts
index 334b8ef0..b9efa32a 100644
--- a/service/rest-service/build.gradle.kts
+++ b/service/rest-service/build.gradle.kts
@@ -55,6 +55,7 @@ dependencies {
implementation(project(":lib:install-cleanup"))
implementation(project(":lib:plugin-mapping"))
implementation(project(":lib:pruner"))
+ implementation(project(":lib:reconciler"))
implementation(project(":lib:s3"))
implementation(project(":lib:metrics"))
diff --git a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/DatasetPostMetaImpl.java b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/DatasetPostMetaImpl.java
index d4a76447..38784dc1 100644
--- a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/DatasetPostMetaImpl.java
+++ b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/DatasetPostMetaImpl.java
@@ -47,8 +47,8 @@ public class DatasetPostMetaImpl implements DatasetPostMeta {
@JsonProperty("dependencies")
private List dependencies;
- @JsonProperty("createdOn")
+ @JsonProperty("createdOn")
private OffsetDateTime createdOn;
@JsonProperty("datasetType")
diff --git a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/Error.java b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/Error.java
index 4ef1bc1b..5038e851 100644
--- a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/Error.java
+++ b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/model/Error.java
@@ -11,15 +11,15 @@
property = "status"
)
@JsonSubTypes({
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.ConflictError.class),
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.MethodNotAllowedError.class),
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.FailedDependencyError.class),
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.ServerError.class),
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.BadRequestError.class),
- @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.UnauthorizedError.class),
@JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.NotFoundError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.ServerError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.FailedDependencyError.class),
@JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.ForbiddenError.class),
@JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.UnprocessableEntityError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.UnauthorizedError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.BadRequestError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.ConflictError.class),
+ @JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.MethodNotAllowedError.class),
@JsonSubTypes.Type(java.lang.String.class),
@JsonSubTypes.Type(org.veupathdb.service.vdi.generated.model.Error.class)
})
diff --git a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/Admin.java b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/Admin.java
index 3f2f22c9..e11d61ec 100644
--- a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/Admin.java
+++ b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/Admin.java
@@ -25,6 +25,10 @@
@Path("/admin")
public interface Admin {
+ @POST
+ @Path("/reconciler")
+ PostAdminReconcilerResponse postAdminReconciler();
+
@POST
@Path("/proxy-upload")
@Produces("application/json")
@@ -79,6 +83,26 @@ GetAdminListAllDatasetsResponse getAdminListAllDatasets(
@QueryParam("project_id") String projectId,
@QueryParam("include_deleted") @DefaultValue("false") Boolean includeDeleted);
+ class PostAdminReconcilerResponse extends ResponseDelegate {
+ private PostAdminReconcilerResponse(Response response, Object entity) {
+ super(response, entity);
+ }
+
+ private PostAdminReconcilerResponse(Response response) {
+ super(response);
+ }
+
+ public static PostAdminReconcilerResponse respond204() {
+ Response.ResponseBuilder responseBuilder = Response.status(204);
+ return new PostAdminReconcilerResponse(responseBuilder.build());
+ }
+
+ public static PostAdminReconcilerResponse respond409() {
+ Response.ResponseBuilder responseBuilder = Response.status(409);
+ return new PostAdminReconcilerResponse(responseBuilder.build());
+ }
+ }
+
class PostAdminProxyUploadResponse extends ResponseDelegate {
private PostAdminProxyUploadResponse(Response response, Object entity) {
super(response, entity);
diff --git a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/VdiDatasets.java b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/VdiDatasets.java
index 8acca442..dfef8fd4 100644
--- a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/VdiDatasets.java
+++ b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/resources/VdiDatasets.java
@@ -14,6 +14,7 @@
import org.veupathdb.service.vdi.generated.model.DatasetListEntry;
import org.veupathdb.service.vdi.generated.model.DatasetPostRequest;
import org.veupathdb.service.vdi.generated.model.DatasetPostResponse;
+import org.veupathdb.service.vdi.generated.model.FailedDependencyError;
import org.veupathdb.service.vdi.generated.model.ServerError;
import org.veupathdb.service.vdi.generated.model.UnauthorizedError;
import org.veupathdb.service.vdi.generated.model.UnprocessableEntityError;
@@ -102,6 +103,13 @@ public static PostVdiDatasetsResponse respond422WithApplicationJson(
return new PostVdiDatasetsResponse(responseBuilder.build(), entity);
}
+ public static PostVdiDatasetsResponse respond424WithApplicationJson(
+ FailedDependencyError entity) {
+ Response.ResponseBuilder responseBuilder = Response.status(424).header("Content-Type", "application/json");
+ responseBuilder.entity(entity);
+ return new PostVdiDatasetsResponse(responseBuilder.build(), entity);
+ }
+
public static PostVdiDatasetsResponse respond500WithApplicationJson(ServerError entity) {
Response.ResponseBuilder responseBuilder = Response.status(500).header("Content-Type", "application/json");
responseBuilder.entity(entity);
diff --git a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/support/ResponseDelegate.java b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/support/ResponseDelegate.java
index 0ece4085..6c7e8cf5 100644
--- a/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/support/ResponseDelegate.java
+++ b/service/rest-service/src/main/java/org/veupathdb/service/vdi/generated/support/ResponseDelegate.java
@@ -144,15 +144,15 @@ public MultivaluedMap getMetadata() {
return this.delegate.getMetadata();
}
- @Override
- public Object getEntity() {
- return this.entity;}
-
@Override
public MultivaluedMap getStringHeaders() {
return this.delegate.getStringHeaders();
}
+ @Override
+ public Object getEntity() {
+ return this.entity;}
+
@Override
public String getHeaderString(String p0) {
return this.delegate.getHeaderString(p0);
diff --git a/service/rest-service/src/main/kotlin/org/veupathdb/service/vdi/server/controllers/AdminRPC.kt b/service/rest-service/src/main/kotlin/org/veupathdb/service/vdi/server/controllers/AdminRPC.kt
index 7cd0cda6..838c51b4 100644
--- a/service/rest-service/src/main/kotlin/org/veupathdb/service/vdi/server/controllers/AdminRPC.kt
+++ b/service/rest-service/src/main/kotlin/org/veupathdb/service/vdi/server/controllers/AdminRPC.kt
@@ -26,6 +26,7 @@ import vdi.component.install_cleanup.InstallCleaner
import vdi.component.install_cleanup.ReinstallTarget
import vdi.component.pruner.Pruner
import vdi.component.reinstaller.DatasetReinstaller
+import vdi.lib.reconciler.Reconciler
// Broken Import Query Constants
private const val biQueryLimitMinimum = 0
@@ -45,6 +46,16 @@ class AdminRPC : Admin {
.respond200WithApplicationJson(listBrokenDatasets(expanded ?: true))
}
+ override fun postAdminReconciler(): Admin.PostAdminReconcilerResponse {
+ return runBlocking {
+ if (Reconciler.runFull()) {
+ Admin.PostAdminReconcilerResponse.respond204()
+ } else {
+ Admin.PostAdminReconcilerResponse.respond409()
+ }
+ }
+ }
+
override fun postAdminFixBrokenInstalls(
skipRun: Boolean,
entity: InstallCleanupRequest?
diff --git a/service/rest-service/src/main/resources/api.html b/service/rest-service/src/main/resources/api.html
index a93fca38..c57a4df6 100644
--- a/service/rest-service/src/main/resources/api.html
+++ b/service/rest-service/src/main/resources/api.html
@@ -1222,7 +1222,7 @@
bindFilters();
});
- Resources
get /vdi-datasets
List Datasets Returns a list of datasets available to the requesting user, optionally filtered by query parameters.
Results are sorted by creation date in reverse order. This means the most recently created datasets will be first and the oldest dataset will be last in the list.
Parameters chevron_right expand_more
Parameter Type Description Query project_id Project ID string ID of the VEuPathDB project that results should be filtered to.
This means only datasets that are relevant to the project ID given will be returned.
Additionally, this controls the sites on which the dataset installation status will be checked. Meaning, if this parameter is specified and set to, for example, PlasmoDB, the status block in the response objects will only include installation status details for PlasmoDB and not any other sites that the dataset may have been installed into.
Inherits: string
ownership Dataset Ownership Filter string Ownership status filter.
Enum of:
anyownedshared
If set to any the results are not filtered.
If set to owned, the results will be filtered to only results that are owned by the requesting user.
If set to shared, the results will be filtered to only results that are shared with the requesting user.
Default value: "any"
curl -X GET \
+ Resources
get /vdi-datasets
List Datasets Returns a list of datasets available to the requesting user, optionally filtered by query parameters.
Results are sorted by creation date in reverse order. This means the most recently created datasets will be first and the oldest dataset will be last in the list.
Parameters chevron_right expand_more
Parameter Type Description Query project_id Project ID string ID of the VEuPathDB project that results should be filtered to.
This means only datasets that are relevant to the project ID given will be returned.
Additionally, this controls the sites on which the dataset installation status will be checked. Meaning, if this parameter is specified and set to, for example, PlasmoDB, the status block in the response objects will only include installation status details for PlasmoDB and not any other sites that the dataset may have been installed into.
Inherits: string
ownership Dataset Ownership Filter string Ownership status filter.
Enum of:
anyownedshared
If set to any the results are not filtered.
If set to owned, the results will be filtered to only results that are owned by the requesting user.
If set to shared, the results will be filtered to only results that are shared with the requesting user.
Default value: "any"
curl -X GET \
undefined/vdi-datasets?ProjectID=<value>&ownership=<value>
200 OK chevron_right expand_more
Success.
This response means that all checks passed and zero or more dataset records were found for the requesting user.
application/json
application/json
Parameter Type Description [] Dataset List Item object Short entry with basic details about a dataset.
Inherits: object
[].datasetId* Dataset ID string Unique VDI Dataset identifier string.
Pattern: ^[a-zA-Z0-9_-]{14}$
Min. length: 14
Max. length: 14
Inherits: lib.VDI-ID
[].owner* Owner Details object Details about the owner of a VDI dataset.
Inherits: lib.DatasetOwner
[].owner.userId* Owner User ID integer VEuPathDB user ID of the owner of the dataset.
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: lib.User-ID
[].owner.firstName Owner First Name string [].owner.lastName Owner Last Name string [].owner.email Owner Email string [].owner.organization Owner Organization string [].datasetType* Dataset Type object Information about a specific dataset type.
Inherits: lib.DatasetTypeInfo
[].datasetType.name* Type Name string [].datasetType.displayName Type Display Name string Display name for the type. This field is ignored in requests and will always be present in responses.
[].datasetType.version* Type Version string [].visibility* Dataset Visibility string Enum:- private
- protected
- public
Inherits: lib.DatasetVisibility
[].name* Dataset Name string User provided name for the dataset.
[].summary Dataset Summary string User provided summary of the dataset.
[].description Dataset Description string User provided description of the dataset.
[].sourceUrl Source URL string URL of the dataset data source, if the dataset was uploaded via URL.
[].origin* Dataset Origin string String representing the origin of the dataset. Examples include direct-upload, nephele, or galaxy.
[].projectIds* Project IDs array Project IDs for projects the dataset record was submitted to.
[].projectIds[] Project ID string Name or ID of a target VEuPathDB project.
Valid project IDs are:
- AmoebaDB
- ClinEpiDB
- CryptoDB
- FungiDB
- GiardiaDB
- HostDB
- MicrobiomeDB
- MicrosporidiaDB
- PiroplasmaDB
- PlasmoDB
- ToxoDB
- TrichDB
- TriTrypDB
- VectorBase
- VEuPathDB
Inherits: lib.ProjectID
[].status* Status Info object Information about the import and install status of a dataset.
Inherits: lib.DatasetStatusInfo
[].status.import* string Import status of the dataset.
Value Description queuedThe dataset has not yet been processed and is waiting in the queue. in-progressThe dataset is currently being import processed. completeThe dataset has been processed and imported for installation. invalidThe dataset failed import validation. failedThe dataset import failed due to an internal server error.
Enum:- queued
- in-progress
- complete
- invalid
- failed
Inherits: lib.DatasetImportStatus
[].status.install array [].status.install[] Dataset Install Status Entry object Entry in a list of install statuses for a dataset.
Inherits: lib.DatasetInstallStatusEntry
[].status.install[].projectId* string Name or ID of a target VEuPathDB project.
Valid project IDs are:
- AmoebaDB
- ClinEpiDB
- CryptoDB
- FungiDB
- GiardiaDB
- HostDB
- MicrobiomeDB
- MicrosporidiaDB
- PiroplasmaDB
- PlasmoDB
- ToxoDB
- TrichDB
- TriTrypDB
- VectorBase
- VEuPathDB
Inherits: lib.ProjectID
[].status.install[].metaStatus string Enum:- running
- complete
- failed-validation
- failed-installation
- ready-for-reinstall
- missing-dependency
Inherits: lib.DatasetInstallStatus
[].status.install[].metaMessage string [].status.install[].dataStatus string Enum:- running
- complete
- failed-validation
- failed-installation
- ready-for-reinstall
- missing-dependency
Inherits: lib.DatasetInstallStatus
[].status.install[].dataMessage string [].shares* Shared With array [].shares[] object Inherits: object
[].shares[].userId* integer Unique user identifier
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: lib.User-ID
[].shares[].firstName* string [].shares[].lastName* string [].shares[].organization* string [].shares[].accepted* boolean [].fileCount* File Count integer Number of files uploaded for this dataset.
[].fileSizeTotal* File Size Total integer Sum of the sizes of all the files uploaded for this dataset.
Format: int64
[].created* Creation Timestamp datetime Timestamp of the creation of this dataset.
Response Body
[
{
"datasetId": "zaZqAAGLGJhBgg",
@@ -1306,11 +1306,16 @@
]
}
}
+}
424 Failed Dependency chevron_right expand_more
Failed Dependency.
Returned when the dataset data source was a URL and the VDI service encountered a non-success HTTP status code from the target URL. This could be, for example, a 403 error from an expired AWS S3 URL, or a 404 for a file that no longer exists on the remote server.
Failed Dependency Error FailedDependencyError
application/json
Parameter Type Description status* string Enum:- bad-request
- unauthorized
- forbidden
- not-found
- bad-method
- conflict
- gone
- invalid-input
- failed-dependency
- server-error
Inherits: lib.ErrorType
message* string dependency* string
Response Body
{
+ "status": "failed-dependency",
+ "dependency": "google.com",
+ "message": "unexpected status code 403 from google.com"
}
500 Internal Server Error chevron_right expand_more
Internal Server Error.
This status is returned when an unhandled or unexpected issue arises when attempting to process the request.
Internal Server Error ServerError
application/json
Parameter Type Description status* string Enum:- bad-request
- unauthorized
- forbidden
- not-found
- bad-method
- conflict
- gone
- invalid-input
- failed-dependency
- server-error
Inherits: lib.ErrorType
message* string requestId* string
Response Body
{
"status": "server-error",
"message": "Dataset store is unreachable",
"requestId": "b296c3d9-4032-41b1-906e-c97ccfc447e3"
-}
post /admin/proxy-upload
Upload a dataset on behalf of another user.
Headers chevron_right expand_more
Parameter Type Description User-ID* VEuPathDB User ID integer ID of the target user on whose behalf a dataset is being uploaded
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: integer
curl -X POST \
+}
post /admin/reconciler
Execute Reconciler Triggers a full reconciliation run if one is not already in progress.
curl -X POST \
+ undefined/admin/reconciler
204 No Content chevron_right expand_more
Success
409 Conflict chevron_right expand_more
Reconciler already running.
post /admin/proxy-upload
Upload a dataset on behalf of another user.
Headers chevron_right expand_more
Parameter Type Description User-ID* VEuPathDB User ID integer ID of the target user on whose behalf a dataset is being uploaded
Min. value: 1
Max. value: 9223372036854776000
Format: int64
Inherits: integer
curl -X POST \
-H "User-ID: <value>" \
-H "Content-type: multipart/form-data"
-d @file \
diff --git a/settings.gradle.kts b/settings.gradle.kts
index 357e5883..08319936 100644
--- a/settings.gradle.kts
+++ b/settings.gradle.kts
@@ -53,6 +53,7 @@ include(":lib:metrics")
include(":lib:module-core")
include(":lib:pruner")
include(":lib:rabbit")
+include(":lib:reconciler")
include(":lib:s3")
include(":lib:test-utils")