[META] Implement a Communication Mechanism Between Job Scheduler as a Plugin and Anomaly Detection as an Extension

[joshpalis]

Problem

Anomaly Detection currently takes Job Scheduler as a dependency in order to schedule Anomaly Detectors to run for a specified time interval. Job Scheduler has an SPI (Service Provider Interface) design and provides the JobSchedulerExtension interface which Anomaly Detection implements for its own Job Runner. Job Scheduler also implements the Extensible Plugin Interface which provides the capability to class load Anomaly Detection's implementations for the Job Scheduler Extension. This is the currently plugin architecture that enables Anomaly Detection to use Job Scheduler to schedule its own jobs, however in the case of Anomaly Detection as an extension running in a separate JVM process, the SPI design is not compatible.

Objective

In order to provide Anomaly Detection as an extension with the capability to schedule its own jobs, we must design and develop a communication mechanism between Anomaly Detection Extension and Job Scheduler as a plugin installed within OpenSearch. This communication mechanism should be integrated alongside the current plugin architecture in order to support both plugins and extensions.

AD Features Dependent on Job Scheduler Integration

In order to achieve support for Anomaly Detection Extension to schedule jobs, the communication mechanism must be able to facilitate the workflow triggered by the following APIs.

• [META] Features that are dependent on Job Scheduler integration with AD Extension opensearch-sdk-java#216

Start Detector Workflow

The Start Detector API triggers the process in which Anomaly Detector Jobs are configured and scheduled, therefore it is important to first understand the interactions of Job Scheduler and Anomaly Detection throughout this process within the current plugin architecture in order to determine how to build around this. The following workflow diagram describes how Job Scheduler class loads AD's JobSchedulerExtension implementations and also traces the events that invoke this job scheduling and execution.

Note : For Anomaly Detector jobs, Job Scheduler is used only to schedule and execute real-time anomaly detectors, therefore we have omitted exploration of the historical analysis workflow.

Design for Communication Mechanism between Plugins and Extensions

Issues will be created and linked here

• Work item 1 : [Extensions] Enable extensions to provide job index names and job types to Job Scheduler after JS is bootstraped by PluginsService opensearch-sdk-java#262
• Work Item 2 : [Extensions]Create Request/Response for Extension Job Index name / Job type to provide extension job index information for Job Sweeper configuration opensearch-sdk-java#274
• Work Item 3 : [Extensions] Register Transport Actions for ScheduledJobParameter and ScheduledJobRunner opensearch-sdk-java#424
• Work Item 4 [Extensions] Configure a ScheduledJobParser/ScheduledJobRunner to send a parse ScheduledJobParameter/run job request to AD Extension from JS opensearch-sdk-java#309
• Work Item 5 [Extensions] Enable extensions to send a request to acquire a LockModel from JobScheduler's lock service during their job runner execution opensearch-sdk-java#363
• Work Item 6 [Extension]Send a releaseLock request from AD Extension to JS  opensearch-sdk-java#370

Note : The work items above correspond with the purple boxes found on the start detector workflow diagram, which indicate when these various request/responses should executed