Skip to content

Latest commit

 

History

History
306 lines (232 loc) · 15.8 KB

UPGRADING.md

File metadata and controls

306 lines (232 loc) · 15.8 KB
Raw

Upgrading Plugins to OpenSearch and OpenSearch Dashboards

In introducing OpenSearch we promised that "The Amazon OpenSearch Service APIs will be backward compatible with the existing service APIs to eliminate any need for customers to update their current client code or applications. Additionally, just as we did for previous versions of Elasticsearch, we will provide a seamless upgrade path from existing Elasticsearch 6.x and 7.x managed clusters to OpenSearch." We also asked customers about what that would look like, and scoped what upgrading and backwards compatibility mean.

Upgrading to OpenSearch

OpenSearch and OpenSearch Dashboards 1.0 North Star is the following upgrade path.

Upgrading from Elasticsearch OSS and Kibana OSS or Open Distro for Elasticsearch (ODFE) to OpenSearch and OpenSearch Dashboards is just like upgrading between versions of Elasticsearch OSS and Kibana OSS. Specifically, OpenSearch supports rolling upgrades and restart upgrades from Elasticsearch OSS 6.8.0 through Elasticsearch OSS 7.10.2 to OpenSearch 1.0. OpenSearch Dashboards supports restart upgrades from Kibana OSS 6.8.0 through Kibana OSS 7.10.2 to OpenSearch Dashboards 1.0. All 1.x versions of ODFE similarly support upgrades to OpenSearch and OpenSearch Dashboards 1.0.

When making changes, ensure that you are not breaking anything that prevents rolling upgrades to OpenSearch or restart upgrades to OpenSearch Dashboards. Do not do any refactoring or renaming that would risk breaking everything.

You must make branding changes (e.g. text labels, icons).

You do not need to rename any "magic words" (e.g. variable names, index names, parameters) that are required by the system to function in its ALv2 form.

OpenSearch Plugins

The following steps are generally known as required to upgrade plugins to work with OpenSearch and OpenSearch Dashboards 1.0.0, with backwards compatibility, including building, and passing all tests.

Building

  1. Consume artifacts from your dependencies, including OpenSearch, see BUILDING.
  2. Change the namespaces from org.elasticsearch to org.opensearch. Use the naming convention below.
  3. Consume dependencies with OpenSearch/Dashboards 1.0.0-beta1. Continue using org.elasticsearch dependencies for securemock, mocksocket and jna-build.
  4. Ensure CI works end-to-end, build and test your plugin.
  5. Report all runtime failures of OpenSearch to OpenSearch Issues and runtime failures of plugins in plugin repositories.

See anomaly-detection#1 for an example.

Naming Conventions

The following naming convention has been adopted for renaming from Elasticsearch.

Before After
Elasticsearch OpenSearch
elasticsearch opensearch
es opensearch
Es OpenSearch
ES OpenSearch
ELASTICSEARCH OPENSEARCH

You will need to rename namespaces, classes, methods, variables and identifiers.

See below for Kibana-related naming conventions.

Backwards Compatibility

Settings

  1. Assuming your settings live in a single class, make a copy of that class and prepend LegacyOpenDistro to the copy. Do not rename the settings here in the legacy class. For example:

    package com.amazon.opendistroforelasticsearch.jobscheduler;

import org.opensearch.common.settings.Setting;
import org.opensearch.common.unit.TimeValue;

public class LegacyOpenDistroJobSchedulerSettings {

}

  2. Add Setting.Property.Deprecated to each existing setting to mark them deprecated. For example:

    public class LegacyOpenDistroJobSchedulerSettings {

   public static final Setting<TimeValue> REQUEST_TIMEOUT = Setting.positiveTimeSetting(
            "opendistro.jobscheduler.request_timeout",
            TimeValue.timeValueSeconds(10),
            Setting.Property.NodeScope, Setting.Property.Dynamic, Setting.Property.Deprecated);
}

  3. Include legacy settings in Plugin#getSettings. Example:

    public class JobSchedulerPlugin extends Plugin implements ExtensiblePlugin {

   @Override
   public List<Setting<?>> getSettings() {
      List<Setting<?>> settingList = new ArrayList<>();
      settingList.add(LegacyOpenDistroJobSchedulerSettings.REQUEST_TIMEOUT);
      settingList.add(JobSchedulerSettings.REQUEST_TIMEOUT);
      return settingList;
   }
}

    If you do not add your legacy settings here they will be archived upon the node joining the OpenSearch cluster.

  4. In the non-legacy class, rename settings and re-declare them to fallback on the legacy settings.

    public class JobSchedulerSettings {

    public static final Setting<TimeValue> REQUEST_TIMEOUT = Setting.positiveTimeSetting(
         "plugins.jobscheduler.request_timeout",
         LegacyOpenDistroJobSchedulerSettings.REQUEST_TIMEOUT,
         Setting.Property.NodeScope, Setting.Property.Dynamic);
}

    Note that this setting does not have a default value, but defaults to falling back to LegacyOpenDistroJobSchedulerSettings.REQUEST_TIMEOUT.

    Note that the naming convention for plugin settings has changed from opendistro. to plugins. If you have renamed any settings to opensearch., please change them again to plugins..

  5. Make sure you have implemented a listener to update the setting value. If a setting is set in opensearch.yml, you're all set. But if the setting is set via API, the node will start, settings will be read, defaults applied (fallback value will be used), and only afterwards fetched from the cluster and set on your node. Thus your setting will end up with the default value of the fallback.

    To fix this, add a listener. For example:

    clusterService.getClusterSettings().addSettingsUpdateConsumer(JobSchedulerSettings.REQUEST_TIMEOUT,
   value -> {
      this.requestTimeout = value;
      log.debug("Setting request timeout: " + this.requestTimeout.getMinutes());
   });

    Note that only listener on new settings are required. Updates on an old setting will be synced to the corresponding new setting automatically until the new setting is changed directly.

  6. Write tests.

     public void testSettingsGetValueWithLegacyFallback() {
     Settings settings = Settings.builder()
         .put("opendistro.jobscheduler.retry_count", 3)
     .build();
     
     assertEquals(JobSchedulerSettings.RETRY_COUNT.get(settings), Integer.valueOf(3)); 

     assertSettingDeprecationsAndWarnings(new Setting[]{
         LegacyOpenDistroJobSchedulerSettings.RETRY_COUNT
     });
 }

  7. Document your newly renamed settings. For example, documentation-website/.../alerting/settings.md. Many settings and names were bulk-changed in the docs, so this is a good time to ensure that the documentation is accurate.

See job-scheduler#20 for an example.

Rest APIs on 9200 port

All APIs on 9200 which have to be migrated can follow this design. From here on the doc will be focussed on an ODFE plugin as an example.

  1. All plugins extend BaseRestHandler class to implement their own RestAPIs.
  2. All the existing APIs with _opendistro should be supported but for maintenance mode only.
  3. All new APIs will be implemented with _plugins as defined in the Naming Conventions.
  4. The method replacedRoutes() is overriden and this should now have both _opendistro and _plugins APIs. 
    public static final String LEGACY_OPENDISTRO_AD_BASE = "/_opendistro/_anomaly_detection";
public static final String LEGACY_OPENDISTRO_AD_BASE_URI = LEGACY_OPENDISTRO_AD_BASE + "/detectors";
public static final String AD_BASE_URI = "/_plugins/_anomaly_detection";
public static final String AD_BASE_DETECTORS_URI = AD_BASE_URI + "/detectors";

@Override
public List<ReplacedRoute> replacedRoutes() {
    return ImmutableList
        .of(
            // delete anomaly detector document
            new ReplacedRoute(
                RestRequest.Method.DELETE,
                String.format(Locale.ROOT, "%s/{%s}", AnomalyDetectorPlugin.AD_BASE_DETECTORS_URI, DETECTOR_ID),
                RestRequest.Method.DELETE,
                String.format(Locale.ROOT, "%s/{%s}", AnomalyDetectorPlugin.LEGACY_OPENDISTRO_AD_BASE_URI, DETECTOR_ID)
            )
        );
}
  5. Add new tests to cover both old and new APIs. 
    public void testBackwardCompatibilityWithOpenDistro() throws IOException {
    // Create a detector
    AnomalyDetector detector = TestHelpers.randomAnomalyDetector(TestHelpers.randomUiMetadata(), null);
    String indexName = detector.getIndices().get(0);
    TestHelpers.createIndex(client(), indexName, toHttpEntity("{\"name\": \"test\"}"));

    // Verify the detector is created using legacy _opendistro API
    Response response = TestHelpers
        .makeRequest(
            client(),
            "POST",
            TestHelpers.LEGACY_OPENDISTRO_AD_BASE_DETECTORS_URI,
            ImmutableMap.of(),
            toHttpEntity(detector),
            null
        );
    assertEquals("Create anomaly detector failed", RestStatus.CREATED, restStatus(response));
    Map<String, Object> responseMap = entityAsMap(response);
    String id = (String) responseMap.get("_id");
    int version = (int) responseMap.get("_version");
    assertNotEquals("response is missing Id", AnomalyDetector.NO_ID, id);
    assertTrue("incorrect version", version > 0);

    // Get the detector using new _plugins API
    AnomalyDetector createdDetector = getAnomalyDetector(id, client());
    assertEquals("Get anomaly detector failed", createdDetector.getDetectorId(), id);

    // Delete the detector using legacy _opendistro API
    response = TestHelpers
        .makeRequest(
            client(),
            "DELETE",
            TestHelpers.LEGACY_OPENDISTRO_AD_BASE_DETECTORS_URI + "/" + createdDetector.getDetectorId(),
            ImmutableMap.of(),
            "",
            null
        );
    assertEquals("Delete anomaly detector failed", RestStatus.OK, restStatus(response));
}
  6. Add documentation for the new APIs. For example, documentation-website

See anomaly-detection#35 for an example.

Rest APIs on 9600 or other ports

All APIs on 9600 or other ports which have to be migrated can follow this design. From here on the doc will be focussed on an ODFE plugin as an example.

  1. All plugins uses HttpServer to create a server which handles HTTPS requests and is bound to an IP address and port number.
  2. All the existing APIs with _opendistro should be supported but for maintenance mode only.
  3. All new APIs will be implemented with _plugins as defined in the Naming Conventions.
  4. The method createContext is used to create HttpContext's for both _opendistro and _plugins APIs. 
    public static final String QUERY_URL = "/_plugins/_performanceanalyzer/metrics";
public static final String LEGACY_OPENDISTRO_QUERY_URL =
            "/_opendistro/_performanceanalyzer/metrics";
public static final String BATCH_METRICS_URL = "/_plugins/_performanceanalyzer/batch";
public static final String LEGACY_OPENDISTRO_BATCH_METRICS_URL =
            "/_opendistro/_performanceanalyzer/batch";
QueryMetricsRequestHandler queryMetricsRequestHandler =
                    new QueryMetricsRequestHandler(netClient, metricsRestUtil, appContext);
httpServer.createContext(QUERY_URL, queryMetricsRequestHandler);
httpServer.createContext(LEGACY_OPENDISTRO_QUERY_URL, queryMetricsRequestHandler);

QueryBatchRequestHandler queryBatchRequestHandler =
                    new QueryBatchRequestHandler(netClient, metricsRestUtil);
httpServer.createContext(BATCH_METRICS_URL, queryBatchRequestHandler);
httpServer.createContext(LEGACY_OPENDISTRO_BATCH_METRICS_URL, queryBatchRequestHandler);
  5. Add new tests to cover both old and new APIs.
  6. Add documentation for the new APIs. For example, documentation-website

See performance-analyzer#18 for an example.

Indices

Do not change index names at this time to preserve backwards compatibility.

OpenSearch Dashboards Plugins

Building

  1. Change the namespaces from Kibana to OpenSearch-Dashboards. Use the naming convention below.
  2. Download and install OpenSearch-Dashboards. Make sure the version specific in package.json matches the OpenSearch version.
  3. Create a plugins directory in the root of the project, if it doesn't exist.
  4. Run yarn osd bootstrap inside opensearch-dashboards/plugins/opensearch-dashboards-plugin.
  5. Build and test your plugin.
  6. Report all runtime failures of OpenSearch Dashboards to OpenSearch Dashboards Issues and runtime failures of plugins in plugin repositories.

Naming Conventions

Before After
kibana opensearchDashboards
Kibana OpenSearchDashboards
KIBANA OPENSEARCH_DASHBOARDS
kbn osd
Kbn Osd
KBN OSD
kib osd
Kib Osd
KIB OSD

See anomaly-detection-dashboards-plugin#1 for an example.

REST API Compatibility with OpenSearch Plugins

If your OpenSearch Dashboards plugin is communicating with a corresponding OpenSearch plugin, you will want to update any migrated API calls (see API changes here) to be compatible with the new endpoints.

For example, the Anomaly Detection OpenSearch Plugin migrated all of its APIs to support the _plugins/* prefix (PR here).

The corresponding change for the Anomaly Detection Dashboards Plugin is captured in the PR here.