Add a new node role 'cluster_manager' as the alternative for 'master' role and deprecate 'master' role

[tlfeng]

Description

Add a new node role cluster_manager, as the replacement for master role, that used in setting node.roles: [ master ].
They have got the same functionality, but can NOT be assigned to node.roles together.

• Add CLUSTER_MANAGER_ROLE in DiscoveryNodeRole class, and deprecate MASTER_ROLE
• Remove MASTER_ROLE from the DiscoveryNodeRole.BUILT_IN_ROLES, and temporarily load the role by the existing method DiscoveryNode.setAdditionalRoles
• Add a method validateRole() in DiscoveryNodeRole class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
• Add deprecation message when assigning master role in setting node.roles
• Replace most MASTER_ROLE with CLUSTER_MANAGER_ROLE in current unit and integration tests
• Add new unit and integration tests to validate CLUSTER_MANAGER_ROLE and MASTER_ROLE can be treated as the same.

More explanation:

• New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be ["cluster_manager","data","ingest"], instead of ["master","data","ingest"]
• "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
• “cluster_manager” role and "master" role have got the same abbreviation name: "m"

Error will be shown if assigning the both role to a node by setting node.roles: [ master, cluster_manager] in opensearch.yml:

[2022-03-17T07:43:44,238][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [ip-172-00-00-000] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: To promote inclusive language, [master] role is deprecated, and is replaced by [cluster_manager] role. The two roles can not be assigned together to a node.

Deprecation message will be logged when assigning master role to the setting node.roles.
In logs/opensearch_deprecation.log and logs/opensearch_deprecation.json:

[2022-03-17T23:12:34,969][DEPRECATION][o.o.d.c.n.DiscoveryNodeRole] [ip-172-31-15-168] Assigning [master] role in setting [node.roles] is deprecated. To promote inclusive language, please use [cluster_manager] role instead.

REST API changes:
Current response of Nodes Info API:

curl http://localhost:9200/_nodes/_all
...
 "nodes": {
    "USpTGYaBSIKbgSUJR2Z9lg": {
        "name": "node-0",
        "roles": ["master","data","ingest"]
        ...
    }
 }

Proposed response of Nodes Info API:

curl http://localhost:9200/_nodes/_all
...
 "nodes": {
    "USpTGYaBSIKbgSUJR2Z9lg": {
        "name": "node-0",
        "roles": ["cluster_manager","data","ingest"]
        ...
    }
 }

Current response of Cluster Stats API:

    "nodes": {
        "count": {
            "total": 1,
            "coordinating_only": 0,
            "data": 1,
            "ingest": 1,
            "master": 1,
            "remote_cluster_client": 1
        },
...

Proposed response of Cluster Stats API:

    "nodes": {
        "count": {
            "total": 1,
            "cluster_manager": 1,
            "coordinating_only": 0,
            "data": 1,
            "ingest": 1,
            "master": 1,
            "remote_cluster_client": 1
        },
...

Note: When a node has got either master or cluster_manager role, the count for both master and cluster_manager roles will be increased by 1. This behavior aims to keep backwards compatibility, so that any existing script that counting the number of master role can still get correct number.

Current response of cluster settings API:

http://localhost:9200/_cluster/settings?include_defaults
...
   "node" : {
        "data" : "true",
        "roles" : [
            "master",
            "remote_cluster_client",
            "data",
            "ingest"
       ],
        "max_local_storage_nodes" : "1",
        "processors" : "4",
        "store" : {
            "allow_mmap" : "true"
        },
        "ingest" : "true",
        "master" : "true",
...

Proposed response of cluster settings API:

http://localhost:9200/_cluster/settings?include_defaults

...

  "node": {
      "data": "true",
      "roles": [
          "cluster_manager",
          "remote_cluster_client",
          "data",
          "ingest"
      ],
      "max_local_storage_nodes": "1",
      "processors": "8",
      "store": {
          "allow_mmap": "true"
      },
      "ingest": "true",
      "master": "true",
...

Testing:

./gradlew ':rest-api-spec:yamlRestTest' --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {p0=nodes.info/10_basic/*}"

./gradlew ':rest-api-spec:yamlRestTest' --tests "org.opensearch.test.rest.ClientYamlTestSuiteIT" -Dtests.method="test {p0=cluster.stats/10_basic/*}"

BWC
./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=nodes.info/10_basic/*}"

Issues Resolved

Part of #1549

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-ci-bot]

Can one of the admins verify this patch?

[opensearch-ci-bot]

❌   Gradle Check failure 11c4ff5
Log 3217

Reports 3217

[opensearch-ci-bot]

❌   Gradle Check failure 8cc54e3
Log 3218

Reports 3218

[opensearch-ci-bot]

❌   Gradle Check failure f06766e
Log 3219

Reports 3219

[opensearch-ci-bot]

❌   Gradle Check failure e8e2e30
Log 3220

Reports 3220

[reta]

x Gradle Check failure e8e2e30 Log 3220

Reports 3220

Should be fixed by #2430

[opensearch-ci-bot]

❌   Gradle Check failure 62662abe606c0a2593fef0221ae08a9838604d37
Log 3235

Reports 3235

[opensearch-ci-bot]

❌   Gradle Check failure f59a27b
Log 3503

Reports 3503

[opensearch-ci-bot]

❌   Gradle Check failure 90f143a
Log 3504

Reports 3504

[tlfeng]

In log 3503:

> Task :qa:remote-clusters:integTest

REPRODUCE WITH: ./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=FC2233777A31216 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=de-LU -Dtests.timezone=Asia/Hong_Kong -Druntime.java=17

org.opensearch.cluster.remote.test.RemoteClustersIT > testHAProxyModeConnectionWorks FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([FC2233777A31216:8B4F6FC4C25644B]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks(RemoteClustersIT.java:125)

It's reported in issue #1703

In log 3504:

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats" -Dtests.seed=D48688166EC0CB86 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr -Dtests.timezone=GMT -Druntime.java=17

org.opensearch.indices.stats.IndexStatsIT > testFilterCacheStats FAILED
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([D48688166EC0CB86:B1B97553DEC8CCFF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats(IndexStatsIT.java:1225)

It's not reproducible locally. I created a new issue #2501

[opensearch-ci-bot]

✅   Gradle Check success 69c3795
Log 3506

Reports 3506

[opensearch-ci-bot]

✅   Gradle Check success f57741d
Log 3508

Reports 3508

[kid1412621]

ha, the AWS OpenSearch 2.5 now returned both:

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                                          cluster_manager name
x.x.x.x            44          94   7    0.44    0.66     0.89 dir       data,ingest,remote_cluster_client                   -               1f7f97b495a014000287c05398a72092
x.x.x.x           52          93   2    0.66    1.12     1.22 dir       data,ingest,remote_cluster_client                   -               87e3198d6e1a0e34a0e7cc8552aef858
x.x.x.x           42          88   2    0.47    0.69     0.83 immr      cluster_manager,ingest,master,remote_cluster_client -               d6bfe2a6e0277c2473807ab0a74b42ce
x.x.x.x           54          86   9    0.82    1.08     1.41 immr      cluster_manager,ingest,master,remote_cluster_client *               9b930c9d6c7776cfc39b1e08d40e7783
x.x.x.x           36          86   3    0.50    0.85     1.03 immr      cluster_manager,ingest,master,remote_cluster_client -               a33c3223d276d8b2509a81c8a78d1993

[dblock]

@kid1412621 I believe that was done for backwards compatibility and will be removed in 3.x