Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KNOX-3026 - End-users can exclude certain services or roles from CM service discovery #893

Merged
merged 1 commit into from
Mar 28, 2024

Conversation

smolnar82
Copy link
Contributor

@smolnar82 smolnar82 commented Mar 27, 2024

What changes were proposed in this pull request?

I added two new gateway-level configurations:

  • gateway.cloudera.manager.service.discovery.excluded.service.types
  • gateway.cloudera.manager.service.discovery.excluded.role.types

Both configurations can store a collection of comma-separated Cloudera Manager service or role types that should be excluded from CM service discovery. They both default to an empty set, meaning full CM discovery.
During a CM service discovery attempt, those configurations are checked and based on the supplied values the discovery logic skips the given service(s) or role(s) discovery.

How was this patch tested?

I added and updated existing JUnit tests.
Tested on a live CM cluster:

Screenshot 2024-03-28 at 13 14 01

After restarting Knox, I exported discovery-related logs into cm_discovery.txt

grep "discovery.cm" /var/log/knox/gateway/gateway.log > cm_discovery.txt

Here is the list of relevant log entries for this PR:

$ grep -i skipping cm_discovery.txt 
2024-03-28 12:15:25,967 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$2(352)) - Skipping service discovery: CORE_SETTINGS-1 (CORE_SETTINGS)
2024-03-28 12:15:25,968 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$2(352)) - Skipping service discovery: KUDU-1 (KUDU)
2024-03-28 12:15:25,968 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$2(352)) - Skipping service discovery: YARN-1 (YARN)
2024-03-28 12:15:25,968 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$2(352)) - Skipping service discovery: IMPALA-1 (IMPALA)
2024-03-28 12:15:26,313 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HDFS-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:27,117 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HBASE-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:28,326 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:28,326 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:28,326 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)
2024-03-28 12:15:30,067 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:30,068 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:30,068 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:30,170 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SPARK_ON_YARN-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:30,170 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SPARK_ON_YARN-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:30,170 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SPARK_ON_YARN-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:30,538 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: KAFKA-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:31,943 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE_ON_TEZ-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:31,943 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE_ON_TEZ-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:31,943 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: HIVE_ON_TEZ-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:32,550 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: LIVY-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:32,551 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: LIVY-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:32,551 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: LIVY-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:32,974 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: OZONE-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)
2024-03-28 12:15:32,974 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: OZONE-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:32,974 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: OZONE-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:34,801 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:34,801 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:34,801 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)
2024-03-28 12:15:38,121 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:38,121 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:38,121 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:38,187 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:38,187 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:38,187 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)
2024-03-28 12:15:40,426 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:40,426 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:40,426 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:40,490 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:40,490 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:40,490 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)
2024-03-28 12:15:46,313 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-2 (GATEWAY)
2024-03-28 12:15:46,313 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-3 (GATEWAY)
2024-03-28 12:15:46,313 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: SQOOPCLIENT-GATEWAY-1 (GATEWAY)
2024-03-28 12:15:46,377 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-962722fbc5dd79be35fd4c71f50d9db4 (GATEWAY)
2024-03-28 12:15:46,377 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-6b773e2d140b3c8597330125d17829ce (GATEWAY)
2024-03-28 12:15:46,377 INFO  discovery.cm (ClouderaManagerServiceDiscovery.java:lambda$4(431)) - Skipping role discovery: TEZ-1-GATEWAY-70be993d2303e6d7523ef15597ea5a1c (GATEWAY)

@smolnar82 smolnar82 self-assigned this Mar 27, 2024
@smolnar82 smolnar82 marked this pull request as draft March 27, 2024 20:01
Copy link
Contributor

@pzampino pzampino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

Copy link
Contributor

@moresandeep moresandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@smolnar82 smolnar82 marked this pull request as ready for review March 28, 2024 12:17
@smolnar82 smolnar82 merged commit af09c1d into apache:master Mar 28, 2024
2 checks passed
@smolnar82 smolnar82 deleted the KNOX-3026 branch March 28, 2024 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants