Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle duplicate config yml keys for additional configs #102

Merged
merged 3 commits into from
Feb 7, 2024
Merged

Handle duplicate config yml keys for additional configs #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9c1e9b8
Handle duplicate config yml keys for additional configs
gaiksaya Feb 7, 2024
fee39d0
Revert test
gaiksaya Feb 7, 2024
20fbaaf
typo
gaiksaya Feb 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 21 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,24 @@
# CDK for deploying single-node and multi-node OpenSearch cluster with OpenSearch Dashboards

- [Getting Started](#getting-started)
- [Deployment](#deployment)
- [Required context parameters](#required-context-parameters)
- [Interacting with OpenSearch cluster](#interacting-with-opensearch-cluster)
- [Restricting Server Access](#restricting-server-access)
- [Enable Remote Store Feature](#enable-remote-store-feature)
- [Check Logs](#check-logs)
- [Access EC2 Instances](#access-ec2-instances)
- [Port Mapping](#port-mapping)
- [Teardown](#teardown)
- [Contributing](#contributing)
- [Getting Help](#getting-help)
- [Code of Conduct](#code-of-conduct)
- [Security](#security)
- [License](#license)
- [CDK for deploying single-node and multi-node OpenSearch cluster with OpenSearch Dashboards](#cdk-for-deploying-single-node-and-multi-node-opensearch-cluster-with-opensearch-dashboards)
- [Getting Started](#getting-started)
- [Deployment](#deployment)
- [Required context parameters](#required-context-parameters)
- [Sample command to set up multi-node cluster with security enabled on x64 AL2 machine](#sample-command-to-set-up-multi-node-cluster-with-security-enabled-on-x64-al2-machine)
- [Interacting with OpenSearch cluster](#interacting-with-opensearch-cluster)
- [Sample commands](#sample-commands)
- [Restricting Server Access](#restricting-server-access)
- [Please note the load-balancer url is internet facing and can be accessed by anyone.](#please-note-the-load-balancer-url-is-internet-facing-and-can-be-accessed-by-anyone)
- [Enable Remote Store Feature](#enable-remote-store-feature)
- [Check logs](#check-logs)
- [Access EC2 Instances](#access-ec2-instances)
- [Port Mapping](#port-mapping)
- [Teardown](#teardown)
- [Contributing](#contributing)
- [Getting Help](#getting-help)
- [Code of Conduct](#code-of-conduct)
- [Security](#security)
- [License](#license)

This project enables user to deploy either a single-node or a multi-node OpenSearch cluster.
There are two stacks that get deployed:
Expand Down Expand Up @@ -56,8 +60,8 @@ In order to deploy both the stacks the user needs to provide a set of required a
| dataInstanceType | Optional | string | EC2 instance type for data node. Defaults to r5.xlarge. See options in `lib/opensearch-config/node-config.ts` for available options. E.g., `-c dataInstanceType=m5.xlarge` |
| mlInstanceType | Optional | string | EC2 instance type for ml node. Defaults to r5.xlarge. See options in `lib/opensearch-config/node-config.ts` for available options. E.g., `-c mlInstanceType=m5.xlarge` |
| jvmSysProps | Optional | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| additionalConfig | Optional | string | Additional opensearch.yml config parameters passed as JSON. e.g., `--context additionalConfig='{"plugins.security.nodes_dn": ["CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE", "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"], "plugins.security.nodes_dn_dynamic_config_enabled": false}'` |
| additionalOsdConfig | Optional | string | Additional opensearch_dashboards.yml config parameters passed as JSON. e.g., `additionalOsdConfig='{"data.search.usageTelemetry.enabled": "true"}'` |
| additionalConfig | Optional | string | Additional opensearch.yml config parameters passed as JSON. Please be aware that this JSON merges with original opensearch.yml overwriting duplicate keys e.g., `--context additionalConfig='{"plugins.security.nodes_dn": ["CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE", "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"], "plugins.security.nodes_dn_dynamic_config_enabled": false}'` |
| additionalOsdConfig | Optional | string | Additional opensearch_dashboards.yml config parameters passed as JSON. Please be aware that this JSON merges with original opensearch-dashboards.yml overwriting duplicate keys. e.g., `additionalOsdConfig='{"data.search.usageTelemetry.enabled": "true"}'` |
| suffix | Optional | string | An optional string identifier to be concatenated with infra stack name. |
| networkStackSuffix (Optional) | string | An optional string identifier to be concatenated with network stack name. | |
| region | Optional | string | User provided aws region |
Expand Down
36 changes: 20 additions & 16 deletions lib/infra/infra-stack.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -407,7 +407,7 @@
}

if (this.singleNodeCluster) {
console.log('Single node value is true, creating single node configurations');

Check warning on line 410 in lib/infra/infra-stack.ts

View workflow job for this annotation

GitHub Actions / build 

Unexpected console statement
singleNodeInstance = new Instance(this, 'single-node-instance', {
vpc: props.vpc,
instanceType: singleNodeInstanceType,
Expand Down Expand Up @@ -657,6 +657,8 @@

const cfnInitConfig: InitElement[] = [
InitPackage.yum('amazon-cloudwatch-agent'),
InitCommand.shellCommand('sudo wget -nv https://github.com/mikefarah/yq/releases/download/v4.40.5/yq_linux_amd64 '
+ '-O /usr/bin/yq && sudo chmod +x /usr/bin/yq'),
CloudwatchAgent.asInitFile('/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json',
{
agent: {
Expand Down Expand Up @@ -875,11 +877,12 @@
}

if (this.additionalConfig.toString() !== 'undefined') {
cfnInitConfig.push(InitCommand.shellCommand(`set -ex; cd opensearch; echo "${this.additionalConfig}">>config/opensearch.yml`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
cfnInitConfig.push(InitCommand.shellCommand(`set -ex; cd opensearch/config; echo "${this.additionalConfig}">additionalConfig.yml; `
+ 'yq eval-all -i \'. as $item ireduce ({}; . * $item)\' opensearch.yml additionalConfig.yml -P',
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

if (this.customConfigFiles !== 'undefined') {
Expand Down Expand Up @@ -907,12 +910,12 @@
ignoreErrors: false,
}));
} else {
// eslint-disable-next-line max-len
cfnInitConfig.push(InitCommand.shellCommand(`set -ex;cd opensearch; sudo -u ec2-user nohup env OPENSEARCH_INITIAL_ADMIN_PASSWORD=${this.adminPassword} ./opensearch-tar-install.sh >> install.log 2>&1 &`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
cfnInitConfig.push(InitCommand.shellCommand('set -ex;cd opensearch; '
+ `sudo -u ec2-user nohup env OPENSEARCH_INITIAL_ADMIN_PASSWORD=${this.adminPassword} ./opensearch-tar-install.sh >> install.log 2>&1 &`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

// If OpenSearch-Dashboards URL is present
Expand Down Expand Up @@ -941,11 +944,12 @@
}

if (this.additionalOsdConfig.toString() !== 'undefined') {
cfnInitConfig.push(InitCommand.shellCommand(`set -ex;cd opensearch-dashboards; echo "${this.additionalOsdConfig}">>config/opensearch_dashboards.yml`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
cfnInitConfig.push(InitCommand.shellCommand(`set -ex;cd opensearch-dashboards/config; echo "${this.additionalOsdConfig}">additionalOsdConfig.yml; `
+ 'yq eval-all -i \'. as $item ireduce ({}; . * $item)\' opensearch_dashboards.yml additionalOsdConfig.yml -P',
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

// Starting OpenSearch-Dashboards
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@opensearch-project/opensearch-cluster-cdk",
"version": "1.0.1",
"version": "1.1.0",
"bin": {
"cdk_v2": "bin/app.js"
},
Expand Down
59 changes: 59 additions & 0 deletions test/opensearch-cluster-cdk.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -768,3 +768,62 @@ test('Should not throw error when security is enabled and version is less than 2
env: { account: 'test-account', region: 'us-east-1' },
});
});

test('Test additionalConfig overriding values', () => {
const app = new App({
context: {
securityDisabled: true,
minDistribution: false,
distributionUrl: 'www.example.com',
cpuArch: 'x64',
singleNodeCluster: false,
dashboardsUrl: 'www.example.com',
distVersion: '1.0.0',
serverAccessType: 'ipv4',
restrictServerAccessTo: 'all',
managerNodeCount: 0,
dataNodeCount: 3,
dataNodeStorage: 200,
customRoleArn: 'arn:aws:iam::12345678:role/customRoleName',
additionalConfig: '{ "cluster.name": "custom-cdk", "network.port": "8041" }',
additionalOsdConfig: '{ "something.enabled": "true", "something_else.enabled": "false" }',
},
});

// WHEN
const networkStack = new NetworkStack(app, 'opensearch-network-stack', {
env: { account: 'test-account', region: 'us-east-1' },
});

// @ts-ignore
const infraStack = new InfraStack(app, 'opensearch-infra-stack', {
vpc: networkStack.vpc,
securityGroup: networkStack.osSecurityGroup,
env: { account: 'test-account', region: 'us-east-1' },
});

// THEN
const infraTemplate = Template.fromStack(infraStack);
infraTemplate.resourceCountIs('AWS::IAM::Role', 0);
infraTemplate.hasResource('AWS::AutoScaling::AutoScalingGroup', {
/* eslint-disable max-len */
Metadata: {
'AWS::CloudFormation::Init': {
config: {
commands: {
'011': {
command: "set -ex; cd opensearch/config; echo \"cluster.name: custom-cdk\nnetwork.port: '8041'\n\">additionalConfig.yml; yq eval-all -i '. as $item ireduce ({}; . * $item)' opensearch.yml additionalConfig.yml -P",
cwd: '/home/ec2-user',
ignoreErrors: false,
},
'016': {
command: "set -ex;cd opensearch-dashboards/config; echo \"something.enabled: 'true'\nsomething_else.enabled: 'false'\n\">additionalOsdConfig.yml; yq eval-all -i '. as $item ireduce ({}; . * $item)' opensearch_dashboards.yml additionalOsdConfig.yml -P",
cwd: '/home/ec2-user',
ignoreErrors: false,
},
},
},
},
},
});
});
Loading