Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added feature to pass additional cluster settings from command line. … #64

Merged
merged 1 commit into from
Oct 24, 2023
Merged

Added feature to pass additional cluster settings from command line. … #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 26 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,31 +32,32 @@ There are two stacks that get deployed:

In order to deploy both the stacks the user needs to provide a set of required and optional parameters listed below:

| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| dashboardsUrl (Optional) | string | OpenSearch Dashboards tar distribution url |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| suffix (Optional) | string | An optional string identifier to be concatenated with infra stack name. |
| region (Optional) | string | User provided aws region |
| account (Optional) | string | User provided aws account |
| dataNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| mlNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| dashboardsUrl (Optional) | string | OpenSearch Dashboards tar distribution url |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| additionalConfig (Optional) | string | Additional opensearch.yml config parameters passed as JSON. e.g., `--context additionalConfig='{"plugins.security.nodes_dn": ["CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE", "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"], "plugins.security.nodes_dn_dynamic_config_enabled": false}'` |
| suffix (Optional) | string | An optional string identifier to be concatenated with infra stack name. |
| region (Optional) | string | User provided aws region |
| account (Optional) | string | User provided aws account |
| dataNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| mlNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |


* Before starting this step, ensure that your AWS CLI is correctly configured with access credentials.
Expand Down
13 changes: 12 additions & 1 deletion lib/infra/infra-stack.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@
readonly mlNodeCount: number,
readonly dataNodeStorage: number,
readonly mlNodeStorage: number,
readonly jvmSysPropsString?: string
readonly jvmSysPropsString?: string,
readonly additionalConfig?: string,
}

export class InfraStack extends Stack {
Expand Down Expand Up @@ -110,7 +111,7 @@
}

if (props.singleNodeCluster) {
console.log('Single node value is true, creating single node configurations');

Check warning on line 114 in lib/infra/infra-stack.ts

View workflow job for this annotation

GitHub Actions / build 

Unexpected console statement

Check warning on line 114 in lib/infra/infra-stack.ts

View workflow job for this annotation

GitHub Actions / build 

Unexpected console statement
singleNodeInstance = new Instance(this, 'single-node-instance', {
vpc: props.vpc,
instanceType: ec2InstanceType,
Expand Down Expand Up @@ -406,7 +407,7 @@

fileContent['cluster.name'] = `${scope.stackName}-${scope.account}-${scope.region}`;

console.log(dump(fileContent).toString());

Check warning on line 410 in lib/infra/infra-stack.ts

View workflow job for this annotation

GitHub Actions / build 

Unexpected console statement

Check warning on line 410 in lib/infra/infra-stack.ts

View workflow job for this annotation

GitHub Actions / build 

Unexpected console statement
opensearchConfig = dump(fileContent).toString();
cfnInitConfig.push(InitCommand.shellCommand(`set -ex;cd opensearch; echo "${opensearchConfig}" > config/opensearch.yml`,
{
Expand Down Expand Up @@ -467,6 +468,16 @@
}));
}

// @ts-ignore
if (props.additionalConfig.toString() !== 'undefined') {
// @ts-ignore
cfnInitConfig.push(InitCommand.shellCommand(`set -ex; cd opensearch; echo "${props.additionalConfig}">>config/opensearch.yml`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

// final run command based on whether the distribution type is min or bundle
if (props.minDistribution) { // using (stackProps.minDistribution) condition is not working when false value is being sent
cfnInitConfig.push(InitCommand.shellCommand('set -ex;cd opensearch; sudo -u ec2-user nohup ./bin/opensearch >> install.log 2>&1 &',
Expand Down
13 changes: 13 additions & 0 deletions lib/os-cluster-entrypoint.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { Stack, StackProps } from 'aws-cdk-lib';
import {
AmazonLinuxCpuType, IVpc, SecurityGroup, Vpc,
} from 'aws-cdk-lib/aws-ec2';
import { dump } from 'js-yaml';
import { NetworkStack } from './networking/vpc-stack';
import { InfraStack } from './infra/infra-stack';

Expand All @@ -35,6 +36,7 @@ export class OsClusterEntrypoint {
let infraStackName: string;
let dataNodeStorage: number;
let mlNodeStorage: number;
let ymlConfig: string = 'undefined';

const vpcId: string = scope.node.tryGetContext('vpcId');
const securityGroupId = scope.node.tryGetContext('securityGroupId');
Expand Down Expand Up @@ -134,6 +136,16 @@ export class OsClusterEntrypoint {

const jvmSysProps = `${scope.node.tryGetContext('jvmSysProps')}`;

const osConfig = `${scope.node.tryGetContext('additionalConfig')}`;
if (osConfig.toString() !== 'undefined') {
try {
const jsonObj = JSON.parse(osConfig);
ymlConfig = dump(jsonObj);
} catch (e) {
throw new Error(`Encountered following error while parsing additionalConfig json parameter: ${e}`);
}
}

const suffix = `${scope.node.tryGetContext('suffix')}`;

const network = new NetworkStack(scope, 'opensearch-network-stack', {
Expand Down Expand Up @@ -179,6 +191,7 @@ export class OsClusterEntrypoint {
dataNodeStorage,
mlNodeStorage,
jvmSysPropsString: jvmSysProps,
additionalConfig: ymlConfig,
...props,
});

Expand Down
2 changes: 2 additions & 0 deletions test/os-cluster.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { OsClusterEntrypoint } from '../lib/os-cluster-entrypoint';

test('Test Resources with security disabled multi-node', () => {
const app = new App({

context: {
securityDisabled: true,
minDistribution: false,
Expand All @@ -21,6 +22,7 @@ test('Test Resources with security disabled multi-node', () => {
distVersion: '1.0.0',
serverAccessType: 'ipv4',
restrictServerAccessTo: 'all',
additionalConfig: '{ "name": "John Doe", "age": 30, "email": "[email protected]" }',
},
});

Expand Down
Loading