Skip to content

Commit

Permalink
Added feature to pass additional cluster settings from command line. (#…
Browse files Browse the repository at this point in the history 
…21)

* make OSD optional and other minor improvements

Signed-off-by: Rishabh Singh <[email protected]>

* Feature to pass additional cluster settings from command line

Signed-off-by: Rishabh Singh <[email protected]>

---------

Signed-off-by: Rishabh Singh <[email protected]>
(cherry picked from commit 5307477)
  • Loading branch information
@rishabh6788
rishabh6788 committed Oct 24, 2023
1 parent 8874063 commit 4e68443
Show file tree
Hide file tree
Showing 4 changed files with 61 additions and 23 deletions.
47 changes: 26 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,27 +32,32 @@ There are two stacks that get deployed:

In order to deploy both the stacks the user needs to provide a set of required and optional parameters listed below:

| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| dashboardsUrl (required) | string | OpenSearch Dashboards tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |

| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| dashboardsUrl (Optional) | string | OpenSearch Dashboards tar distribution url |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| additionalConfig (Optional) | string | Additional opensearch.yml config parameters passed as JSON. e.g., `--context additionalConfig='{"plugins.security.nodes_dn": ["CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE", "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"], "plugins.security.nodes_dn_dynamic_config_enabled": false}'` |
| suffix (Optional) | string | An optional string identifier to be concatenated with infra stack name. |
| region (Optional) | string | User provided aws region |
| account (Optional) | string | User provided aws account |
| dataNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| mlNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
* Before starting this step, ensure that your AWS CLI is correctly configured with access credentials.
* Also ensure that you're running these commands in the current directory

Expand Down
15 changes: 14 additions & 1 deletion lib/infra/infra-stack.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ export interface infraProps extends StackProps{
readonly ingestNodeCount: number,
readonly clientNodeCount: number,
readonly mlNodeCount: number,
readonly jvmSysPropsString?: string
readonly dataNodeStorage: number,
readonly mlNodeStorage: number,
readonly jvmSysPropsString?: string,
readonly additionalConfig?: string,
}

export class InfraStack extends Stack {
Expand Down Expand Up @@ -471,6 +474,16 @@ export class InfraStack extends Stack {
}));
}

// @ts-ignore
if (props.additionalConfig.toString() !== 'undefined') {
// @ts-ignore
cfnInitConfig.push(InitCommand.shellCommand(`set -ex; cd opensearch; echo "${props.additionalConfig}">>config/opensearch.yml`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

// final run command based on whether the distribution type is min or bundle
if (props.minDistribution) { // using (stackProps.minDistribution) condition is not working when false value is being sent
cfnInitConfig.push(InitCommand.shellCommand('set -ex;cd opensearch; sudo -u ec2-user nohup ./bin/opensearch >> install.log 2>&1 &',
Expand Down
20 changes: 19 additions & 1 deletion lib/os-cluster-entrypoint.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { Stack, StackProps } from 'aws-cdk-lib';
import {
AmazonLinuxCpuType, IVpc, SecurityGroup, Vpc,
} from 'aws-cdk-lib/aws-ec2';
import { dump } from 'js-yaml';
import { NetworkStack } from './networking/vpc-stack';
import { InfraStack } from './infra/infra-stack';

Expand All @@ -32,6 +33,10 @@ export class OsClusterEntrypoint {
let clientCount: number;
let ingestCount: number;
let mlCount: number;
let infraStackName: string;
let dataNodeStorage: number;
let mlNodeStorage: number;
let ymlConfig: string = 'undefined';

const vpcId: string = scope.node.tryGetContext('vpcId');
const securityGroupId = scope.node.tryGetContext('securityGroupId');
Expand Down Expand Up @@ -120,7 +125,19 @@ export class OsClusterEntrypoint {

const jvmSysProps = `${scope.node.tryGetContext('jvmSysProps')}`;

const network = new NetworkStack(scope, 'OpenSearch-Network-Stack', {
const osConfig = `${scope.node.tryGetContext('additionalConfig')}`;
if (osConfig.toString() !== 'undefined') {
try {
const jsonObj = JSON.parse(osConfig);
ymlConfig = dump(jsonObj);
} catch (e) {
throw new Error(`Encountered following error while parsing additionalConfig json parameter: ${e}`);
}
}

const suffix = `${scope.node.tryGetContext('suffix')}`;

const network = new NetworkStack(scope, 'opensearch-network-stack', {
cidrBlock: cidrRange,
maxAzs: 3,
vpcId,
Expand Down Expand Up @@ -155,6 +172,7 @@ export class OsClusterEntrypoint {
securityGroup: this.securityGroup,
singleNodeCluster: isSingleNode,
jvmSysPropsString: jvmSysProps,
additionalConfig: ymlConfig,
...props,
});

Expand Down
2 changes: 2 additions & 0 deletions test/os-cluster.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { OsClusterEntrypoint } from '../lib/os-cluster-entrypoint';

test('Test Resources with security disabled multi-node', () => {
const app = new App({

context: {
securityDisabled: true,
minDistribution: false,
Expand All @@ -21,6 +22,7 @@ test('Test Resources with security disabled multi-node', () => {
distVersion: '1.0.0',
serverAccessType: 'ipv4',
restrictServerAccessTo: 'all',
additionalConfig: '{ "name": "John Doe", "age": 30, "email": "[email protected]" }',
},
});

Expand Down

0 comments on commit 4e68443

Please sign in to comment.