Skip to content

Commit

Permalink
Merge branch 'main' into SplitRfsPhasesIntoSeparateGradleProjects
Browse files Browse the repository at this point in the history
  • Loading branch information
gregschohn authored Jun 18, 2024
2 parents 67bdd19 + 2fa83e2 commit 6381c73
Show file tree
Hide file tree
Showing 14 changed files with 267 additions and 158 deletions.
32 changes: 32 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,35 @@ The release process is standard across repositories in this org and is run by a
2. The [release-drafter.yml](.github/workflows/release-drafter.yml) will be automatically kicked off and a draft release will be created.
3. This draft release triggers the [jenkins release workflow](https://build.ci.opensearch.org/job/opensearch-migrations-release) as a result of which the opensearch-migrations toolset is released and published on artifacts.opensearch.org example as https://artifacts.opensearch.org/migrations/0.1.0/opensearch-migrations-0.1.0.tar.gz.
4. Once the above release workflow is successful, the drafted release on GitHub is published automatically.

## Publishing

This project can be published to a local maven repository with:
```sh
./gradlew publishToMavenLocal
```

And subsequently imported into a separate gradle project with (replacing name with any subProject name)
```groovy
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
implementation group: "org.opensearch.migrations.trafficcapture", name: "captureKafkaOffloader", version: "0.1.0-SNAPSHOT"
//... other dependencies
}
```

The entire list of published subprojects can be viewed with
```sh
./gradlew listPublishedArtifacts
```


To include a testFixture dependency, define the import like

```groovy
testImplementation testFixtures('org.opensearch.migrations.trafficcapture:trafficReplayer:0.1.0-SNAPSHOT')
```
4 changes: 2 additions & 2 deletions RFS/src/main/java/com/rfs/common/Uid.java
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,13 @@ private static String decodeBase64Id(byte[] idBytes, int offset, int length) {
}

/** Decode an indexed id back to its original form.
* @see #encodeId */
* @see <a href="https://github.com/elastic/elasticsearch/blob/6.8/server/src/main/java/org/elasticsearch/index/mapper/Uid.java#L232">org.elasticsearch.index.mapper.Uid#encodeId</a> */
public static String decodeId(byte[] idBytes) {
return decodeId(idBytes, 0, idBytes.length);
}

/** Decode an indexed id back to its original form.
* @see #encodeId */
* @see <a href="https://github.com/elastic/elasticsearch/blob/6.8/server/src/main/java/org/elasticsearch/index/mapper/Uid.java#L232">org.elasticsearch.index.mapper.Uid#encodeId</a> */
public static String decodeId(byte[] idBytes, int offset, int length) {
if (length == 0) {
throw new IllegalArgumentException("Ids can't be empty");
Expand Down
47 changes: 0 additions & 47 deletions TrafficCapture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,50 +176,3 @@ echo 'export OS_MIGRATIONS_GRADLE_SCAN_TOS_AGREE_AND_ENABLED=' >> ~/.zshrc
```

Access your detailed build reports by following the link provided at the end of your Gradle command's output.

## Publishing

This project can be published to a local maven repository with:
```sh
../gradlew publishToMavenLocal
```

And subsequently imported into a separate gradle project with (replacing name with any subProject name)
```groovy
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
implementation group: "org.opensearch.migrations.trafficcapture", name: "captureKafkaOffloader", version: "0.1.0-SNAPSHOT"
//... other dependencies
}
```

The entire list of published subprojects is
```text
captureKafkaOffloader
captureOffloader
captureProtobufs
coreUtilities
jsonJMESPathMessageTransformer
jsonJMESPathMessageTransformerProvider
jsonJoltMessageTransformer
jsonJoltMessageTransformerProvider
jsonMessageTransformerInterface
jsonMessageTransformers
nettyWireLogging
openSearch23PlusTargetTransformerProvider
transformationPlugins
testUtilities
trafficCaptureProxyServer
trafficCaptureProxyServerTest
trafficReplayer
```

To include a testFixture dependency, define the import like

```groovy
testImplementation testFixtures('org.opensearch.migrations.trafficcapture:trafficReplayer:0.1.0-SNAPSHOT')
```
72 changes: 0 additions & 72 deletions TrafficCapture/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -27,78 +27,6 @@ subprojects {
}
}
}

task javadocJar(type: Jar, dependsOn: javadoc) {
archiveClassifier.set('javadoc')
from javadoc.destinationDir
}
task sourcesJar(type: Jar) {
archiveClassifier.set('sources')
from sourceSets.main.allSource
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
def excludedProjects = [
'buildSrc',
'dockerSolution',
]
if (!(project.name in excludedProjects)) {
publishing {
publications {
mavenJava(MavenPublication) {
from components.java
artifact javadocJar
artifact sourcesJar


group = 'org.opensearch.migrations.trafficcapture'

// support -Dbuild.version, but include default
version = System.getProperty("build.version", "0.1.0")

// support -Dbuild.snapshot=false, but default to true
if (System.getProperty("build.snapshot", "true") == "true") {
version += "-SNAPSHOT"
}


pom {
name = project.name
description = 'Everything opensearch migrations'
url = 'http://github.com/opensearch-project/opensearch-migrations'

licenses {
license {
name = 'The Apache License, Version 2.0'
url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}
developers {
developer {
name = "OpenSearch"
url = "https://github.com/opensearch-project/opensearch-migrations"
}
}
scm {
connection = "scm:[email protected]:opensearch-project/opensearch-migrations.git"
developerConnection = "scm:[email protected]:opensearch-project/opensearch-migrations.git"
url = "[email protected]:opensearch-project/opensearch-migrations.git"
}
}

// Suppress POM metadata warnings for test fixtures
suppressPomMetadataWarningsFor('testFixturesApiElements')
suppressPomMetadataWarningsFor('testFixturesRuntimeElements')
}
}
repositories {
maven { url = "${rootProject.buildDir}/repository"}
maven {
url "https://aws.oss.sonatype.org/content/repositories/snapshots"
name = 'staging'
}
}
}
}
}

allprojects {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ services:
- sharedReplayerOutput:/shared-replayer-output
- ./migrationConsole/lib/console_link/services.yaml:/etc/migration_services.yaml
# this is a convenience thing for testing -- it should be removed before this makes it to prod.
- ./migrationConsole/lib/console_link:/root/lib/console_link
# - ./migrationConsole/lib/console_link:/root/lib/console_link
- ~/.aws:/root/.aws
environment:
- MIGRATION_KAFKA_BROKER_ENDPOINTS=kafka:9092
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,6 @@ tqdm==4.66.4

# Specified [h5py==3.10.0] as 3.11.0 version was causing issues with ARM64 builds
# Error: Building h5py requires pkg-config unless the HDF5 path is explicitly specified using the environment variable HDF5_DIR
h5py==3.10.0
h5py==3.10.0
# Numpy 2.0 was released on 2024-06-16 and h5py is not yet compatible with it. Locking to a sub-2.0 version for now.
numpy>=1.26.4,<2.0
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ class CloudwatchMetricsSource(MetricsSource):
def __init__(self, config: Dict) -> None:
super().__init__(config)
logger.info(f"Initializing CloudwatchMetricsSource from config {config}")
if "aws_region" in config["cloudwatch"]:
if type(config["cloudwatch"]) is dict and "aws_region" in config["cloudwatch"]:
self.aws_region = config["cloudwatch"]["aws_region"]
self.boto_config = botocore.config.Config(region_name=self.aws_region)
else:
Expand Down
97 changes: 97 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,101 @@ plugins {
task buildDockerImages() {
dependsOn(':TrafficCapture:dockerSolution:buildDockerImages')
dependsOn(':RFS:buildDockerImages')
}

subprojects {
apply plugin: 'java'
apply plugin: 'maven-publish'

tasks.withType(Test) {
// Getting javadoc to compile is part of the test suite to ensure we are able to publish our artifacts
dependsOn project.javadoc
}

task javadocJar(type: Jar, dependsOn: javadoc) {
archiveClassifier.set('javadoc')
from javadoc.destinationDir
}

task sourcesJar(type: Jar) {
archiveClassifier.set('sources')
from sourceSets.main.allSource
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
def excludedProjectPaths = [
':RFS',
':TrafficCapture',
':TrafficCapture:dockerSolution',
]
if (!(project.path in excludedProjectPaths)) {
publishing {
publications {
mavenJava(MavenPublication) {
from components.java
artifact javadocJar
artifact sourcesJar

group = 'org.opensearch.migrations.trafficcapture'

// support -Dbuild.version, but include default
version = System.getProperty("build.version", "0.1.0")

// support -Dbuild.snapshot=false, but default to true
if (System.getProperty("build.snapshot", "true") == "true") {
version += "-SNAPSHOT"
}

pom {
name = project.name
description = 'Everything opensearch migrations'
url = 'http://github.com/opensearch-project/opensearch-migrations'

licenses {
license {
name = 'The Apache License, Version 2.0'
url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}
developers {
developer {
name = "OpenSearch"
url = "https://github.com/opensearch-project/opensearch-migrations"
}
}
scm {
connection = "scm:[email protected]:opensearch-project/opensearch-migrations.git"
developerConnection = "scm:[email protected]:opensearch-project/opensearch-migrations.git"
url = "[email protected]:opensearch-project/opensearch-migrations.git"
}
}

// Suppress POM metadata warnings for test fixtures
suppressPomMetadataWarningsFor('testFixturesApiElements')
suppressPomMetadataWarningsFor('testFixturesRuntimeElements')
}
}
repositories {
maven { url = "${rootProject.buildDir}/repository"}
maven {
url "https://aws.oss.sonatype.org/content/repositories/snapshots"
name = 'staging'
}
}
}
}
}

task listPublishedArtifacts {
doLast {
subprojects.each { proj ->
def publishingExtension = proj.extensions.findByType(PublishingExtension)
if (publishingExtension) {
publishingExtension.publications.each { publication ->
if (publication instanceof MavenPublication) {
println "${publication.groupId}.${publication.artifactId}"
}
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@ import * as yaml from 'yaml';

export class ClusterYaml {
endpoint: string = '';
no_auth?: string;
basic_auth?: object;
no_auth?: string | null;
basic_auth?: object | null;
}

export class MetricsSourceYaml {
type: string = 'cloudwatch';
region?: string;
cloudwatch? : object | null = null;
}

export class ServicesYaml {
Expand All @@ -21,6 +20,9 @@ export class ServicesYaml {
source_cluster: this.source_cluster,
target_cluster: this.target_cluster,
metrics_source: this.metrics_source
},
{
'nullStr': ''
})
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,8 @@ export class MigrationConsoleStack extends MigrationServiceCore {

const openSearchPolicy = createOpenSearchIAMAccessPolicy(this.partition, this.region, this.account)
const openSearchServerlessPolicy = createOpenSearchServerlessIAMAccessPolicy(this.partition, this.region, this.account)
let servicePolicies = [replayerOutputMountPolicy, openSearchPolicy, openSearchServerlessPolicy, ecsUpdateServicePolicy, clusterTasksPolicy, listTasksPolicy, artifactS3PublishPolicy, describeVPCPolicy, getSSMParamsPolicy]
let servicePolicies = [replayerOutputMountPolicy, openSearchPolicy, openSearchServerlessPolicy, ecsUpdateServicePolicy, clusterTasksPolicy,
listTasksPolicy, artifactS3PublishPolicy, describeVPCPolicy, getSSMParamsPolicy, getMetricsPolicy]
if (props.streamingSourceType === StreamingSourceType.AWS_MSK) {
const mskAdminPolicies = this.createMSKAdminIAMPolicies(props.stage, props.defaultDeployId)
servicePolicies = servicePolicies.concat(mskAdminPolicies)
Expand Down Expand Up @@ -318,8 +319,8 @@ export class MigrationConsoleStack extends MigrationServiceCore {
environment: environment,
taskRolePolicies: servicePolicies,
cpuArchitecture: props.fargateCpuArch,
taskCpuUnits: 512,
taskMemoryLimitMiB: 1024,
taskCpuUnits: 1024,
taskMemoryLimitMiB: 2048,
...props
});
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ test('Test that services yaml parameter is created', () => {
const yamlFileContents = value['Fn::Join'][1].join('')
expect(yamlFileContents).toContain('source_cluster')
expect(yamlFileContents).toContain('target_cluster')
expect(yamlFileContents).toContain('metrics_source:\n type: cloudwatch')
expect(yamlFileContents).toContain('metrics_source:\n cloudwatch:')

// Validates that the file can be parsed as valid yaml and has the expected fields
const parsedFromYaml = yaml.parse(yamlFileContents);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ import { ClusterYaml, ServicesYaml } from "../lib/migration-services-yaml"
test('Test default servicesYaml can be stringified', () => {
const servicesYaml = new ServicesYaml();
expect(servicesYaml.metrics_source).toBeDefined();
expect(servicesYaml.metrics_source.type).toBe("cloudwatch");
expect(Object.keys(servicesYaml.metrics_source)).toContain("cloudwatch");
const yaml = servicesYaml.stringify();
expect(yaml).toBe("metrics_source:\n type: cloudwatch\n");
expect(yaml).toBe("metrics_source:\n cloudwatch:\n");
})

test('Test servicesYaml with cluster can be stringified', () => {
Expand All @@ -15,7 +15,7 @@ test('Test servicesYaml with cluster can be stringified', () => {

expect(servicesYaml.target_cluster).toBeDefined();
const yaml = servicesYaml.stringify();
expect(yaml).toBe(`target_cluster:\n endpoint: ${cluster.endpoint}\n no_auth: ""\nmetrics_source:\n type: cloudwatch\n`);
expect(yaml).toBe(`target_cluster:\n endpoint: ${cluster.endpoint}\n no_auth: ""\nmetrics_source:\n cloudwatch:\n`);
})

test('Test servicesYaml with cluster can be stringified', () => {
Expand All @@ -31,5 +31,5 @@ test('Test servicesYaml with cluster can be stringified', () => {
expect(servicesYaml.source_cluster).toBeDefined();
const yaml = servicesYaml.stringify();
const sourceClusterYaml = `source_cluster:\n endpoint: ${sourceCluster.endpoint}\n basic_auth:\n user: ${sourceClusterUser}\n password: ${sourceClusterPassword}\n`
expect(yaml).toBe(`${sourceClusterYaml}target_cluster:\n endpoint: ${targetCluster.endpoint}\n no_auth: ""\nmetrics_source:\n type: cloudwatch\n`);
expect(yaml).toBe(`${sourceClusterYaml}target_cluster:\n endpoint: ${targetCluster.endpoint}\n no_auth: ""\nmetrics_source:\n cloudwatch:\n`);
})
Loading

0 comments on commit 6381c73

Please sign in to comment.