Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge submodules #2654

Open
wants to merge 38 commits into
base: integration
Choose a base branch
from
Open

Merge submodules #2654

wants to merge 38 commits into from

Conversation

ivakegg
Copy link
Collaborator

@ivakegg ivakegg commented Nov 25, 2024 •
edited
Loading

This PR pulls all of the submodules into the main datawave repo. Using the "subrepo" git mechanism (https://github.com/ingydotnet/git-subrepo#readme) we can merge in updates to submodules for existing PRs. All subsequent PRs against microservices/starters/util modules should be done against the main datawave repo.

Added profiles activated by properties of the same name to include the various submodules into a build/test. Also included several profiles/properties: utils, services, and starters that will include all of the modules of that type into the build/test. Updated the github test workflow to include everything.

It is probably easier to look at the separate commits instead. The subrepo automated commits simply pulled in the main branch of the various submodules. There is one additional commit that changes the various profiles that is worth looking at.

ivakegg added 30 commits November 25, 2024 14:50
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-type…
1f96d1a 
…-utils.git core/utils/type-utils

subrepo:
  subdir:   "core/utils/type-utils"
  merged:   "55d92d5d99"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-type-utils.git"
  branch:   "main"
  commit:   "55d92d5d99"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-util…
c5ed797 
…s.git contrib/datawave-utils

subrepo:
  subdir:   "contrib/datawave-utils"
  merged:   "4348fc36a3"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-utils.git"
  branch:   "main"
  commit:   "4348fc36a3"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-base…
342deba 
…-rest-responses.git core/base-rest-responses

subrepo:
  subdir:   "core/base-rest-responses"
  merged:   "cb0f550615"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-base-rest-responses.git"
  branch:   "main"
  commit:   "cb0f550615"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-in-m…
ed41c73 
…emory-accumulo.git core/in-memory-accumulo

subrepo:
  subdir:   "core/in-memory-accumulo"
  merged:   "8a9d2f46d2"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-in-memory-accumulo.git"
  branch:   "main"
  commit:   "8a9d2f46d2"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-metr…
6f03263 
…ics-reporter.git core/metrics-reporter

subrepo:
  subdir:   "core/metrics-reporter"
  merged:   "992378d629"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metrics-reporter.git"
  branch:   "main"
  commit:   "992378d629"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-accu…
10f3c6c 
…mulo-utils.git core/utils/accumulo-utils

subrepo:
  subdir:   "core/utils/accumulo-utils"
  merged:   "638b3eda97"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-accumulo-utils.git"
  branch:   "main"
  commit:   "638b3eda97"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
88f9ea3 
…ave-common-utils.git core/utils/common-utils

subrepo:
  subdir:   "core/utils/common-utils"
  merged:   "c96ed21342"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-common-utils.git"
  branch:   "main"
  commit:   "c96ed21342"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
5006bea 
…ave-metadata-utils.git core/utils/metadata-utils

subrepo:
  subdir:   "core/utils/metadata-utils"
  merged:   "6169ce3833"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metadata-utils.git"
  branch:   "main"
  commit:   "6169ce3833"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-pare…
4632c3c 
…nt.git microservices/microservice-parent

subrepo:
  subdir:   "microservices/microservice-parent"
  merged:   "6207c9d657"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-parent.git"
  branch:   "main"
  commit:   "6207c9d657"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-serv…
037d4b0 
…ice-parent.git microservices/microservice-service-parent

subrepo:
  subdir:   "microservices/microservice-service-parent"
  merged:   "8064d20ccf"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-service-parent.git"
  branch:   "main"
  commit:   "8064d20ccf"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
a4e7047 
…ave-accumulo-service.git microservices/services/accumulo

subrepo:
  subdir:   "microservices/services/accumulo"
  merged:   "7ecb4dd560"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-accumulo-service.git"
  branch:   "main"
  commit:   "7ecb4dd560"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
495a494 
…ave-audit-service.git microservices/services/audit

subrepo:
  subdir:   "microservices/services/audit"
  merged:   "2714c97b19"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-audit-service.git"
  branch:   "main"
  commit:   "2714c97b19"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
c5861a0 
…ave-authorization-service.git microservices/services/authorization

subrepo:
  subdir:   "microservices/services/authorization"
  merged:   "292be85b63"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-authorization-service.git"
  branch:   "main"
  commit:   "292be85b63"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
97d7ba1 
…ave-config-service.git microservices/services/config

subrepo:
  subdir:   "microservices/services/config"
  merged:   "fc187c0e4b"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-config-service.git"
  branch:   "main"
  commit:   "fc187c0e4b"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-dict…
4207b81 
…ionary-service.git microservices/services/dictionary

subrepo:
  subdir:   "microservices/services/dictionary"
  merged:   "653cf5b596"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-dictionary-service.git"
  branch:   "main"
  commit:   "653cf5b596"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-file…
4197425 
…-provider-service.git microservices/services/file-provider

subrepo:
  subdir:   "microservices/services/file-provider"
  merged:   "27080f3b94"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-file-provider-service.git"
  branch:   "main"
  commit:   "27080f3b94"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
a998ec6 
…ave-hazelcast-service.git microservices/services/hazelcast

subrepo:
  subdir:   "microservices/services/hazelcast"
  merged:   "8abc2ef7e9"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-hazelcast-service.git"
  branch:   "main"
  commit:   "8abc2ef7e9"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-map-…
84118fe 
…service.git microservices/services/map

subrepo:
  subdir:   "microservices/services/map"
  merged:   "473ec43708"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-map-service.git"
  branch:   "main"
  commit:   "473ec43708"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
9c4cb96 
…ave-mapreduce-query-service.git microservices/services/mapreduce-query

subrepo:
  subdir:   "microservices/services/mapreduce-query"
  merged:   "22af8f93c8"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-mapreduce-query-service.git"
  branch:   "main"
  commit:   "22af8f93c8"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
5de15ca 
…ave-modification-service.git microservices/services/modification

subrepo:
  subdir:   "microservices/services/modification"
  merged:   "768007f319"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-modification-service.git"
  branch:   "main"
  commit:   "768007f319"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-quer…
28f4951 
…y-service.git microservices/services/query

subrepo:
  subdir:   "microservices/services/query"
  merged:   "14c65febd3"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-service.git"
  branch:   "main"
  commit:   "14c65febd3"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
5b0f7db 
…ave-query-executor-service.git microservices/services/query-executor

subrepo:
  subdir:   "microservices/services/query-executor"
  merged:   "4517c4c29e"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-executor-service.git"
  branch:   "main"
  commit:   "4517c4c29e"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-quer…
f153b52 
…y-metric-service.git microservices/services/query-metric

subrepo:
  subdir:   "microservices/services/query-metric"
  merged:   "4bce0c8921"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-metric-service.git"
  branch:   "main"
  commit:   "4bce0c8921"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
3c95341 
…ave-spring-boot-starter-audit.git microservices/starters/audit

subrepo:
  subdir:   "microservices/starters/audit"
  merged:   "ef18c9e652"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-audit.git"
  branch:   "main"
  commit:   "ef18c9e652"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone [email protected]:NationalSecurityAgency/datawave-spri…
aeef6cc 
…ng-boot-starter-cache.git microservices/starters/cache

subrepo:
  subdir:   "microservices/starters/cache"
  merged:   "3019600791"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cache.git"
  branch:   "main"
  commit:   "3019600791"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
6f42aab 
…ave-spring-boot-starter-cached-results.git microservices/starters/cached-results

subrepo:
  subdir:   "microservices/starters/cached-results"
  merged:   "b22a3d6a17"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cached-results.git"
  branch:   "main"
  commit:   "b22a3d6a17"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
8ec75c0 
…ave-spring-boot-starter.git microservices/starters/datawave

subrepo:
  subdir:   "microservices/starters/datawave"
  merged:   "081e0a028a"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter.git"
  branch:   "main"
  commit:   "081e0a028a"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
16261fb 
…ave-spring-boot-starter-metadata.git microservices/starters/metadata

subrepo:
  subdir:   "microservices/starters/metadata"
  merged:   "28337e6c23"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-metadata.git"
  branch:   "main"
  commit:   "28337e6c23"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
ea39a27 
…ave-spring-boot-starter-query.git microservices/starters/query

subrepo:
  subdir:   "microservices/starters/query"
  merged:   "ae730eab06"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query.git"
  branch:   "main"
  commit:   "ae730eab06"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
git subrepo clone (merge) [email protected]:NationalSecurityAgency/dataw…
a9278cd 
…ave-spring-boot-starter-query-metric.git microservices/starters/query-metric

subrepo:
  subdir:   "microservices/starters/query-metric"
  merged:   "fad3ed7859"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query-metric.git"
  branch:   "main"
  commit:   "fad3ed7859"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
Copy link
Collaborator Author

ivakegg commented Nov 25, 2024

I think the README should be updated to add instructions for submodule PRs and subsequent tagging.

alerman
alerman previously approved these changes Nov 26, 2024
View reviewed changes
@ivakegg
Updated profiles to optionally add former submodules into build/test
c868816 
Updated test workflow to add all former submodules into build/.test
Changed README.md instructions from submodules to subrepo
Updated build readme
@awrigh11
Copy link
Collaborator

awrigh11 commented Nov 27, 2024 •
edited
Loading

The docker readme could also use an update since it references the sub modules
datawave/docker/README.md

ivakegg and others added 4 commits November 29, 2024 14:39
@ivakegg
git subrepo pull core/utils/metadata-utils
aac3eb4 
subrepo:
  subdir:   "core/utils/metadata-utils"
  merged:   "7f05830ac6"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metadata-utils.git"
  branch:   "main"
  commit:   "7f05830ac6"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
Updated other readme files that mention submodules
21470e7
@ivakegg
Merge branch 'integration' into merge-submodules
95b22f7
@alerman
Merge branch 'integration' into merge-submodules
f147983
alerman
alerman reviewed Dec 3, 2024
View reviewed changes
.github/workflows/tests.yml
@@ -110,14 +110,14 @@ jobs:
# USER_NAME: ${{ secrets.USER_NAME }}
# ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
# run: |
# BUILD="mvn -s $GITHUB_WORKSPACE/.github/workflows/settings.xml -V -B -e -Pdev,examples,assemble,spotbugs -Dmaven.build.cache.enabled=false -DskipServices -Ddeploy -Ddist -T1C -pl "-:config-service" clean install -DskipTests"
# BUILD="mvn -s $GITHUB_WORKSPACE/.github/workflows/settings.xml -V -B -e -Dstarters -Dservices -Dutils -Pdev,examples,assemble,spotbugs -Dmaven.build.cache.enabled=false -DonlyServiceApis -Ddeploy -Ddist -T1C -pl "-:config-service" clean install -DskipTests"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is commented out, but in the before it was specifically skipping services. Why are we now not skipping services?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is correct here. See the comment at the top of this comment block. This build was intended to build all of the starters, utilities, and service apis which is achieved with -Dstarters -Dservices -Dutils -DonlyServiceApis

alerman
alerman reviewed Dec 3, 2024
View reviewed changes
README.md

If you don't want to build the starters, util modules, and services then try this
```
mvn -Pdocker,dist -Dstarters -Dservices -Dutils clean install -T 1C
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says "if you dont want to build..." but the command adds the properties to build them. Did you mean "if you want to"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, this one was in error.... fixing

alerman
alerman requested changes Dec 3, 2024
View reviewed changes
Copy link
Collaborator

@alerman alerman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of minor updates needed to documentation and workflows

@ivakegg
Merge remote-tracking branch 'origin/integration' into merge-submodules
e8032dd
@ivakegg
git subrepo pull core/utils/metadata-utils
4bc062f 
subrepo:
  subdir:   "core/utils/metadata-utils"
  merged:   "9b84bb8b5a"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metadata-utils.git"
  branch:   "main"
  commit:   "9b84bb8b5a"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"
@ivakegg
Updated per review comments
cde0b9c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alerman alerman alerman approved these changes

@jschmidt10 jschmidt10 Awaiting requested review from jschmidt10

@keith-ratcliffe keith-ratcliffe Awaiting requested review from keith-ratcliffe

@billoley billoley Awaiting requested review from billoley

@jsduncan2 jsduncan2 Awaiting requested review from jsduncan2

@apmoriarty apmoriarty Awaiting requested review from apmoriarty

@hlgp hlgp Awaiting requested review from hlgp

@dtspence dtspence Awaiting requested review from dtspence

@lbschanno lbschanno Awaiting requested review from lbschanno

@foster33 foster33 Awaiting requested review from foster33

@awrigh11 awrigh11 Awaiting requested review from awrigh11

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ivakegg @awrigh11 @alerman