Skip to content

Commit

Permalink
Release 1.4.0 (#416)
Browse files Browse the repository at this point in the history
* Updated PR template and CONTRIBUTING.md to include new conventions for formatting and labeling PR's (#136)

* updated to include new conventions for formatting and labeling PR's

* added checklist item to correct base branch

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Made a new PR meant to commit in rebase-1.4.0. Same change to that one typo

* Rebase latest changes from main into release-1.4.0 (#222)

* Add a markdown file for dashboard descriptions, link new file in README.md (#138)

* Add a markdown file for dashboard descriptions

## 🗣 Description ##

<!-- Describe the "what" of your changes in detail. -->

Add dashboard-descriptions.md in /docs/markdown/reference.
Add a link to this file within the main README.md's table of contents.

### 💭 Motivation and context 

<!-- Why is this change required? -->
<!-- What problem does this change solve? How did you solve it? -->
<!-- Mention any related issue(s) here using appropriate keywords such -->
<!-- as "closes" or "resolves" to auto-close them on merge. -->

The LME repository does not have a location for dashboard descriptions.

## 🧪 Testing 

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

N/A

## ✅ Pre-approval checklist ##

- [x] Changes are limited to a single goal **AND** 
      the title reflects this in a clear human readable format
- [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document.
- [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR)
- [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide).
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.

## ✅ Post-merge Checklist

- [x] Squash all commits into one PR level commit 
- [x] Delete the branch to keep down number of branches

* Update README.md to include dashboard-descriptions.md

* Update wording for computer software overview dashboard

* Fix some grammatical changes in dashboard-descriptions.md

* Release 1.3.1 merge into main  (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <[email protected]>

* Fixes dashboard update not importing on fresh install (#167) (#169)

* Fixes dashboard update not importing on fresh install
#165

* Update upgrading.md to include status on v1.3.2, along with revisions to the document overall

* remove step 4 from upgrading.md; add additional instructions for v1.3.2

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Add proof of concept selenium tests

* Correct the script name in the doc string

* User Security Selenium Tests for No Results Panels

* First full selenium test. Currently just User Security

* WIP User HR

* Completed all dashboards. Requires testing now

* Debugging a couple unit tests that error out. Two left

* Cut dev comments

Co-authored-by: Alden Hilton <[email protected]>

* Switched script to headless mode

* added switch for headless, detached, and debug mode. Bug where driver.quit does not close window.

* Install LME in the testbed from a single script (#150)

* Adding the configure scripts

* Add scripts to zip and copy to a container for downloading in the server

* Grab the expiry time properly in copy file

* Overwrite the blob if it exists

* Add the script to download file into DC

* Script that unzips the files in a container

* Adds username argument to download files

* Add script to run scripts in container

* Adds username argument to gpo script

* Modifies the url name in the client GPO

* Adds the functionality for chapter 1 and first half of chapter 2

* Imports the sysmon GPO

* Update the variables for sysmon gpo

* Name the scripts so they are grouped together in a listing

* Echos the file download url

* Expands the domain name correctly in create ou

* Write the url output of copy file to container to a different output stream

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Echos the file download url

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all of the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Adds InstallTestbed instructions to Readme.md

* Modifies parameters to be pascal case

* ls1 not being set on DC1

* Adds Linux Only install to SetupTestbed

* Remove separate linux only script

* Update testing/Readme.md

Co-authored-by: Alden Hilton <[email protected]>

* Make number of clients consisten between scripts

* Add ports for elk stack for testing

* Update readmes to change ResourceGroupName to ResourceGroup

* Adds a switch to install linux only

* Adds simple tests to check install

* Removes the error if the old configure zip is not found.

* Adds variables to linux tests run command

* Move credential extraction to lib for use by other scripts.

* Adds npm for other testing

* Adds latest version of nodejs for testing

* Make output.log readable for tests

* Add the -m parameter in the testing readme

* Download the latest version or a specified version

* Reboot for 1.3.0

* Notes that we could have different expect scripts

* Put back in the restart after all of the domain updates

* Scp uses ls1 instead of ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Fixes chmod of the output.log for tests

* Adds venv to the gitignore

* Adds the ability to pass a branch to the installer

* Remove node installer

* Change timeout in expect script for slow connections

* Make shell files executable

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>

* Fix deploy.sh data retention failure error  (#190)

* Fix deploysh data retention failure (#179)

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Remove free (#188)

* changed the word free to no-cost or no-cost to users

* rephrased wording to 'which comes at no cost to users'

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update upgrading.md with data retention failure resolution (#189)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Automatically Add Tags to Azure Resources (#186)

* Add tags to all Azure resource creations calls

---------

Co-authored-by: Clint Baxley <[email protected]>

* Refactored long line and added switch for debug mode

* Removed unnecessary comments

* Update pull_request_template.md (#198)

* Update pull_request_template.md

Moved Squash commits from post-merge to pre-merge.

* overriding default PR template for preferred LME template

* overriding default PR template for preferred LME template

* updating issue template to shorten the template

---------

Co-authored-by: mreeve-snl <[email protected]>

* Python testbed setup (#183)

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Hide pytest_cache

* Add pycache to gitignore

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Create helpers and conftest for python tests

* Setup for using test explorer in the dev environment

* Adding azure shell requirements to docker image

* Adding Python API tests

* Merges additional tests

* Made changes to fix tests that were failing

* Separate linux only tests from others

* Create a workflow for building test environments

* Make the docker user be the same as the vbox user id

* Set up to run the installer in docker

* Pick up different fs types in data_retention

* Change the build path for building lme container

* Install lme after build

* Make lme installer executable

* Set up the build for tests

* Add the cluster workflow for github actions

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Rishi <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#206)

Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged.

* Made changes to facilitate HTML Reports on test execution  (#211)

* Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting

* Fixed Typos on Readme

* Fixed Typos on Readme

* removed tags flag from nsg because it was preventing some rules from being created (#214)

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#217)

Using keywords like "fixes" or "closes"  only auto-closes the corresponding issue if the PR is going to be merged into main.  

For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue.  

Added some documentation to clarify this.

* Create new workflow for automating the release process (#199)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: unknown <[email protected]>
Co-authored-by: Grant (SNL) <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>
Co-authored-by: Brown <[email protected]>
Co-authored-by: mreeve-snl <[email protected]>
Co-authored-by: Rishi <[email protected]>
Co-authored-by: rishagg01 <[email protected]>

* Update and rename replica function to current elastic standards (#223)

* update configelasticsearch function

* Update deploy.sh

* Update deploy.sh

* Add grammatical improvements to LME  (#239)

* Add a markdown file for dashboard descriptions, link new file in README.md (#138)

* Add a markdown file for dashboard descriptions

## 🗣 Description ##

<!-- Describe the "what" of your changes in detail. -->

Add dashboard-descriptions.md in /docs/markdown/reference.
Add a link to this file within the main README.md's table of contents.

### 💭 Motivation and context 

<!-- Why is this change required? -->
<!-- What problem does this change solve? How did you solve it? -->
<!-- Mention any related issue(s) here using appropriate keywords such -->
<!-- as "closes" or "resolves" to auto-close them on merge. -->

The LME repository does not have a location for dashboard descriptions.

## 🧪 Testing 

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

N/A

## ✅ Pre-approval checklist ##

- [x] Changes are limited to a single goal **AND** 
      the title reflects this in a clear human readable format
- [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document.
- [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR)
- [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide).
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.

## ✅ Post-merge Checklist

- [x] Squash all commits into one PR level commit 
- [x] Delete the branch to keep down number of branches

* Update README.md to include dashboard-descriptions.md

* Update wording for computer software overview dashboard

* Fix some grammatical changes in dashboard-descriptions.md

* Release 1.3.1 merge into main  (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <[email protected]>

* Fixes dashboard update not importing on fresh install (#167) (#169)

* Fixes dashboard update not importing on fresh install
#165

* Update upgrading.md to include status on v1.3.2, along with revisions to the document overall

* remove step 4 from upgrading.md; add additional instructions for v1.3.2

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Add proof of concept selenium tests

* Correct the script name in the doc string

* User Security Selenium Tests for No Results Panels

* First full selenium test. Currently just User Security

* WIP User HR

* Completed all dashboards. Requires testing now

* Cut dev comments

Co-authored-by: Alden Hilton <[email protected]>

* Debugging a couple unit tests that error out. Two left

* Install LME in the testbed from a single script (#150)

* Adding the configure scripts

* Add scripts to zip and copy to a container for downloading in the server

* Grab the expiry time properly in copy file

* Overwrite the blob if it exists

* Add the script to download file into DC

* Script that unzips the files in a container

* Adds username argument to download files

* Add script to run scripts in container

* Adds username argument to gpo script

* Modifies the url name in the client GPO

* Adds the functionality for chapter 1 and first half of chapter 2

* Imports the sysmon GPO

* Update the variables for sysmon gpo

* Name the scripts so they are grouped together in a listing

* Echos the file download url

* Expands the domain name correctly in create ou

* Write the url output of copy file to container to a different output stream

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Echos the file download url

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all of the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Adds InstallTestbed instructions to Readme.md

* Modifies parameters to be pascal case

* ls1 not being set on DC1

* Adds Linux Only install to SetupTestbed

* Remove separate linux only script

* Update testing/Readme.md

Co-authored-by: Alden Hilton <[email protected]>

* Make number of clients consisten between scripts

* Add ports for elk stack for testing

* Update readmes to change ResourceGroupName to ResourceGroup

* Adds a switch to install linux only

* Adds simple tests to check install

* Removes the error if the old configure zip is not found.

* Adds variables to linux tests run command

* Move credential extraction to lib for use by other scripts.

* Adds npm for other testing

* Adds latest version of nodejs for testing

* Make output.log readable for tests

* Add the -m parameter in the testing readme

* Download the latest version or a specified version

* Reboot for 1.3.0

* Notes that we could have different expect scripts

* Put back in the restart after all of the domain updates

* Scp uses ls1 instead of ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Fixes chmod of the output.log for tests

* Adds venv to the gitignore

* Adds the ability to pass a branch to the installer

* Remove node installer

* Change timeout in expect script for slow connections

* Make shell files executable

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>

* Fix deploy.sh data retention failure error  (#190)

* Fix deploysh data retention failure (#179)

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Remove free (#188)

* changed the word free to no-cost or no-cost to users

* rephrased wording to 'which comes at no cost to users'

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update upgrading.md with data retention failure resolution (#189)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Automatically Add Tags to Azure Resources (#186)

* Add tags to all Azure resource creations calls

---------

Co-authored-by: Clint Baxley <[email protected]>

* Switched script to headless mode

* added switch for headless, detached, and debug mode. Bug where driver.quit does not close window.

* Refactored long line and added switch for debug mode

* Removed unnecessary comments

* Update pull_request_template.md (#198)

* Update pull_request_template.md

Moved Squash commits from post-merge to pre-merge.

* overriding default PR template for preferred LME template

* overriding default PR template for preferred LME template

* updating issue template to shorten the template

---------

Co-authored-by: mreeve-snl <[email protected]>

* Python testbed setup (#183)

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Hide pytest_cache

* Add pycache to gitignore

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Create helpers and conftest for python tests

* Setup for using test explorer in the dev environment

* Adding azure shell requirements to docker image

* Adding Python API tests

* Merges additional tests

* Made changes to fix tests that were failing

* Separate linux only tests from others

* Create a workflow for building test environments

* Make the docker user be the same as the vbox user id

* Set up to run the installer in docker

* Pick up different fs types in data_retention

* Change the build path for building lme container

* Install lme after build

* Make lme installer executable

* Set up the build for tests

* Add the cluster workflow for github actions

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Rishi <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#206)

Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged.

* Made changes to facilitate HTML Reports on test execution  (#211)

* Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting

* Fixed Typos on Readme

* Fixed Typos on Readme

* removed tags flag from nsg because it was preventing some rules from being created (#214)

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#217)

Using keywords like "fixes" or "closes"  only auto-closes the corresponding issue if the PR is going to be merged into main.  

For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue.  

Added some documentation to clarify this.

* Create new workflow for automating the release process (#199)

* Update chapter4.md

* Update bug-or-error-report.md

Grammatical changes

* Update feature_request.md

grammatical changes

* Update PULL_REQUEST_TEMPLATE.md

punctuation changes

* Update Readme.md

grammatical changes

* Update chapter2.md

grammatical update

* Update chapter4.md

grammatical changes

* Update prerequisites.md

grammatical changes

* Update CONTRIBUTING.md

grammatical changes

* Update README.md

grammatical changes

* Update RELEASES.md

grammatical changes

* Update Readme.md

Grammtical changes

* Update Readme.md

more changes

* Update README.md

grammatical updates

* Update README.md

sentence updates

* Update CONTRIBUTING.md

changes

* Update RELEASES.md

comma changes

* Update RELEASES.md

grammar updates

* Update Readme.md

grammar changes

* Update chapter4.md

* Update faq.md

* Update troubleshooting.md

some small grammar changes

* Update prerequisites.md

verb changes

* Update chapter1.md

grammatical updates and changes

* Update guide_to_ous.md

updates

* Update chapter3.md

sentence structure changes and grammar updates

* Update resilience.md

removing passive voice

* Update filtering.md

sentence fixes

* Update retention.md

plural fixes

* Update other-logging.md

grammar changes

* Update backups.md

Grammatical changes

* Update certificates.md

grammar changes and sentence updates

* Update upgrading.md

sentence rewrites

* added modifications per Dan Cernoch

* small clarifications

* small clarifications added

* minor grammatical edits

* Update prerequisites.md

sentence changes

* Update chapter4.md

grammar updates and some changes

* Update troubleshooting.md

* Update CONTRIBUTING.md

* Update README.md

* Update Readme.md

* Update chapter1.md

* Update guide_to_ous.md

* Update chapter3.md

* Update upgrading.md

grammar fixes

* Update certificates.md

removing passive voice

* Update backups.md

removing passive voice

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: unknown <[email protected]>
Co-authored-by: Grant (SNL) <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>
Co-authored-by: Brown <[email protected]>
Co-authored-by: mreeve-snl <[email protected]>
Co-authored-by: Rishi <[email protected]>
Co-authored-by: rishagg01 <[email protected]>
Co-authored-by: BDCISA <[email protected]>
Co-authored-by: Christopher Morris <[email protected]>

* replacing old gpos with new ones (#346)

* Upgrading GPOs and Formatting Improvements (#347)

* Merge release 1.4.0 (#394)

* Add a markdown file for dashboard descriptions, link new file in README.md (#138)
* Add a markdown file for dashboard descriptions

## 🗣 Description ##

<!-- Describe the "what" of your changes in detail. -->

Add dashboard-descriptions.md in /docs/markdown/reference.
Add a link to this file within the main README.md's table of contents.

### 💭 Motivation and context 

<!-- Why is this change required? -->
<!-- What problem does this change solve? How did you solve it? -->
<!-- Mention any related issue(s) here using appropriate keywords such -->
<!-- as "closes" or "resolves" to auto-close them on merge. -->

The LME repository does not have a location for dashboard descriptions.

## 🧪 Testing 

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

N/A

## ✅ Pre-approval checklist ##

- [x] Changes are limited to a single goal **AND** 
      the title reflects this in a clear human readable format
- [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document.
- [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR)
- [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide).
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.

## ✅ Post-merge Checklist

- [x] Squash all commits into one PR level commit 
- [x] Delete the branch to keep down number of branches

* Update README.md to include dashboard-descriptions.md

* Update wording for computer software overview dashboard

* Fix some grammatical changes in dashboard-descriptions.md

* Release 1.3.1 merge into main  (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <[email protected]>

* Fixes dashboard update not importing on fresh install (#167) (#169)

* Fixes dashboard update not importing on fresh install
#165

* Update upgrading.md to include status on v1.3.2, along with revisions to the document overall

* remove step 4 from upgrading.md; add additional instructions for v1.3.2

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Add proof of concept selenium tests

* Correct the script name in the doc string

* User Security Selenium Tests for No Results Panels

* First full selenium test. Currently just User Security

* WIP User HR

* Completed all dashboards. Requires testing now

* Cut dev comments

Co-authored-by: Alden Hilton <[email protected]>

* Debugging a couple unit tests that error out. Two left

* Install LME in the testbed from a single script (#150)

* Adding the configure scripts

* Add scripts to zip and copy to a container for downloading in the server

* Grab the expiry time properly in copy file

* Overwrite the blob if it exists

* Add the script to download file into DC

* Script that unzips the files in a container

* Adds username argument to download files

* Add script to run scripts in container

* Adds username argument to gpo script

* Modifies the url name in the client GPO

* Adds the functionality for chapter 1 and first half of chapter 2

* Imports the sysmon GPO

* Update the variables for sysmon gpo

* Name the scripts so they are grouped together in a listing

* Echos the file download url

* Expands the domain name correctly in create ou

* Write the url output of copy file to container to a different output stream

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Echos the file download url

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all of the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Adds InstallTestbed instructions to Readme.md

* Modifies parameters to be pascal case

* ls1 not being set on DC1

* Adds Linux Only install to SetupTestbed

* Remove separate linux only script

* Update testing/Readme.md

Co-authored-by: Alden Hilton <[email protected]>

* Make number of clients consisten between scripts

* Add ports for elk stack for testing

* Update readmes to change ResourceGroupName to ResourceGroup

* Adds a switch to install linux only

* Adds simple tests to check install

* Removes the error if the old configure zip is not found.

* Adds variables to linux tests run command

* Move credential extraction to lib for use by other scripts.

* Adds npm for other testing

* Adds latest version of nodejs for testing

* Make output.log readable for tests

* Add the -m parameter in the testing readme

* Download the latest version or a specified version

* Reboot for 1.3.0

* Notes that we could have different expect scripts

* Put back in the restart after all of the domain updates

* Scp uses ls1 instead of ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Fixes chmod of the output.log for tests

* Adds venv to the gitignore

* Adds the ability to pass a branch to the installer

* Remove node installer

* Change timeout in expect script for slow connections

* Make shell files executable

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>

* Fix deploy.sh data retention failure error  (#190)

* Fix deploysh data retention failure (#179)

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Remove free (#188)

* changed the word free to no-cost or no-cost to users

* rephrased wording to 'which comes at no cost to users'

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update upgrading.md with data retention failure resolution (#189)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Automatically Add Tags to Azure Resources (#186)

* Add tags to all Azure resource creations calls

---------

Co-authored-by: Clint Baxley <[email protected]>

* Switched script to headless mode

* added switch for headless, detached, and debug mode. Bug where driver.quit does not close window.

* Refactored long line and added switch for debug mode

* Removed unnecessary comments

* Update pull_request_template.md (#198)

* Update pull_request_template.md

Moved Squash commits from post-merge to pre-merge.

* overriding default PR template for preferred LME template

* overriding default PR template for preferred LME template

* updating issue template to shorten the template

---------

Co-authored-by: mreeve-snl <[email protected]>

* Python testbed setup (#183)

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Hide pytest_cache

* Add pycache to gitignore

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Create helpers and conftest for python tests

* Setup for using test explorer in the dev environment

* Adding azure shell requirements to docker image

* Adding Python API tests

* Merges additional tests

* Made changes to fix tests that were failing

* Separate linux only tests from others

* Create a workflow for building test environments

* Make the docker user be the same as the vbox user id

* Set up to run the installer in docker

* Pick up different fs types in data_retention

* Change the build path for building lme container

* Install lme after build

* Make lme installer executable

* Set up the build for tests

* Add the cluster workflow for github actions

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Rishi <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#206)

Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged.

* Made changes to facilitate HTML Reports on test execution  (#211)

* Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting

* Fixed Typos on Readme

* Fixed Typos on Readme

* removed tags flag from nsg because it was preventing some rules from being created (#214)

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#217)

Using keywords like "fixes" or "closes"  only auto-closes the corresponding issue if the PR is going to be merged into main.  

For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue.  

Added some documentation to clarify this.

* Create new workflow for automating the release process (#199)

* Github workflows for building environments (#195)

* Run the correct installer file

* Run the installer from the root directory

* Try a self hosted github runner

* Reduce logging for docker pull.

* Adds quiet flag to docker pull command

* Pull the images before expect to reduce run time

* Install docker early in order to speed up install

* Builds the right docker-compose file

* Increase timeout for linux install expect script

* Change timeout on expect script

* Change the way expect watches the script

* Expand the timeout when waiting for Elasticsearch

* Search for more output in the expect script

* Change the match for the dots in expect

* Change the regex for matching dots

* Change the output for catching dots

* Add chrome to Dockerfile for selenium

* Import selenium tests and run python tests

* Activate venv when running tests

* Correct path for venv in the container

* Correct path for venv in the container

* Running only linux tests

* Adjust scripts to run as a non super user

* Change the permissions on the output log to source for environment variables later

* Check for output log

* Make output log available to test instantiation

* Change pytest cache dir to home for user

* Change pytest cache dir to home for user

* Change pytest cache dir permissions

* Hide get-docker.sh from installs

* Cleanup test files in workflow

* Add the cluster workflow for github actions

* Adds a cluster build

* Run the test cluster in pwsh

* Fail pipeline when commands fail

* Catch the error from powershell

* Remove duplicate run command

* Set env vars explicitly

* Modify the escape char for env vars

* Try a different method of catching errors in pwsh script

* Check failure of pwsh script

* Test successful run of build_cluster

* Test failure of script

* Capture the output from the az commands

* Continue on error condition

* Simplify run command

* Try catching failures in a new way.

* Test failure capture

* Setting error action to continue

* Remove ErrorAction

* Use docker-compose run instead

* Capture exit code to fail step

* Try propigating errors from pwsh

* Capture external command exit code

* Send lastexitcode

* Don't exit right away

* Disable immediate stop on exit

* Run simple test for exit code

* Cd to docker compose file

* Catch exec exit code

* Remove unneded flags from the command

* Adds back in the build script

* Adds an explicit exit for powershell script

* Remove spaces after escape character

* Escape the exitcode variable in the shell command

* Remove extra exit from build_cluster.ps1

* Add a passing command for build_cluster.ps1

* Move to the install directory

* Run setup testbed to get an error

* Try to build a cluster with the build_cluster.ps1 script.

* Check resource group variable

* Set the resource group name differently

* Build a cluster using the generated resource group

* Make the paths relative in the build_cluster script

* Move to the right directory to do an install

* Destroy cluster on pipeline finish

* Change the owner of the files to match the host in the development container

* Su user to remove testing files

* Run the docker-compose as root to clean up

* Run as root to clean up containers

* Build the cluster in azure

* List the files in the current directory on exec

* Run the files from the new path

* Investigate more about the file environment

* Update the envornment for building the cluster

* Update the environment users before docker up

* Try to start hung job

* List all the files with their owners in the container

* Escape the powershell commands

* Check the paths and files with bash

* Find the path we are on

* Check powershell environment

* Cd to home directory in powershell

* Cd to home directory in powershell

* Rebuild docker compose as the right user

* Change directory to source directory for powershell

* Change to proper directory for powershell

* Build a full cluster in pipeline

* Run the linux tests and check permissions of files

* Change permissions on output file with sudo

* Turn off cluster creation for speed

* Comment out building cluster in steps

* Only delete the resource group if it exists

* Adds ability to get the public ip for fw rules

* Put the tags in quotes when creating nsg rules

* Output the command being run for nsg rules

* Remove tags for nsg port definitions

* Install lme on the cluster

* Builds the full cluster install

* Cleans up the useage of the environment variables in pipeline

* Extract environment variables from the build script and use them in the GitHub workflow.

* Do a minimal linux install

* Fix the path for retrieving env vars

* Check setting of github env

* Source the env file and push it to github env

* Print some debug information to the console

* Check setting of each key in functions

* Parse the output for the passwords better

* Uses a unique id instead of run_id to make sure it is unique

* Double quote the file name for sed in output.log

* Changes the way we get passwords from output.log

* Make sure key doesn't have newline

* Escape dollar sign

* Properly escape double quotes inside of docker-compose command

* Escape all of the dollar signs in the compose command

* Write the environment variables to the githut environment

* Clean up debugging output

* Remove more debugging output

* Remove set e

* Adds function to write passwords to a file for actions

* List files in directory after writing passwords

* Export the env vars in the github file

* Fail the workflow if the environment is not set correctly

* Clean up the environment vars for the container

* Set the variables on run of the pwsh command

* Run commands on the domain controller

* Get the envrionment checker to pass

* Update passing variables to remote script

* Escape the powershell environment variables

* Change the case of the resource group env var

* Don't destroy cluster so we can manually test

* Build the entire cluster to run commands against

* Run a command on the linux machine

* Run remote tests

* Run minimal installs to debug tests

* Fix escaping for test commands

* Move to the correct directory for tests

* Add continuation characters to the lines in the script

* Remove nested double quotes

* Uses the ip of LS1 to run the tests on

* Put the cluster build command on one line

* Destroy clusters at the end

* Quote output log correctly on build

* Run all api tests on cluster

* Build full cluster and add verbose logging to pytest

* Stop deleting the cluster in the destroy_cluster.ps1 script

* Modify installer to use the new winlogbeat index pattern

* Try to get the dns to resolve ls1

* Add ls1 to the hosts file so it resolves always

* Modify tests to pass on a working cluster

* Skip the fragile test for mapping

* Set up to run selenium tests on the cluster

* Testing

* Rerun build after rebasing to the right branch

* Pass the minimal install flag to install lme

* Build complete cluster and run all tests

* Pull the images quietly if running without a terminal.

* Run the simple tests on PR checkin and the longer ones when triggered

* Build the linux docker container upon check in of a pr

* Build lme container fresh before install

* Runs an end to end build in docker and cluster

* Print out the download feedback when pulling images

* Build 1.4.0 branch

* Build the cluster using the main branch of the repository

* Allow passing branch to installers from the pipeline

* Run tests from a different base branch

* Remove the ampersand typo

* Allow passing arguments to the installer scripts

* Rearrange install arguments

* Test passing arguments in install lme

* Build lme without arguments

* Install lme with no arguments

* Run command as string in install_lme.ps1

* Build by passing arguments

* Run a complete build using arguments

* Update the sources to allow for updating in the pipeline

* Build the cluster using the latest branch

* Set up the latest branch var

* Runs an upgrade in the pipeline

* Run the upgrade in the remote linux machine

* Run upgrade on minimal install

* Checks out the current branch to run an upgrade on linux

* Capture the exit code of the upgrade script

* Check the directories we are working in

* Clone the git repository to run the upgrade

* Checkout the proper branch from origin

* Get the remote username and home dir for the remote server

* Set the home directory for the az user

* Use origin when checking out in the upgrade script

* Revert the changes to deploy.sh

* Set a dumb terminal to avoid terminal errors

* Export the terminal variable correctly

* Capture the output of the upgrade script to fail pipeline if it fails

* Revert previous changes as they seemed to break upgrade

* Use a different format for executing the pwsh script

* Destroy the cluster when done

* Output the upgrade information to the terminal

* Try capturing the docker-compose output

* Directly capture the output of the compose command

* Fixes unbalanced quote

* Build and run full cluster with an upgrade

* Builds the current brand for the cluster

* Add a unique id for the docker-compose so you can run multiple instances of the same docker-compose file

* Adds upgrade.yml to gh workflows

* Runs both a build and an upgrade

* Adds upgrade to the gh workflows

* Get gh to notice new workflow

* Match build names to parent branch

* Trigger gh to see the workflow

* Get gh actions to trigger workflow

* Update code to get gh to see the actions

* Update code to use the new workflow module.

* Trigger gh actions to run

* Get gh to run workflows

* Try to get gh to run workflows

* Change upgrade branch pulling

* Checking out branch for upgrade in a new way

* Rename workflow for upgrade

* Convert to docker compose

* Run all three builds using docker compose and -p

* Clean up docker containers

* Build the docker containers fresh for the linux_only workflow

* Adds readme and checks an upgrade where the upgrade version is the same as the current version

* Fixes typo in the workflow file

* Runs docker as sudo

* Remove the privileged flag from the lme container

* Try leaving the swarm on the host if running in non privileged environment

* Leave the swarm on the host

* Reset to run docker as privileged

* Installs the current branch in linux only

* Stop pruning system to see if elastc starts faster

* Don't take down the docker containers to see why they aren't working

* Removes the gh actions shell escape vulnerability

* Remove the docker containers at end of run

* changing .github/README.md name to prevent it apperaing on main web page (#260)

* Append the flags to the end of the password file (#263)

* Append the flags to the end of the password file

* Prints the contents of password.txt to the console

* Extract the credentials in a new way to compensate for the flags being in the file

* Tests a build that runs locally on github

* Keep container running for debugging purposes

* Fix the credentials parsing function

* Create a workflow for a burndown chart (#302)

* Display the chart in the burndown summary
* Get workflow dispatch to show
* Adds defaults for the burndown chart workflow

* Clean up debugging information from the workflow (#310)

* Clean up debugging information from the workflow

* Increase column count to match the number of columns in the board.

* Break up selenium tests (#281)

* Adding selenium directory and readme

* Separate out the selenium tests so they can be run separately

* Run selenium tests in pipeline

* Puts the variables for env one to a line

* Issue # 289 selenium test for Computer Software Overview  dashboard (#290)

* Updated Selenium tests for Computer Overview Dashboard

* Updated Selenium tests for Computer Overview Dashboard

* Updated Selenium test scripts for Health Check Dashboard (#292)

* Set up selenium tests to run on cluster test

* Point tests to the proper test folder

* Update Selenium tests for Process Explorer Dashboard (#295)

* Rewrite completed for Selenium test scripts for Security Dashboard - Security Log (#300)

* Rewrote Selenium Tests for Sysmon Summary Dashboard (#301)

* Rewrite Selenium Tests for User HR Dashboard

* Rewrite of Selenium Tests for User Security Dashboard (#304)

---------

Co-authored-by: rishagg01 <[email protected]>
Co-authored-by: Rishi <[email protected]>

* API calls code for Data Insertion (#343)

* modified:   testing/tests/api_tests/helpers.py
	new file:   testing/tests/api_tests/selenium_tests/__init__.py
	new file:   testing/tests/api_tests/selenium_tests/conftest.py
	new file:   testing/tests/api_tests/selenium_tests/fixtures/hosts.json
	new file:   testing/tests/api_tests/selenium_tests/fixtures/logonevents.json
	new file:   testing/tests/api_tests/selenium_tests/queries/filter_hosts.json
	new file:   testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json
	new file:   testing/tests/api_tests/selenium_tests/test_server.py

* commit	renamed:    testing/tests/api_tests/selenium_tests/__init__.py -> testing/tests/api_tests/data_insertion_tests/__init__.py
commit	renamed:    testing/tests/api_tests/selenium_tests/conftest.py -> testing/tests/api_tests/data_insertion_tests/conftest.py
commit	renamed:    testing/tests/api_tests/selenium_tests/fixtures/hosts.json -> testing/tests/api_tests/data_insertion_tests/fixtures/hosts.json
commit	renamed:    testing/tests/api_tests/selenium_tests/fixtures/logonevents.json -> testing/tests/api_tests/data_insertion_tests/fixtures/logonevents.json
commit	renamed:    testing/tests/api_tests/selenium_tests/queries/filter_hosts.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_hosts.json
commit	renamed:    testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_logonevents.json
commit	renamed:    testing/tests/api_tests/selenium_tests/test_server.py -> testing/tests/api_tests/data_insertion_tests/test_server.py
commit	modified:   testing/tests/api_tests/helpers.py

* Updated selenium tests for USER HR dashboard panels post data insertion (#358)

* Updated User HR Dashboard Selenium Test for User HR Logon Title panel (#385)

* Updated selenium tests for USER HR dashboard panels post data insertion

* Updated User HR Dashboard Selenium Test for User HR Logon Title panel

* Adds some extra files to gitignore

* 1.4gitignore

* Skip the non passing test

* Skip failing tests

* Comment out failing tests

* Update the merging version

---------

Co-authored-by: mitchelbaker-cisa <[email protected]>
Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: unknown <[email protected]>
Co-authored-by: Grant (SNL) <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>
Co-authored-by: Brown <[email protected]>
Co-authored-by: mreeve-snl <[email protected]>
Co-authored-by: Rishi <[email protected]>
Co-authored-by: rishagg01 <[email protected]>
Co-authored-by: Connor <[email protected]>

* AD ID Logging Dashboards and New Wec Config XML File (#388)

* New wec config xml for ad id logging and new dashboards

* updated zip to contain new GPOs

* Divine's dashboards

* Fixed bug in lme_wec_config, missing asterisk in queries

* Fixed casing on Query ID, must be Id

* Fixed typo empty path string

* Fixed scheduler typo

* Renamed dashboards to fit convention

* Dashboard descriptions

* Fixed spacing

* Fixed spacing 2

* Adjusted dashboard names

* Typo fix

* Upgrade instructions for release 1.4.0 for ID Logging

* removing text

* Adds code to upgrade to 1.4.0

* Check the latest and version in the upgrade section

* Adds a better check for upgrading from 1.3.x to 1.4.x

* Modify the version check for upgrades

* Updates the dashboards to be all exported by the exporter script

* Adds the updated menu to all of the dashboards

* Update the upgrade directions for those who downloaded the version

---------

Co-authored-by: Brown <rgbrow>
Co-authored-by: unknown <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: cbaxley <[email protected]>

* 1.4.0 merge conflicts (#418)

* Add a markdown file for dashboard descriptions, link new file in README.md (#138)

* Add a markdown file for dashboard descriptions

## 🗣 Description ##

<!-- Describe the "what" of your changes in detail. -->

Add dashboard-descriptions.md in /docs/markdown/reference.
Add a link to this file within the main README.md's table of contents.

### 💭 Motivation and context 

<!-- Why is this change required? -->
<!-- What problem does this change solve? How did you solve it? -->
<!-- Mention any related issue(s) here using appropriate keywords such -->
<!-- as "closes" or "resolves" to auto-close them on merge. -->

The LME repository does not have a location for dashboard descriptions.

## 🧪 Testing 

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

N/A

## ✅ Pre-approval checklist ##

- [x] Changes are limited to a single goal **AND** 
      the title reflects this in a clear human readable format
- [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document.
- [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR)
- [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide).
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.

## ✅ Post-merge Checklist

- [x] Squash all commits into one PR level commit 
- [x] Delete the branch to keep down number of branches

* Update README.md to include dashboard-descriptions.md

* Update wording for computer software overview dashboard

* Fix some grammatical changes in dashboard-descriptions.md

* Release 1.3.1 merge into main  (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <[email protected]>

* Fixes dashboard update not importing on fresh install (#167) (#169)

* Fixes dashboard update not importing on fresh install
#165

* Update upgrading.md to include status on v1.3.2, along with revisions to the document overall

* remove step 4 from upgrading.md; add additional instructions for v1.3.2

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Add proof of concept selenium tests

* Correct the script name in the doc string

* User Security Selenium Tests for No Results Panels

* First full selenium test. Currently just User Security

* WIP User HR

* Completed all dashboards. Requires testing now

* Cut dev comments

Co-authored-by: Alden Hilton <[email protected]>

* Debugging a couple unit tests that error out. Two left

* Install LME in the testbed from a single script (#150)

* Adding the configure scripts

* Add scripts to zip and copy to a container for downloading in the server

* Grab the expiry time properly in copy file

* Overwrite the blob if it exists

* Add the script to download file into DC

* Script that unzips the files in a container

* Adds username argument to download files

* Add script to run scripts in container

* Adds username argument to gpo script

* Modifies the url name in the client GPO

* Adds the functionality for chapter 1 and first half of chapter 2

* Imports the sysmon GPO

* Update the variables for sysmon gpo

* Name the scripts so they are grouped together in a listing

* Echos the file download url

* Expands the domain name correctly in create ou

* Write the url output of copy file to container to a different output stream

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Echos the file download url

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all of the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Adds InstallTestbed instructions to Readme.md

* Modifies parameters to be pascal case

* ls1 not being set on DC1

* Adds Linux Only install to SetupTestbed

* Remove separate linux only script

* Update testing/Readme.md

Co-authored-by: Alden Hilton <[email protected]>

* Make number of clients consisten between scripts

* Add ports for elk stack for testing

* Update readmes to change ResourceGroupName to ResourceGroup

* Adds a switch to install linux only

* Adds simple tests to check install

* Removes the error if the old configure zip is not found.

* Adds variables to linux tests run command

* Move credential extraction to lib for use by other scripts.

* Adds npm for other testing

* Adds latest version of nodejs for testing

* Make output.log readable for tests

* Add the -m parameter in the testing readme

* Download the latest version or a specified version

* Reboot for 1.3.0

* Notes that we could have different expect scripts

* Put back in the restart after all of the domain updates

* Scp uses ls1 instead of ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Fixes chmod of the output.log for tests

* Adds venv to the gitignore

* Adds the ability to pass a branch to the installer

* Remove node installer

* Change timeout in expect script for slow connections

* Make shell files executable

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>

* Fix deploy.sh data retention failure error  (#190)

* Fix deploysh data retention failure (#179)

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Remove free (#188)

* changed the word free to no-cost or no-cost to users

* rephrased wording to 'which comes at no cost to users'

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update upgrading.md with data retention failure resolution (#189)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Automatically Add Tags to Azure Resources (#186)

* Add tags to all Azure resource creations calls

---------

Co-authored-by: Clint Baxley <[email protected]>

* Switched script to headless mode

* added switch for headless, detached, and debug mode. Bug where driver.quit does not close window.

* Refactored long line and added switch for debug mode

* Removed unnecessary comments

* Update pull_request_template.md (#198)

* Update pull_request_template.md

Moved Squash commits from post-merge to pre-merge.

* overriding default PR template for preferred LME template

* overriding default PR template for preferred LME template

* updating issue template to shorten the template

---------

Co-authored-by: mreeve-snl <[email protected]>

* Python testbed se…
  • Loading branch information
18 people authored Aug 28, 2024
1 parent dcc5dc5 commit 657b844
Show file tree
Hide file tree
Showing 76 changed files with 1,926 additions and 436 deletions.
30 changes: 22 additions & 8 deletions .github/ISSUE_TEMPLATE/bug-or-error-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,38 @@ assignees: ''
## **BEFORE CREATING THE ISSUE, CHECK THE FOLLOWING GUIDES**:
- [ ] [FAQ](https://github.com/cisagov/LME/blob/main/docs/markdown/reference/faq.md)
- [ ] [Troubleshooting](https://github.com/cisagov/LME/blob/main/docs/markdown/reference/troubleshooting.md)
- [ ] Search current/closed issues for similar questions, and utilize github/google search to see if an answer exists for the error I'm encountering.
- [ ] Search current/closed issues for similar questions and utilize github/google search to see if an answer exists for the error you are encountering.

If the above did not answer your question, proceed with creating an issue below:

## Describe the bug
<!-- A clear and concise description of what the bug is. -->
<!-- A clear and concise description of what the software flaw you are experiencing looks like, or what the behavior is. -->

## Expected behavior
A clear and concise description of what you expected to happen.

## To Reproduce
<!-- Steps to reproduce the behavior. These should be clear enough that our team can understand your running environment, software/operating system versions, and anything else we might need to debug the issue. -->
<!-- Good examples can be found here: [Issue 1](https://github.com/cisagov/LME/issues/15) [Issue 2](https://github.com/cisagov/LME/issues/19). -->

### Please complete the following information
#### **Desktop:**
- OS: [e.g. Windows 10]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Sysmon v15.0, Winlogbeat 8.11.1]

#### **Setup**
- Are you running the LME machines in a virtual environment (i.e. Docker) or are you running natively on the machines?
- Which version of LME are you installing?
- Is this a first-time installation or are you upgrading? If upgrading, what was your previous version?

#### **Desktop:** (Client Machines)
- OS: [e.g. Windows 10]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Sysmon v15.0]

#### **Domain Controller:**
- OS: [e.g. Windows Server]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Winlogbeat 8.11.1]

#### **Server:**
#### **ElasticSearch/Kibana Server:**
- OS: [e.g. Ubuntu 22.04]
- Software Versions:
- ELK: [e.g. 8.7.1]
Expand All @@ -51,7 +65,7 @@ Increase the number of lines if your issue is not present, or include a relevant
## Expected behavior
A clear and concise description of what you expected to happen.

## Screenshots **OPTIONAL**
## Screenshots
If applicable, add screenshots to help explain your problem.

## Additional context
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ assignees: ''
---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
A clear and concise description of what the problem is. Ex. When I try ABC, this happens instead [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.
Expand Down
17 changes: 10 additions & 7 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@
## 🧪 Testing

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- Include details of your testing environment, and the tests you ran to. -->
<!-- see how your change affects other areas of the code, etc. -->

## ✅ Pre-approval checklist ##

- [ ] There is a [gitIssue](https://github.com/cisagov/LME/issues) that this PR resolves
- [ ] Git Issue that this PR solves has been selected in the Development section
- [ ] The PR's base branch has been modified to be the proper branch.
- [ ] Changes are limited to a single goal **AND**
the title reflects this in a clear human readable format
- [ ] Issue that this PR solves has been selected in the Development section
Expand All @@ -33,11 +35,12 @@

## ✅ Pre-merge Checklist

- [ ] All tests pass
- [ ] PR has been tested and the documentation for testing is above
- [ ] Squash and merge all commits into one PR level commit
- [ ] All tests pass.
- [ ] PR has been tested and the documentation for testing is above.
- [ ] Squash and merge all commits into one PR level commit.

## ✅ Post-merge Checklist

- [ ] Delete the branch to keep down number of branches

- [ ] Delete the branch to keep down number of branches.
- [ ] The PR is labeled with `feat` for an added new feature, `update` for an update, **OR** `fix` for a fix.
- [ ] The PR contains `Resolves #<issue #>` so that merging it closes out the corresponding issue. For example `Resolves #132`.
64 changes: 32 additions & 32 deletions .github/workflows/cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,16 @@ jobs:
build-and-test-cluster:
runs-on: self-hosted
env:
UNIQUE_ID:
IP_ADDRESS:
UNIQUE_ID:
IP_ADDRESS:
LS1_IP:
BRANCH_NAME:
elastic:

steps:
- name: Checkout repository
uses: actions/[email protected]

- name: Setup environment variables
run: |
PUBLIC_IP=$(curl -s https://api.ipify.org)
Expand All @@ -34,38 +34,38 @@ jobs:
else
echo "BRANCH_NAME=${GITHUB_REF##*/}" >> $GITHUB_ENV
fi
- name: Set up Docker Compose
run: |
sudo curl -L "https://github.com/docker/compose/releases/download/v2.3.3/docker-compose-$(uname -s)-$(uname -m)" \
-o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
- name: Set the environment for docker-compose
- name: Set the environment for docker-compose
run: |
cd testing/development
# Get the UID and GID of the current user
echo "HOST_UID=$(id -u)" > .env
echo "HOST_GID=$(id -g)" >> .env
# - name: Run Docker Compose Build to fix a user id issue in a prebuilt container
# run: |
# cd testing/development
# docker compose -p ${{ env.UNIQUE_ID }} build --no-cache
# docker compose -p ${{ env.UNIQUE_ID }} build --no-cache

- name: Run Docker Compose
run: docker compose -p ${{ env.UNIQUE_ID }} -f testing/development/docker-compose.yml up -d

- name: List docker containers to wait for them to start
run: |
docker ps
- name: List files in home directory
run: |
cd testing/development
docker compose -p ${{ env.UNIQUE_ID }} exec -T lme bash -c "pwd && ls -la"
- name: Check powershell environment
- name: Check powershell environment
run: |
set +e
cd testing/development
Expand All @@ -80,8 +80,8 @@ jobs:
if [ "$EXIT_CODE" -ne 0 ]; then
exit $EXIT_CODE
fi
- name: Build the cluster
- name: Build the cluster
run: |
set +e
cd testing/development
Expand All @@ -107,8 +107,8 @@ jobs:
. configure/lib/functions.sh
extract_ls1_ip 'LME-pipe-${{ env.UNIQUE_ID }}.cluster.output.log'
echo "LS1_IP=$LS1_IP" >> $GITHUB_ENV
- name: Install lme on cluster
- name: Install lme on cluster
run: |
set +e
cd testing/development
Expand All @@ -129,8 +129,8 @@ jobs:
if [ "$EXIT_CODE" -ne 0 ]; then
exit $EXIT_CODE
fi
- name: Set the environment passwords for other steps
- name: Set the environment passwords for other steps
run: |
cd testing/development
docker compose -p ${{ env.UNIQUE_ID }} exec -T lme bash -c "
Expand All @@ -146,8 +146,8 @@ jobs:
echo "logstash_system=$logstash_system" >> $GITHUB_ENV
echo "logstash_writer=$logstash_writer" >> $GITHUB_ENV
echo "dashboard_update=$dashboard_update" >> $GITHUB_ENV
- name: Check that the environment variables are set
- name: Check that the environment variables are set
run: |
cd testing/development
docker compose -p ${{ env.UNIQUE_ID }} exec -T lme bash -c "
Expand All @@ -158,12 +158,12 @@ jobs:
echo 'Elastic password is set'
fi
"
# - name: Run a command on the domain controller
# - name: Run a command on the domain controller
# run: |
# set +e
# set +e
# cd testing/development
# docker compose -p ${{ env.UNIQUE_ID }} exec -T lme pwsh -Command "& {
# docker compose -p ${{ env.UNIQUE_ID }} exec -T lme pwsh -Command "& {
# cd /home/admin.ackbar/LME/testing; \
# \$env:AZURE_CLIENT_ID='${{ secrets.AZURE_CLIENT_ID }}'; \
# \$env:AZURE_SECRET='${{ secrets.AZURE_SECRET }}'; \
Expand All @@ -181,12 +181,12 @@ jobs:
# }"
# EXIT_CODE=$?
# echo "Exit code: $EXIT_CODE"
# set -e
# set -e
# if [ "$EXIT_CODE" -ne 0 ]; then
# exit $EXIT_CODE
# fi
- name: Run a command on the linux machine

- name: Run a command on the linux machine
run: |
set +e
cd testing/development
Expand All @@ -212,7 +212,7 @@ jobs:
if [ "$EXIT_CODE" -ne 0 ]; then
exit $EXIT_CODE
fi
# This only passes when you do a full install
- name: Run api tests in container
run: |
Expand Down Expand Up @@ -245,7 +245,7 @@ jobs:
&& pip install -r requirements.txt \
&& sudo chmod ugo+w /home/admin.ackbar/LME/ -R \
&& pytest -v selenium_tests/"
# - name: Run selenium tests in container
# run: |
# set +e
Expand All @@ -258,8 +258,8 @@ jobs:
# && pip install -r requirements.txt \
# && sudo chmod ugo+w /home/admin.ackbar/LME/ -R \
# && python selenium_tests.py --domain ${{ env.LS1_IP }} -v"
- name: Cleanup environment

- name: Cleanup environment
if: always()
run: |
cd testing/development
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,7 @@ lme_update.sh
testing/tests/assets/style.css
.history/
**/get-docker.sh
*.vim
**.password.txt
**.ip.txt
**.swp
19 changes: 10 additions & 9 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Welcome #

We're so glad you're thinking about contributing to this open-source project! If you're unsure or hesitant to make a recommendation, just ask, submit the issue, or pull request. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contribution(s), and don't want a wall of rules to stifle innovation.
Users are welcome to contribute to LME. If you're unsure or hesitant to make a recommendation, just ask, submit the issue or pull request. The LME team appreciates any sort of contribution, and does not want to stifle innovation.

Before contributing, we encourage you to read our CONTRIBUTING policy (you are here), our LICENSE, and our README, all of which are in this repository.
Before contributing, please read the CONTRIBUTING policy (you are here), LICENSE, and README, all of which are in this repository.

## Issues

If you want to report a bug or request a new feature, the most direct method is to [create an issue](https://github.com/cisagov/development-guide/issues) in this repository.
We recommend that you first search through existing issues (both open and closed) to check if your particular issue has already been reported.
We recommend that you first search through existing issues (both open and closed) to check if another users has reported your particular issue and there is already an answer.

If it has then you might want to add a comment to the existing issue.
If your question is in an existing issue, then you might want to add a comment to the existing issue.

If it hasn't then please create a new one.
If it hasn't, then please create a new one.

Please follow the provided template and fill out all sections. We have a `BUG` and `FEATURE REQUEST` Template

Expand All @@ -25,20 +25,21 @@ Example:

## Pull Requests (PR)

If you choose to submit a pull request, it will be required to pass various sanity checks in our continuous integration (CI) pipeline, before we merge it. Your pull request may fail these checks, and that's OK. If you want you can stop there and wait for us to make the necessary corrections to ensure your code passes the CI checks, you're more than within your rights; however, it helps our team greatly if you fix the issues found by our CI pipeline.
If you choose to submit a pull request, your pull request must pass various sanity checks in the continuous integration (CI) pipeline, before merging it. Your pull request may fail these checks, and that's OK. If you want, you can stop there and wait for us to make the necessary corrections to ensure your code passes the CI checks. It helps our community if you fix the issue found by our CI pipeline.

Below are some loose requirements we'd like all PR's to follow. Our release process is documented in [Releases](releases.md).

### Quality assurance and code reviews

All PRs will be tested, vetted, and reviewed by our team before being merged with the main code base. All should be pull requested into whatever the upcoming release branch is. Find that by searching for the highest SEMVER `release-X.Y.Z` branch or following our release documentation.
Our team will test, vet and review all PR's before our team merges a PR with the main code base. All code should be pull requested into the upcoming release branch. You can find that by searching for the highest SEMVER `release-X.Y.Z` branch or following our release documentation.

### Steps to submit a PR
- All PRs should request merges back into LME's *CLOSEST* Major or Minor upcoming release branch `release-X.Y.Z`. This will be viewable in the branch list on Github. You can also refer to our release documentation for guidance.
- If the PR corresponds to an issue we are already tracking on LME's public Github [project](https://github.com/orgs/cisagov/projects/68), please comment the PR in the issue, and we will update the issue.
- If the PR does not have an issue, please create a new issue and name your branch according to the conventions [here](#branch-naming-conventions). Add a comment at the top of the pull request describing the PR and how it fits into LME's project/code. If the PR follows our other requirements listed here, we'll add it into our public project linked previously.
- If the PR does not have an issue, please create a new issue and name your branch according to the conventions [here](#branch-naming-conventions). Add a human readable title describing the PR and how it fits into LME's project/code. If the PR follows our other requirements listed here, we'll add it into our public project linked previously.
- Add the label `feat` for an added new feature, `update` for an update, **or** `fix` for a fix.
- We'll work with you to mold it to our development goals/process, so your work can be merged into LME and your Github profile gets credit for the contributions.
- Before merging we request that all commits be squashed into one commit. This way your changes to the repository are tracked, but our `git log` history does not rapidly expand.
- Before merging, we request that all commits be squashed into one commit. This way your changes to the repository are tracked, but our `git log` history does not rapidly expand.
- Thanks for wanting to submit and develop improvements for LME!!

## Public domain
Expand Down
Loading

0 comments on commit 657b844

Please sign in to comment.