TSC Meeting (01 Feb 2022): New pack content overrides, AWS, v3.7.0 progress

[arm4b]

February 2022 @StackStorm/tsc 1 hour planned meeting: Tuesday, 01 Feb 2022, 09:30 AM US Pacific.

See #33 for more info about how to join.
(TLDR; Zoom link: https://us02web.zoom.us/j/81082101702?pwd=N1V4TWdYRVQ4SXBsaFh1TFYvVDA0UT09)

TSC Meeting Host: @armab

Meeting Agenda

DEMO: New Pack content loading by @amanda11 (15min)

• Add support for overridding whether resources are enabled st2#5506
• Amanda will present a demo showcasing the new feature from the upcoming v3.7.0 that adds configuration in /opt/stackstorm/overrides/ to control the pack content loading: a whitelist/blacklist for sensors, rules, etc.

AWS p1 (bizdev): Change of account ownership request by @punkrokk (BlueCycle)

Context: 1.5+ years ago @punkrokk created a dedicated AWS account and requested on behalf of StackStorm project FOSS credits in AWS Open Source program. This way the original StackStorm AWS account (child org) moved away from Extreme AWS (parent org) to the new umbrella of created AWS FOSS account (parent org). This helped the project to keep its infra and builds operating as before.

• @punkrokk (representing BlueCycle, a StackStorm partner) is looking for someone else to take the AWS account ownership, including the billing and business mgmt.
• StackStorm Partners are invited to the meeting, as the project is seeking a potential new AWS account holder or AWS parent org.

AWS p2 (technical): Future plans

Context: AWS FOSS credits will expire mid 2022 with no possibility for extending, per AWS support. StackStorm release automation and e2e testing rely on AWS API where st2cicd requests on-demand VMs to automatically test the stackstorm end-to-end installation in a fresh environment. This happens for every build x 4 OS flavors we support (EL7/8, U18/20).

• Discuss the future plans about AWS infra
  • Short-term: Further reduce the AWS usage
  • Long-term: Move away from AWS (?)
  • Consequences for the st2cicd, packaging, future releases, OS support, deployments (?)
  • etc

StackStorm v3.7.0 release progress

• Release Manager: @cognifloyd
• Release Assistant: still missing a volunteer. Who from @StackStorm/maintainers?
• Target month: March 2022
• Project & Roadmap: https://github.com/orgs/StackStorm/projects/23
  • New feature: Disable/Enable st2 content on load: Add support for overridding whether resources are enabled st2#5506 by @amanda11
  • CentOS 8 EOL end 2021 #88 plan:
    • RockyLinux will be a drop-in replacement for EOL'ed CentOS8
    • update the build process, packaging, CI/CD
  • Problem: py3.6 was EOL in Dec 2021: still used in EL7, EL8, U18

[arm4b]

Participants

Attendees:

• @StackStorm/maintainers: @m4dcoder @winem @cognifloyd @armab @guzzijones @amanda11 + Partners: @punkrokk (BlueCycle), @mickmcgrath13 (Bitovi), @bishopbm1 (Encore)

Meeting Minutes

DEMO: New Pack content loading blacklist/whitelist /opt/stackstorm/overrides/ feature by @amanda11

• PR & Discussion: Add support for overridding whether resources are enabled st2#5506
• @amanda11's notes from today's demo and TSC feedback:
  • Add support for overridding whether resources are enabled st2#5506 (comment)
• @punkrokk @bishopbm1 and others are invited to document their ideas in more detail in the PR above ^^
• @m4dcoder would be welcomed for review
• More 👀 are needed from the @StackStorm/maintainers

AWS

FOSS AWS Credits for parent AWS org will expire on March 31, 2022.

• @bishopbm1 - Encore is relying heavily on StackStorm in its core automation and will examine what they can do to help with the infrastructure: Internal Encore cloud platform, AWS parent org (if possible). Bradley will need the usage stats, details, budgets, to follow-up with the approval, etc, but very positive about the outcome in supporting the project in general.
• @bishopbm1, @punkrokk, @armab will work together on details and follow-up, keeping the TSC informed.
• @guzzijones will raise the question of sponsoring the st2 infra as well from his work org. @mickmcgrath13 as well?
• @punkrokk to explore the possibility to use the AWS spot instances in e2e testing (as promised in the previous meetings: July '21, Aug '21)
• Short-Term:
  • Ideally: transfer the AWS ownership from @punkrokk BlueCycle to @bishopbm1 Encore
• Mid-Term:
  • Before AWS FOSS Credits expiration (March 31, 2022)
  • Optimise the AWS costs further:
    • docs.stackstorm.com -> to be migrated from S3 to GH static pages (Migrate docs.stackstorm.com to GH static pages st2docs#1096) - needs a volunteer
    • api.stackstorm.com -> to be migrated from S3 to GH static pages (Migrate api.stackstorm.com to GH static pages st2apidocs#16) - needs a volunteer
    • rebuild and right-size the st2cid, remove LDAP, internal monitoring servers, etc
    • IDEA middle-ground:
      • run st2cicd release/control instance in Encore infra, still using the AWS APIs for requesting the short-lived VMs for e2e testing on-demand
      • Can save 80%+ of the AWS monthly costs
      • Less work is needed for st2cicd workflows refactoring, still using AWS API
• Long-Term:
  • Is it possible to get rid of AWS (almost) completely? (To be discussed in the next TSC Meeting)
  • Refactor and migrate the st2cicd release automation and e2e logic from the black AWS-hosted box with limited access
    • Utilize the GH Actions
    • Public transparent CI/CD/Release automation could be observed and verified by anyone
      • better security, fewer supply-chain risks
    • Helpful small e2e step example: @winem is working on running the st2-self-check script in GH Actions
    • More Ideas - to be discussed in the next TSC Meeting

Thanks, everyone for joining!
If you have any adjustments, additions, or notes about the meeting minutes, - please post more details in the comments below.

[arm4b]

New information from Slack Discussion that AWS FOSS credits expiry date is 31 July 2022, not 31 March 2022.