clarifying Mojaloop release contents and release criteria/procedures.

[tdaly61]

Request Summary:

We at the DA need to crisply define:

• What comprises a Mojaloop release
• How do new features get into a Mojaloop release is there a process ?
• Do we have written release criteria for deciding when a release is ready to be released ?
• Do we have release "gates" i.e. documented test criteria that the release must pass prior to being released
• How do we know something is actually "released"

Request Details:

Background : from the DA slack channel
Do we have , or can we have a definition of what comprises a release of Mojaloop and when and how something gets released. This has been a discussion Sam, Miguel and I have been having for a while now and it stems from my surprise a while back that the code in the Charts repo is being included in our releases yet we do seem to have a process for declaring that so and in fact as I write this the Charts repo still is labelled experimental even though we are pulling from it for Mojaloop 14.1 and soon 15.0. Now lest I make Sam nervous (again) my desire here is to agree on definitions and standards , processes that make the content and the methodology clear. ...thanks

• Deadline:
• Impact (Teams): <ideally if we get this right, no impact will ever be really noticed>
• Impact (Components):

Artifacts:

• Artifact to consider [@contributor]

Dependencies:

• Currently awaiting some text/proposals from Tom Daly so as to make the discussion more concrete and productive

Accountability:

• Owner:
• Raised By:

Decision(s):

• Approved By:

Details

• Actual decision made as a result of discussion

Follow-up:

• Actions to implement the decisions

[pedrosousabarreto]

@tdaly61 I really think we should clearly define what is Mojaloop Core and the other optional tools, scripts and other supporting software.
My take is that Mojaloop Core should be the core services, services that are required to run in production, like central-ledger and ml-api-adapter, etc, plus the helm charts. Everything else should be considered not core, ie, optional.

[MichaelJBRichards]

Product Council will define what's in the release from a functional perspective. Release Manager will be responsible for adding technical components - new releases, bug fixes etc.
Need guidelines for technical director
MdB: this should not be a matter for the DA. Who is it for?
JB: we need to make a proposal, then take it to TGB or Product Council for final approval.
TD: we need to have a gating process: what should the tests be?
MdB: this is not a DA thing, it's a product thing.
SK: what should the DA do in this context?
MdB: if product owners come to us and ask questions, we should respond.
MR: can the Product Council step up to the testing requirement?
PSB: we can help with the technical process, but we cannot define the tests.
MdB: if the Product Council can't own this, then Mojaloop has failed.
PSB: if the process isn't working, we could offer help
TD: whatever happens, we need a working release process. MdB, SK: we think it is working. There is a documented process...
SK: we should be able to work with the Product Council, but there are other things that they might not care about... DA can help with the quality standards. So this still a useful exercise.
TD: as product matures, it needs more documentation and process. Everyone agrees with this.
PSB: we should separate the responsibility for executing the process from the question of defining the process. Maybe the DA should offer help to the Product Council. We can define processes well.
JB: one of the main gaps: people find it hard to understand the functional content of a release. Why is this? shrug But we can make it better
PSB: we should meet with Paul Makin and look at defining the process.
SK: documenting the existing process would be a good start. SK working on this: wait until he produces the document.

Action: MR: write to PM and say: we think that the Product Council should have ownership, at least over the functional content of a release and the processes for defining it and testing it. Would he agree and, if so, would he appreciate the DA's help in defining this process?

TD: obviously, we're acting in harmony with our charter.
JB: might be a stretch to apply charter 2.1.b to release management... MdB agrees.
TD: on mature consideration, whenever we come up to a convening, there's lots of pressure to get a release out. We should define what the release cadence is and what drives the release. MR: this should be PI-based and there's bound to be pressure. TD: tension between that and the testing sequence.
SK: in general we do follow the PI cadence.

[tdaly61]

For DA meeting 23rd Aug:

Review of Sam's doc:

Sam has created a document https://docs.google.com/document/d/1qVuN5x5qqsG0E3QT0quwl1qmaXfd5Y2pxcRXa7pnAgc/edit and the DA said they would review and provide feedback

Here is Tom Daly's feedback for discussion at the DA meeting

• think we need a "crisp" statement to say that a Mojaloop release IS and IS comprised of ....

  • the diagram is useful and probably does actually answer the question of what the release "process is" . Should you start with the diagram.
  • be good to scale the text so it was readable from the diagram or simply number the components and detail accurately in text below
  • in the diagram , can you please explain setup/helm ?

• What is the release process (heading) ?

  • 1. currently identifies is some components of a release process , perhaps move this to an intro section

• Where do feature requests originate and get triaged ? Presumably this is the Product Council. If so should this be on the diagram or in the text as I have suggested describing accurately each of the diagrams phases.

• I think we should make the process more generic, I know you are using an example hence referring to a specific release but this example should support the release procedure not define it

• 1.4: re your question about existing implementations and new releases. This would be good to bring up in the meeting

• 1.17 is the QA env still moja1 or is some more generic wording required.

• acceptance criteria (gates) I think should be separately detailed (lets talk about that one)

• minimum review requirements should be listed and detailed e.g. at least 2 reviewers from different organisations might be something to consider in line with recent requirements or if this is not possible then an exception and approval from DA might be required highlighting that it had not happened. (this need not be onerous but it will provide for consistency)

2. proposed updates

   • 1.c I think mini-loop needs to be and is designed to go into the
     q/a pipeline. If not now then certainly for the future releases
     also yes either IaC needs to be vastly slimmed down
     and simplified and / or another way of testing releases
     with security enabled needs to be
     available (or both)
   • 2.4 & 2.5 Yes !!!!!

3. helm release contents

   • this is tricky because here we do need a list of components and at the same time it needs to be very specific yet we know that this is not the same list as the reference architecture. Maybe the best thing to do is leave it mostly as you have it for now.
   • 3.3d Payment Manager inclusion I thought was an open DA issue , so are you not jumping the gun? Also given what we are learning from the vNext process , we obviously need much more community consultation and some review meetings before TGB or Prod Council or DA adopt this into Mojaloop right ? I mean my understanding is that last time this was looked at there were quality issues (again is my understanding correct here?) Just trying to use the same standard given what we have been learning recently about product adoption process.

4. Platform : this needs separate discussion , it can probably be removed from the release process discussion for reasons previously outlined. 


5. Other things I noticed:

   • I don't see any mention of the release manager. Perhaps you can add a small section outlining the roles and responsibilities of the DA, Prod Council, Release Manager , contributors etc. Perhaps this could go as an adjunct to the diagram.

regards
Tom

[elnyry-sam-k]

[Posting the draft content here for access, will post as markdown on a draft PR shortly: Note: Pending changes discussed in the last DA meeting]

Mojaloop Releases

The Mojaloop Release process follows a community led approach, specifically technical work-stream led, in collaboration and consultation with the Product Council and the Design Authority (DA).

The DA frames the policies, guidelines and technical criteria for the releases (such as quality metrics) while Product defines the feature and product requirements where such aspects are involved. This is executed by the relevant workstream(s).

1. Mojaloop Release process

Community event, workstreams, features, release quality, testing, checklist, release candidate, example epic, Release

The Mojaloop Release process follows a

• community led approach,
• specifically technical work-stream led,
• in collaboration and consultation with the Product Council and
• the Design Authority (DA)

Criteria, guidelines:
The DA frames policies, guidelines and technical criteria for the releases while Product defines the feature and product requirements
Example releases notes, criteria for v15.0.0, v15.1.0

Once the release is made, it is tested as well and daily cron jobs run to test, until the next release is made

Current tasks and acceptance criteria for Mojaloop (helm) releases:

Example story: mojaloop/project#3122

• Initial PR with changes for next release drafted and reviewed
• Capture the latest k8s version to be used (for RC and release moving forward) and upgrade environment to use it.
• Deploy, validate, fix issues, redeploy
• With stable deployment test Golden path
• Remove out of scoped services or services that do not meet release criteria
• Validate PISP test provisioning and test collections (TTK)
• Identify issues with QA Scripts if any - fix them and retest
• QA for bugs, regressions, log them
• Fix bugs logged if critical
• Deploy, validate on dev environment (AWS Moja2)
• Validate with TTK GP collection
• TTK test-cases repo v15.0.0 is published
• Release notes for helm v15.0.0 drafted
• Guidance for migration from previous release (may have to be a separate story if sufficiently big / complex)
• Update release notes with Upgrade Strategy Guide link
• Helm v15.0.0 released
• Deployment of released Helm v15.0.0 on dev environment (moja2)
• Helm v15.0.0 is deployed on Moja2 successfully
• Regression testing on Moja2 using TTK collections
• GP collection
• Core Bulk collection
• Third-party collection
• SDK Bulk collection
• Validation using CGS Golden path regression tests etc on Moja2
• Deployment of released Helm v15.0.0 on QA envt (moja1)
• Helm v15.0.0 is deployed on Moja1 successfully
• Validation using Golden path regression tests etc on Moja1
• GP collection
• Core Bulk collection
• Third-party collection
• SDK Bulk collection
• Validation using CGS Golden path regression tests etc on Moja1
• Validate that we can upgrade from the previous stable release, and this is also to pick up any "gotchas" that may need to be addressed as part of the release, or perhaps release notes need to be updated assuming that it would be up to the upgrader to deal with.

To-do:
Add acceptance criteria, ref: release story

2. Mojaloop Release process - proposed updates:

Propose release schedule and timelines

• Example: Feature freeze for a (major) release before six weeks prior to the next PI kick-off (or community event)
• Freeze bug fixes to be included four weeks prior to release date
• RC to be validated by at least one downstream implementer such Mini-loop or IaC or another implementer
• Release to move ahead on time if there are no high or medium priority bugs open in RC and validated on a dev environment and one downstream team / implementer
• Streamline release numbers between various components of the Mojaloop platform, such as the Finance Portal
• Document tests to be run for release validation and any other validation steps that need to be performed (example: validating release notes)
• Include performance numbers with details on the setup on which the baselining was performed
• Resource usage: capture resource footprint for a base release
• Document support mechanisms for Mojaloop releases

3. Mojaloop helm release contents

Mojaloop services that support core functionality for the platform and other key services supporting them, along with tools needed for testing such as simulators

Services needed for:

• Core functionality with configurability:
  • Account lookup
    • Account lookup admin
    • Oracles and
    • ALS
  • Quoting
    • To support persistent / pass-through modes: configurable
  • Transfers (clearing)
  • Support for on-us transfers, configurability
  • Settlement
  • Transaction requests (request-to-pay)
  • 3PPI services
  • API layer (for parties, quotes, transfers, trx-req, )
  • Notifications: ml-api-adapter
  • currency-conversion when released
• Extended functionality:
  • Central event processor
  • Email-notifier (prior to v15)
• Auditing
• Supporting services & Tools for testing:
  • ML TTK
  • ML simulator
  • Sdk-scheme-adapters,
  • Payment manager instances [Test dependency]
• Third-party scheme adapters
• Participant support
  • Simply, easily usable tools for adopters (example: sdk-scheme-adapter)
  • Onboarding functionality and support

4. Mojaloop Platform

1. Primary mojaloop (helm) release and config that supports

• Core clearing engine including support for Bulk
• Quoting
• Account lookup and supporting components
• Settlement engine
• API layer
• Support for request-to-pay (transaction requests)
• Ref: Mojaloop helm release (example: v15.1.0)

2. PISP / 3PPI functionality
3. API Gateway(s)

• Provide a secure API layer
• Provide Ingress, Egress, IP filtering, firewalls
• Support security mechanisms: JWS, mTLS
• Reference: WSO2

4. Security components:

• HSM where relevant / used
• Identity & access management
• Cert management
• Connection management

5. Finance Portal, Reporting

• Portals for Hub Operations, Business Operations teams
• Portals, capabilities for Technical Operations teams
• Ref: FP v3 based on Business Operations Framework

6. Monitoring Support:

• Operational support and tracing (example: EFK, Prometheus, Grafana, Loki)
• IaC uses Grafana, Prometheus and Loki

7. Use IaC as reference, example: https://github.com/mojaloop/iac-aws-platform/releases/tag/v4.1.0

[bushjames]

Update: PR to be opened soon.