Developing CI/CD pipelines is a hard task. They need to be resilient, easy to use and extendable pieces of software that are run by another system (AKA Spinnaker, Jenkins, Tekton, ArgoCD, Waypoint, etc...)
When developing, extending or testing pipelines on any 3rd party system (not the developers computer or a VM) there are a few key problems that come up.
- Local development can be very hard, due to setup or complexity of the downstream systems.
- Stages/Pipelines are not testable, most CI/CD systems don't have a built in testing framework.
- Collaboration and code reusability are flawed, Sharing & Testing code requires jumping through hoops or stacking functionality in one centralized location.
- Reduce time to release/deploy features
- Testable pipelines - promote a culture of testing and resilience.
- Scaling & Collaboration through code reuse.
- Improve Developer Experience - Reduce friction when developing pipelines
- Pipeline development best practices - create a shared mental model for pipeline development (AKA Framework)
- Open source the frameworks and tooling!
- Support For Other CI/CD solutions (
Spinnakeris the only backend we support, more on this later)
*Out of scope items may be made part of the project's scope in the future
- Simon Chammah (Project Owner)
- Eyal Mor (Lead Developer)
- Sergey Liberman (Developer)
- Daniel Kirillov (Developer)
This is a deep dive into specifics, and may carry many technology specific terms.
The document will try to provide a link to each new term as it is encountered.
The space of CI/CD is saturated with solutions. Some are cloud native specific, while others are more general purpose.
Examples (CI/CD):
All these systems solve CI/CD, by running code in a managed service based on an external event. Each of the example systems solve event handling, system management and much more in very different ways.
If we treat a pipeline like a Web API endpoint, for a web request, the pipeline is just a function/method for which the developer doesn't control the means of execution (e.g.: Step-Functions, Lambda).
However, unlike developing Web APIs, the dev, debug & test cycle for CI/CD gets complicated.
Lack of built in debugging & testing tools in these CI/CD solutions hurdles the release of features in a timely manner.
As it stands, there is no formal way to test pipelines, some solutions in the wild are either not available in Autodesk or not ready at all.
The need for testing CI/CD solutions is widely known but hasn't been adequately implemented so far, expand for more details.
CI/CD solutions struggle with testing and best practices.
This is emphasized by open Github and available tooling.
Spinnaker doesn't have open issues on the matter (spinnaker GH issues).
The next pain point is opinionated, but has a massive affect on developer productivity.
Software engineers prefer controlling their dev environment. Some develop in a Docker container, some install software on their personal machine, and some will develop in the Cloud (on a VM).
Others developers may mix and match.
When developing pipelines the only option is to use the Web-UI provided by the CI/CD system or use a specific CLI that only covers creating and triggering pipelines.
As it stands cli tools (specifically spin-cli) need to be extended to support the same level of UX/DX that the web-uis provide. But web-uis are clunky and hard to automate tests around the GUI.
Shore is a framework to make developing, testing and deploying pipelines.
(Marketing phrase ahead, you have been warned!) shore allows developers to build pipelines that deploy software to safe shores.
The framework consists of:
shore-core- The orchestration layershore-cli- CLI tool that runs on developers machine.shore standard libraries- Standard Reusable Libraries.
- Interface with rendering engines (Jsonnet, CUELang, Dhall-Lang, HCL2)
- Interface with CI/CD backend (e.g.
Spinnaker) - Shareable and reusable layer for other projects to consume.
- Load 3rd party libraries (if the rendering engine doesn't support it, e.g.
Jsonnet,HCL) - Provide facilities for Unit/E2E testing
shore-core is the programable layer for shore.
It can be used in other projects and should be easy to interface with.
It provides the tooling necessary to build, execute & test pipelines.
- Render pipelines in the terminal
- Save pipelines to the chosen backend
- Execute pipelines with specific arguments.
- E2E testing of pipelines.
- Unit testing of custom rendering functionality.
shore-cli is the user interface for any shore project (backed by shore-core).
The CLI serves 2 purposes:
- An easy to work with tool for developing
shoreprojects - A
shore-corereference implementation for other project to consumeshore-coreeasily.
The CLI coupled with standard libraries should also provide a mental model on how shore projects are structured.
- Abstract boilerplate from developers.
- Promote structures for reusable, testable pipelines.
Standard libraries are a great way to get projects of the ground quickly.
These libraries are corner stones of any technical solution (e.g. glibc, Python STDLIB)
Libraries provide common useful functionality that is usually necessary for many projects.
Some specific examples for common boilerplate code in Spinnaker pipeline development:
PipelineStage(WaitStage,JobStage,DeployStage)NestedPipelineParallelStage
Some of these are feature specific data structures, some are nice to have functions.
This user is developing a reusable standard deployment pipeline. Examples of reusable pipelines: EC2, ECS, Kubernetes, Lambda, CDN - FE (Cloudfront + S3).
I want to be able to develop, debug, test and maintain pipelines easily.
I would like to mock up ideas and validate them out quickly.
Testing is very important for me and my team. This pipeline could be the most essential piece of software for any deployment in our company. We must have a way to formally validate it after every change (bug fix or new feature).
Our pipeline may be very complex. If possible, we would like to have mostly business logic code in our main code repository.
This user is developing a pipeline for a specific use case for their team. The standard pipelines don't meet their needs, and changing their deployment process isn't a short term option. This developer must deliver this pipeline as quickly as possible.
I need to create a deployment pipeline for my team.
This pipeline must be completed in 1 week.
I can't migrate migrate the dev team to the company's standard deployment at the moment, as we need to release features to customers, and the team doesn't have time to plan out a migration to standard tooling.
As an ad-hoc solution, we would like to have this midterm pipeline to support regionalized deployments via the CPN CD infra we already have.
The pipeline will deploy the whole tech stack together. Lambdas, EC2, Infra and Step Functions.
If it is possible, I would like to reuse the existing tested code as much as possible, as I don't have time to test or validate the pipeline, it just has to work sufficiently well.
We do plan to move to standard tooling in the future, but customer features are currently more important.
This developer is building a deployment platform for their company (surprising!)
The API interface for platform consists of a bunch of files that represent the infra/config as code in a VCS repository.
I am building a reusable deployment platform for my company.
We plan on deploying any/all AWS/GCP/Azure components our customer base may need.
Because of the complexity of this project, we will be using a custom Web API to manage our customers specific product needs.
I would like to reuse whatever pipelines and/or tools that were already developed in this space.
This platform needs to be 100% formally validated and have regression tests.
We prefer not to use a CLI, as we are writing an API, we would like our formal coding tools to validate the APIs we are consuming from an Interface.
This developer is tired of Jsonnet.
I would like a way to change the JSON rendering language.
It has no formal types, making our developer experience horrible. It's slow and unbearable.
Our project has outgrown the language and it's time to switch.
We would like to have a way to switch runtime languages. Our team has the capacity to implement the rendering layer, but not the backend.
The tooling must allow A/B testing, validating we are 100% backwards compatible and haven't missed a single feature while re-writing our project.