This documentation describes approaches to meeting requirements around governance, quality checks, and best practices for technical staff before, during, and after open-source release of software.
The following requirements, and their implied goals, are directly and indirectly inspired by NIST policies and their requirements mentioned in the appendix.
The developers provide rationale below for requirements and how to meet their criteria. Those details are primarily focused on technical leads and developers. Other stakeholders may not be familiar with the methodologies and their strategic benefit to software development projects like Metaschema. We describe the methodologies and their benefits in this section.
Continuous builds and testing are part of the continuous integration methodology of software development. Continuous integration involves developers frequently merging code changes into a central code repository, automatically building software from changed code, testing the code and/or the resulting software. This methodology creates a fast feedback loop to test changes and validate the desired behavior of the software from code changes. Continuous builds and testing increase developer productivity and reduce the risk of infrequently aggregating large changes in the code with a higher likelihood of releasing software with undesired behavior. Reducing the likelihood of degradations or failures as early as possible reduces the reputational risk of end users using poor quality software after the fact.
This project uses GitHub Actions (GHA) as a Continuous Integration and Continuous Deployment (CI/CD) environment to build, test, and release code. The CI is used for all commits against the develop and main branches, and for all Pull Requests. CD is used for all tagged releases. Specialized GHA workflows are used to drive this automation.
References:
- NIST Special Publication 800-204C: Implementation of DevSecOps for a Microservices-based Application with Service Mesh
- USGS Testing and Automation Guidance
Developers can formally or informally identify logically discrete portions of the code during the development process as self-contained units. These units have distinct purposes or functionality that contributes to the complete software as whole. Discretely testing the expected functionality, inputs, and outputs of each unit, not just the resulting software as a whole, has important benefits. It can detect bugs in newly created or modified units of code more efficiently and effectively than debugging all the software at once. Operating all the software to contextually exercise specific parts of its code to locate or fix a bug can consume significant resources. Unit testing reduces the risk of releasing software with erroneous or undesired behavior. It ensures errors are found quickly and efficiently as early as possible in the development cycle.
This project uses the Maven Surefire plugin to automatically execute JUnit 5 unit tests during all CI builds.
References:
Developers can use write tests that exercise discrete units of code and check functionality. Code coverage, and code coverage analysis tools, analyze how tests are structured and what code paths are exercised. This analysis can measure how many lines of code will be executed by one or more tests. If a line of code is not tested, it is not "covered." Tests measure the correctness of code's behavior, and code coverage measures the completeness and thoroughness of the tests.
This project uses Jacoco and the Maven Jacoco plugin to automate code coverage analysis during builds.
The project strives for a 60% test code coverage ratio to ensure adequate test coverage. Work in this area is focused on enhancing unit testing of critical code paths to achieve this goal and to maximize unit testing benefits.
References:
- NIST Cybersecurity White Paper: Combinatorial Coverage Difference Measurement
- USGS Testing and Automation Guidance
As developers write code incrementally, it is not uncommon that undesirable (e.g. insecure code, underperforming) behavior of the code is not immediately apparent to the developer as they add or modify it. There are many factors a developer must continuously track across the codebase to prevent any undesirable behavior caused by how the code is written. Static analyzers are tools that augment the development process by analyzing the code "statically" (i.e. only reading the code as written and not executing it as bundled, finalized software). These tools perform analysis to spot potentially undesired behavior and report it to the developer. Often, these tools will provide recommendations on how to rewrite the code to potentially mitigate the issue. These tools help increase the ongoing awareness and inside into undesired behaviors while reducing the time and cognitive load needed for one or more developers to review the code.
Static analysis in this project is provided by multiple solutions.
- PMD is used to identify common programming errors. PMD is configured to enforce Java programming best practices; to look for flaws in the code style, design, and documentation; and to identify error prone code and potential performance problems.
- SpotBugs identifies potential bugs using over 400 bug patterns. SpotBugs performs analysis during builds using the Maven SpotBugs plugin.
- Checkstyle to enforce a coding style. Checkstyle is run during builds using the Maven checkstyle plugin.
References:
- NIST Computer Security Resource Center Glossary: What is a static code analyzer?
- USGS Testing and Automation Guidance
As developers write code incrementally, it is common that undesirable behavior (e.g. not secure, not performant) is unnoticed until software is executed. In those cases it is not always apparent to the developer as they write or execute the software. Dynamic analysis tools, unlike static analysis tools, run the software in various conditions and with various data inputs to test for undesirable behavior. Dynamic analysis will find bugs that static analysis will often not find and vice versa, so performing both analyses on the code base has benefits. Dynamic analysis tools can recommend configuration and deployment changes to staff to potentially mitigate the undesired behavior. Using such a tool can reduce risk and improve the software's behavior with less developer effort. Detecting and fixing the behavior before an end user observes it also reduces reputational risk for the software.
This project does not use a dynamic analysis solution at this time.
References:
- NIST Computer Security Resource Center Glossary: What is a dynamic code analyzer?
- USGS Testing and Automation Guidance
Modern software must often provide complex functionality combining a variety of media, protocols, and data exchange mechanisms. It is often too resource intensive to implement the prerequisite functionality for these capabilities in before completing the features of the software the the end user wants. Developers use third-party libraries and tools to efficiently develop software and not expend resources on common prerequisites. Supply chain analysis identifies these third-party tools and software libraries and subsequently flag potentially undesired behavior from code changes, discovered security vulnerabilities, and/or performance degradation. When developers perform this analysis, they significantly reduce the risk of supply chain attacks. Not doing any supply chain analysis significantly increases the risk and likelihood of being an unknowing target of this increasingly popular attack vector.
Supply chain analysis is supported in this project by GitHub Dependabot. Dependabot is configured to automatically identify vulnerable and out-of-date Maven dependencies and GitHub actions plugins, and create pull requests to update to newer versions against the develop branch.
References:
It is good practice for developers to document how to change code or build the software from its code. This documentation facilitates knowledge transfer and limits the risk of little or no maintenance for the software after replacement or reduction of staff. Additionally, it is good for developers to provide documentation for end users to onboard them and explain common uses of the software. This documentation benefits the end user by increasing familiarity with the software and reducing ambiguity in how the software operates or the resulting impact on their work.
This aspect is supported in multiple ways.
- Building instructions are included in the README.md.
- Code documenation is provided using Javadocs in the project website. This is accessable for each module under "Project Reports".
- Some modules provide basic usage examples, e.g. Metaschema XML.
Improving documentation is a current work focus for the project.
It is good practice for one developer to add or modify code for software and have an independent developer review the change for correctness and quality. A code review by a developer with "a fresh set of eyes" will use that perspective to analyze the code for a variety of factors, provide feedback, and have the developer who authored the code potentially make changes. Code review reduces the risk of releasing software with undesirable behavior or incorrectly addressing the business case of a particular feature or business case overall. Like other methods above, it similarly increases the ease of knowledge transfer and limits the institutional risk of one singular developer being the sole staff member understanding the software.
All changes to the project's source are made through pull requests, with required reviews. This ensures that all changes are reviewed by another set of eyes. The GitHub CODEOWNERS configuration is used to direct reviews to the people responsible for the changed code.