Jaspar Casey - 25 May 2021
Over the last few years, the Department of Defense has released a series of initiatives aimed at modernizing its digital strategy. These documents form a strategic roadmap for releasing higher-quality software in shorter cycles. Starting in July 2019 with the Digital Modernization Strategy, the DoD has released several key instructions to guide this new way of working:
- DoD Enterprise DevSecOps Initiative
- DoD Instruction 5000.87: Operation of the Software Acquisition Pathway
- DoD Instruction 5000.89: Test and Evaluation
These initiatives were motivated by the fact that the DoD was using outdated software development methods like Waterfall, with software delivery “every 3 to 10 years.” With testing routinely cited by a majority of developers as causing delays, we wanted to highlight the role of test automation in fulfilling the DoD’s vision for a full embrace of DevSecOps. In particular, we will look at how test automation is necessary for compliance with DoDI 5000.87 and 5000.89.
It’s not just about compliance
For understandable reasons, defense organizations tend to have rigid hierarchies and strict rules governing conduct. However, these aren’t just arbitrary structures aimed at creating compliance; they are key to delivering excellence. The approach to software delivery in these contexts is no different.
Nicolas Chaillan, the US Air Force’s Chief Software Officer, has been instrumental in putting DevSecOps into practice across the Air Force’s software ecosystem. Chaillan says that compared to their previous way of working, for every 5 years of planned development time, they are already “saving about 18 months of program time for every program moving to DevSecOps.” As of July 2020, the adoption of DevSecOps methods and tools had “saved 100 years of program time” in just one year.
The role of testing
The DoD formalizes the importance of testing in the software development life cycle. According to DoD Instruction 5000.87, “software development testing, government developmental testing, system safety assessment, security certification, and operational test and evaluation will be integrated, streamlined, and automated to the maximum extent practicable to accelerate delivery timelines based on early and iterative risk assessments.” In other words, every aspect of testing that can be automated should be automated.
Additionally, the DoD’s software pathway policy “includes a requirement to create a test strategy.” According to DoDI 5000.89, “the test strategy for software intensive systems should include … how automated testing, test tools, and system telemetry will support the product throughout its life cycle.”
Development teams also need to “verify that the system will meet both functional and non-functional performance requirements.” This means that performance and load testing must be performed so that there is a good understanding of how the software itself, as well as connected systems, will perform in the heat of battle.
Testing embedded software
These guidelines don’t just apply to standalone applications; embedded software and firmware needs to be tested based on the above criteria as well as “in the context of the hardware with which it will eventually be integrated.” This amplifies the importance of being able to test the behavior of software in specific hardware contexts, and means that test automation should be part of your end-to-end testing strategy.
DoDI 5000.89 also stipulates that tests should consider “model-based environments, digital twins, and simulations, as well as plans for tests on a production-representative system.”
Because of the highly secure nature of many of these hardware and software environments, it is vital that any testing solution is non-invasive. There are two relevant aspects of being non-invasive: one is that tests can be performed at interface level without access to underlying code; the second is that it can be used to test systems that it isn’t installed on itself.
Excellence at the speed of DevSecOps
It’s important to keep in mind the thinking behind the initiative of shifting to DevOps and DevSecOps. Not only does the quality of releases improve, but the speed of delivery as well. In the context of national defense, this speed aspect is vital, because creating hoops to jump through could disadvantage national interests. As Chaillan says, “keep in mind that if we do have a [manual] gate, [we need to ask if] it really is necessary for actual safety — for actual cyber security — and not just checking the box and pure compliance reasons.”
Test automation of software, no matter where it sits in the defense technology stack, is vital for ensuring excellence both in terms of overall quality and in speed of delivery.