In a recent Eggplant survey on retail trends it was apparent that companies are facing some common challenges in delivering a high-quality digital experience. Nearly every retailer we surveyed recognizes the importance of evaluating how the user experience impacts business outcomes, yet 30% have a drop off rate of 50% or more on online properties.
"The quality of your company’s software has a direct impact on the quality of your company’s financial results. You know it. Management knows it. And the importance of quality will only continue to grow with the need for 24x7 operations, high availability requirements, aggressive service-level agreements, and the need to roll out innovative new web-based services."
This was the first paragraph of a paper I wrote in 2005 about how to build your organization around a Testing Center of Excellence.
15 years on, we are still struggling with these concepts. The focus has turned towards project outcomes rather than business outcomes. Reasons include faster release cycles, more complex technology, and more technically astute end-users, with the result that QA lost sight of who was really using their applications.
With the Shift-Up series thus far, we have explored the importance of testing and thinking as a customer. The basic premise is that we need to add another dimension to Quality Assurance other than Shift-Left and Shift-Right. This new dimension focuses on how your customer is actually using your application and if the intersection of your application, customer behavior, and your company’s business objectives all align.
No matter what industry you’re in, providing an exceptional digital experience to your customers is paramount. It’s particularly tricky in financial services, as more and more users ditch physical branch locations for online banking. By 2019 in the UK, mobile banking is expected to overtake desktop as the preferred channel. And for two-thirds of Americans, a recent survey found that online and mobile banking represent their primary banking channels.
To keep up with DevOps, testing and QA teams typically adopt a shift-up approach to move quality further up the software development lifecycle. The goal is to complete system testing, integration testing, and user acceptance testing (UAT) to ensure a bug-free release. While product quality has a direct correlation to increased revenue and positive business outcomes, this isn’t enough in the 21st-century marketplace. QA’s job isn’t just to de-risk applications by finding defects earlier but to help de-risk business strategy and potential problems with your user base by reporting customer experience defects.
Testing is critical for organizations like NASA, the US Army, Northrop Grumman, BAE Systems, Lockheed Martin, MBDA, the UK’s Ministry of Defense and the Metropolitan and Scottish Police, where lives are on the line. As we've worked with customers like these over many years, we've noticed how much more testing is than just making sure the system works — it’s about ensuring we test for mission success and continuously optimize mission outcomes. Whether you're designing systems for command and control (C2); to provide support for complex police operations, such as hostage negotiations; or for shooting down an enemy missile, you should plan your testing and monitoring strategy to continuously test against the desired mission outcomes.
Some of my customers are trying to design an automated script to perform specific workflows with a predicted outcome. Unfortunately, the automated workflow they want to execute has many variations in their environment, and they’re having trouble creating a dynamic, automated script that handles environment deviation.
Quality assurance (QA) used to be a compliance activity. You were releasing a product and needed to test it and stamp it “approved.” QA was about testing that the code worked. You might manually test the code. You might have even tried some automation — coding a set of test scripts that would try to capture regressions or errors that you had eradicated in the past, but which somehow crept back in. All in all, you were reasonably satisfied that you achieved a level of test coverage that met your goals. Then, you put your code into production and crossed your fingers that nothing went wrong. And if it did, you tried to fix it as quickly as humanly possible.
Note: Test engineer Reena Kuni and software engineer Bekki Freeman also contributed to this blog.
On the Eggplant Functional team, the relationship between Dev and QA is very collaborative. We work closely together, use our Slack channel to organize regular walk breaks together, and frequently talk about ways to increase product quality.