Michael Giacometti - 25 April 2018
It’s no secret that the digital revolution is quickly changing the way businesses and customers interact with each other. Like Blockbuster, companies that don’t understand the evolving needs and tastes of their customers will die, while companies like Netflix that fail fast, quickly adopt technology, and evolve, will thrive.
If you want to see this in action now, look at Domino’s Pizza vs. its competitors. The pace of technological change and how Domino’s understands its customer base is very different compared to other pizza chains. One small misstep however, and the bottom can fall out. Several techniques have been developed to ensure cohesion between business and customer — one of them is net promoter score (NPS).
NPS is a very important indicator of customer satisfaction for your organization or business. Generally, it’s associated with changes to an overall customer satisfaction initiative where any rise in NPS should relate to more revenue and positive brand recognition, while a drop in NPS could have your organization going viral for all the wrong reasons. It’s so important, in fact, that a lot of compensation is tied to it — maybe even yours. While NPS isn’t the only indicator of customer satisfaction, it is the most well-known. In addition, customers have access to an expanding number of channels not even thought of just 10 years ago. Companies like G2 Crowd, and apps like the iTunes App Store and Google Store, etc., can present a positive or negative perception to thousands of users within minutes of an otherwise perfect release. These perceptions can cause great success or irreparable damage to a company or brand.
What does this mean if you’re a quality professional? Defects, test automation, test coverage, and quality gates are still very important, but you need to shift up. The purpose behind shift up is very simple: Look up, look around, and see how the work you’re doing as a quality professional is impacting the people using your products, applications, or services. Shifting up goes beyond traditional user acceptance testing. It’s about understanding, advocating for, and thinking of yourself as a customer. Shifting up is the real way we avoid failure in production, and to ensure that the software, apps, and services we build truly match up to customer and business expectations.
No longer are we looking at moving up the IT chain toward the project management office (PMO) with shift left as success. That’s expected. No longer are we looking to get to production faster by shifting right. That’s status quo if you want to be competitive in the marketplace. You need to ensure that the customer is satisfied with the product or service your company is putting out into the market. Think about it this way: Even the most flawlessly built and functioning applications designed according to the best technical and business requirements can still negatively affect revenue, branding, and competitive position. Or, put another way, all the positive metrics and key performance indicators (KPIs) used to measure the success of shift left and shift right in a release are not a complete gauge of success in production.
Let me reiterate. All the positive metrics and KPIs used to measure the success of shift left and shift right in a release are not a complete gauge of success in production.
Shift up is the concept of taking a diverse set of users with individual needs, wants, and experiences, and ensuring your application can meet all their needs quickly, while respecting processes and timelines defined in your shift-left and shift-right processes. To experience success in shift up, you need to leverage tools and techniques that weren’t even available when UI/UX and usability testing were just getting a foothold in the market. You need artificial intelligence, real-time user monitoring, analytics, and a way to ensure interoperability across the digital spectrum.
To understand how to measure the uniqueness of a customer and how they use your product, we need to define two essential concepts for creating, acquiring, and leveraging the analytics necessary to shift up.
Ethnography is defined as the scientific study of people and cultures. But from a shift-up perspective, I like the definition given by ExperienceUX: “Ethnography is a study through direct observation of users in their natural environment rather than in a lab. The objective of this type of research is to gain insights into how users interact with things in their natural environment.” In a QA setting, that means we need to be concerned with what the user is feeling. For example, are we looking at someone new to technology who’s frustrated at a supermarket self-scan machine? Are there distractions around? Was the user just fired from a job? Can any of these factors affect the way an application is used?
Psychometric analysis, according to the Australian Institute of Psychometric Coaching, is the standard and scientific study of mental capabilities and behaviors. In short, metrics that ensure your application is appropriate for the capabilities and behaviors of your target audience: Children’s educational toys, a web service catering to senior citizens, etc.
Hopefully by this point, the implications of not looking at your application through an ethnographic or psychometric lens are now becoming a little clearer. Think of common issues encountered by your technical support or customer success groups. Now, think about them from an economic or branding aspect. Are people no longer using your application because of true performance or infrastructure problems or because the user can’t go as fast through the system as they used to? As a result, are they more attracted to your competitor’s service? Is a message displaying that a lane open or closed on your automated point-of-sale machine considered a bug because the user is looking at the common color scheme as opposed to the words “open” or “closed?” Is this helping your business goal of reducing cashier headcount in favor of automated systems? Finally, if you are a franchiser and notice a difference in quality and revenue from two similar locations in different parts of the country, is it a design flaw or a management problem, or are external factors like the local education system and cultural differences between locations to blame? How do you know? How do you design for this? More important, how do you test for it?
Traditionally, organizations have used usability, UI/UX, and field testing to uncover these problems. Tools exercising those concepts are good and do serve a purpose. But while those tools may tell you in UI/UX testing if localization is correct or ir the GUI is aesthetically pleasing, they don’t generally account for the user’s emotions or behaviors. Field testing, whether in beta or in the wild (as another company likes to put it), could be difficult to organize, logistically difficult to execute, and may not be completely effective in protecting your IP.
How can you ensure proper QA without having to resort to resource-, time-intensive, and expensive options? With an intelligent automated testing solution that use artificial intelligence, machine learning, continuous monitoring, and analytics.
Newly added, real-time user monitoring functionality in Eggplant AI — the brain behind our Digital Automation Intelligence Suite — gives you all the capabilities you need to successfully shift up without negatively impacting existing processes. Eggplant AI is built on the foundation of Eggplant Functional’s fusion engine facilitates interactions with APIs, internet of things (IoT) devices, databases, and open source testing tools to help teams execute automated test cases without having to re-engineer your entire suite. Eggplant Functional tests an application through the eyes of the user — exactly as someone would see and interact with it — without any complicated scripting.
One of our clients in the telecommunications industry experienced this firsthand with a defect in a customer ordering process that only Eggplant AI was able to detect. To sign up for or transfer service, users would fill out a web form. But if the user went back a screen to edit data and clicked “continue” to proceed, any information previously captured was lost, an error screen appeared, and as a result, users were abandoning the site. The designers didn’t consider that a back button on a browser would be more useful and intuitive rather than the custom-designed back button. Eggplant AI incorporates data on user behaviors and creates test cases to find bugs that even the most talented QA professionals may miss.
With Eggplant AI, you can model your application, connect actions to snippets, and dynamically create and execute test cases. The solution leverages risk-based testing, testing coverage analysis, bugs detected in earlier releases — and now with the addition of real-time user monitoring, specific ways your customer is using your product in production to ensure that the quality of that release meets business, technical, and customer standards. Psychometric KPIs can be built into the system, and Eggplant AI comes out of the box with auto-ethnography metrics and KPI capabilities.
Your organization lives and dies by how it’s perceived by your customers, as well as how they use your products, applications, and services. Each customer is unique, with different needs, experiences, and behaviors. What may work in one town, country, even continent, may not work in another. The future of QA and how it can provide a real impact to your business requires how well you can shift up. Remember, the customer — every one of them — is always right.