by Eggplant, on Jan 22, 2020 2:50:47 PM
Overview
Headquartered in Tokyo, Japan, NEC Personal Computers, Ltd., manufactures 24 different PCs for consumers and businesses. The products have a 64-bit operating system and two kinds of preinstalled applications running on the OS; one produced by NEC Personal Computers and the other by independent software vendors (ISVs).
NEC Personal Computers’ Software Integration Department is tasked with analyzing any technology performance, integration or compatibility issues and finding the root cause of these problems before shipping decisions are made. To do this, the company had been employing a mix of manual testing and limited automation, however, the approach was extremely resource-intensive and also failed to account for how customers use the technology. As a result, NEC Personal Computers began a search for a modern test automation solution that also tests how users interact with the technology.
A Shift to automated user experience testing
The Software Integration Department’s mission is to deliver products to NEC Personal Computers’ customers without any OS problems. Led by Ichiro Mori, PM Group Manager, the team analyzes any defects found by the development or test departments and reproduces and fixes these issues prior to shipping. Mori states, “The problems include fatal errors that have an occurrence rate of less than .1 percent. Therefore, it’s necessary to repeat the same operation more than 1,000 times to reproduce a rare and fatal problem, resulting in an investigation of the cause taking a significant amount of employee hours.”
NEC Personal Computers was able to automate some of this testing via its proprietary testing tool, but the technology lacked the ability to handle the graphical user interface (GUI) and use keyboard inputs. Mori elaborates, “For example, if any user input like signin after a restart causes an error, the tool itself cannot reproduce it, resulting in the need for manual intervention and many employee hours of reproduction tests.”
Mori knew there was a better approach that could automate this critical user experience testing while also maximizing human testers’ time and began searching for a test automation partner to deliver these benefits.
More Comprehensive Testing with Fewer Resources
NEC Personal Computers evaluated multiple vendors before selecting Eggplant. The company was particularly impressed with Eggplant’s ease of use, its ability to test across a range of devices and applications and its intelligent image recognition technology.
“When capturing which button the tester pressed and replaying it, the method of using the coordinates on the screen does not allow the same script to be used between devices. However, by using intelligent image recognition it can locate the same icon to click, and by using text recognition it can search for a string to find the same button to click. Therefore, both recognition methods can not only use the same script between devices with a different resolution but can also determine whether the test result has displayed a successful or unsuccessful message on the screen.” - Ichiro Mori, PM Group ManagerHigh-Quality Technical Support Drives Immediate ROI
Mori was also very appreciative of the responsive and consultative Eggplant support the team received during implementation. He states, “Their technical support made it possible to automate tests even though we had no tester familiar with Eggplant. When he did not know how to use a command, he emailed Eggplant’s technical support and got an answer right away."
As a result, NEC Personal Computers was able to quickly begin realizing returns on its Eggplant investment. For example, the company compared the reproduction test of a rare OS error before and after automation with impressive results. Mori explains, “Before automation, 1,000 trials required 47 employee hours, while this reduced to 21 hours of the testing team’s time after automation. A shipping decision in an OS error investigation required 12,000 trials, which means that 553 employee hours for the manual reproduction test can reduce to just 32—resulting in a reduction rate of one-seventeenth.” This reduction in manpower is equivalent to a one-year Eggplant license and maintenance fee. As Mori puts it, “This means that our Eggplant investment is covered through a single reproduction test of only one OS error.”
NEC Personal Computers expects to reap further efficiency benefits from Eggplant in the coming months. The company plans to expand automation by customizing its test scripts for similar OS errors, and also using Eggplant to automate different kinds of production tests, including OS recovery operations which are likely to cause problems.
“I take action against an average of 120 OS errors annually. Of these, approximately 12 rare errors are detected in a continuous restart test. If reproduction tests for all of these could be automated, I think our Eggplant ROI would be excellent.” - Ichiro Mori, PM Group Manager.
