Deep Learning, Testing, and Bug-hunting

By Antony Edwards | 3/15/17

testing-and-bugs-450x300.png

Over the past few weeks I've spent a lot of time playing with the various "deep learning" libraries that are available as we prototype the best ways to use this exciting new technology to testing.

For those of you who haven't used "deep learning" libraries here's a quick summary of how they work:

  • You define a decision/arithmetic formula. Maybe something like, "Will I have a salad for lunch" is determined by today's weather, what I had for dinner last night, what my lunch buddy Chris is having for lunch, and whether I read an article about healthy eating in the last 3 days.
  • You then run a large number of examples through the formula; e.g. today it was 22C, I had a healthy dinner last night, Chris had a salad, and I haven't read any articles in the last 3 days, and I did NOT have a salad.
  • The "deep learning" library uses these examples to determine the relationship between the inputs and the output, e.g. Chris having a salad does not make it more likely that I will have a salad, but if he doesn't have one then I definitely won't have one.

It takes a little time to get your head around it, but there are only a few core concepts, so once you understand them it's easy to play around with. I'd recommend playing around with Google's TensorFlow as it's easy to use, has some decent visualization tools, and relatively good documentation.

So how does this help testing? The clear application for me is bug hunting. Understanding the relationships between inputs (e.g. user data) and the control flow (i.e. which paths are taken through the application) and finding defects. For example, tests which change user settings in the middle of a test often find bugs, going through the help screens never increases the change of finding bugs, going through transaction screens has as low likelihood of finding bugs but the bugs found are highly severe. If we had an engine that could automatically generate paths through an application (stay tuned for that product release coming soon), automatically generate user data, and had "deep learning" providing the feedback loop, that would be an awesome bug-hunting machine.

Antony Edwards

Written by Antony Edwards

He’s our COO. He guides where the product goes. Antony likes haiku.