Jeannette Smith - 17 May 2018
Note: Test engineer Reena Kuni and software engineer Bekki Freeman also contributed to this blog.
On the Eggplant Functional team, the relationship between Dev and QA is very collaborative. We work closely together, use our Slack channel to organize regular walk breaks together, and frequently talk about ways to increase product quality.
Recently, someone brought up the idea that maybe too much collaboration could lead to bias — specifically, the developer of a feature inadvertently creating bias in how QA approaches testing that feature. This is an interesting point. As we grow and re-evaluate the way we do things, we’re trying to find the best approach for our size that will hopefully scale as we do.
We know that after the QA team members familiarize themselves with the concept of the feature and begin to create tests for it, they’re anxious to receive a build with the feature so they can get their hands on it as early as possible to begin working with it. There’s an art to determining the right time for Devs to release the product to QA. This begs the question: “How agile are we?” Delaying the Dev-to-QA handoff starts to look like the waterfall methodology. But a product should be test-worthy; providing a feature too early can result in a lot of repeat work and back-and-forth activity between Dev and QA.
We’re balancing the timing of the handoff knowing that getting a feature to QA early generally highlights missing or awkward functionality. In release 18.0, specifically, we vastly underestimated how much time would be required after a core feature was completed, due to discussions over edge cases and additional functionality that were only discovered once QA started using the feature and designing tests for it.
To address the handoff transition, one thing we’ve been doing more often is a feature show and tell — where the developer of a feature gives a short demo of it. This is a really good entryway into testing it. Everyone at the demo can see the feature in action and ask questions, giving the developer a chance to answer and address those questions all at once. Soon after this demo, the feature will go to QA for testing. After initial testing, we give a second demo to a wider audience that includes our technical consultants, support, and documentation teams. This invites more questions and ideas from people who may have different expectations for the feature.
Once we do the initial Dev-to-QA handoff, we’ve been trying something else: tracking those initial exploratory findings in a document. We’ll talk more about this in our next blog about bug tracking. Stay tuned.
Collaboration During the Release Cycle
Based on our discussions, this is how we’re approaching collaboration during the release cycle:
- Start working with new features early on to more quickly find bugs and usability issues.
- Never assume a new feature is low risk or can be delayed until the end.
- Assess the impact of the new feature on related, existing features — especially when the change impacts core code.
- Ensure a huge amount of collaboration and discussion happens once a feature is ready for testing.
- Clearly communicate what’s been implemented and what doesn’t work yet so that QA knows what to focus on.
For us, the sweet spot seems to be a combination of art and practice between developers and testers to determine when a bug or feature is ready to move from Dev to QA. We also think it varies greatly within different teams, and involves lots of factors, including:
- The size of the development team.
- How well the bug was explained or how well a feature was defined.
- Some developers have a better understanding of how a user will use a feature and some testers are more technical. The latter is especially true among our customers, who vary greatly in their technical skills.
- How well QA knows the bug or feature.
- If anyone was able to reproduce the bug in-house.
- When it’s ready (especially new features) for that first handoff to QA. Are there still outstanding questions or does something need more clarification?
- We talk about wanting to be collaborative, but then we talk about the number of times a bug or feature goes back and forth between Dev and QA. Is frequent back-and-forth activity more acceptable in a smaller company than a larger one? Does this suggest we want more Dev code reviews to reduce the back and forth between QA and Dev? Or, is there value in the QA perspective and more back-and-forth interactions?
What do you think?
If you’ve solved this problem for your organization, please share what works and what doesn’t. We’d love to hear from you.