Our recent release of DVR Examiner 3 marked the largest and most complex update of the application to date. Not only did it include a brand-new user interface, but it also includes dozens of new features, three new filesystems and a completely new backend architecture. A develop effort of that magnitude can be a testing nightmare. However, I am proud, and a little relieved to say, we were able to pull it off with no major issues reported. I attribute this to the quality of the developers we have at DME, the investments we made in Quality Assurance and the acceptance test program we put in place for this release. I’d like to focus on the acceptance program in this post.
A typical software acceptance testing program involves alpha testing and a beta testing. Alpha testing is usually done in-house, early in the release cycle and is designed to catch bugs before the software is finalized and sent out to customers to use in real world environments for beta testing. This works great and has become the standard for performing acceptance testing. However, testing forensic tools is a little different than testing other software applications, so our approach has to be a little different and very thorough. So instead of the typical testing process that looks something like the following:
We added a few steps to the second half or the user acceptance testing part of the release test process. In doing so, our process looks like the following:
We added an external alpha phase where we sent out an early release of the software to selected users to not only find bugs, but to give us their opinions on the usability and features early in the process. This allowed us to make a number of usability improvements based on our users’ feedback as well as fix bugs the found.
The second step we added was the “bug off”. The bug off is an internal competition that we set up for every member of the company to find as many bugs as they could in a 6=hour period of time. Because we were conducting this competition during a global pandemic, the event was held virtually. We sent food delivery gift cards to everyone (because food makes everything better!), went through the rules of the competition, and offered small prizes for the top bug finders. Then we let the team loose on the new software. No bug was too big or too small. The goal was to uncover every bug before the next phase beta test, where a large group of customers would use the application for the very first time. Being somewhat new to the organization, I was taken back by the level of competitiveness the DME team has.
The game was on and bugs were being found by everyone! While no Product Manager wants to find bugs in the software, the team was uncovering bugs before the application was reviewed by customers in the beta test, and that is a good thing. It allowed our beta test participants to test the software and provide feedback on usability and workflow, instead of reporting bugs. Yes, our beta test participants did uncover bugs, but nowhere near the number that would have been found, or worse, would not have been found, if we didn’t have the bug off. Because the bug off was so central to the successful release of DVR Examiner 3, we have made it a regular step in our acceptance test program.
If you have not had a chance to use DVR Examiner 3 yet, download it now. If you are not currently a customer request a free trial.