The prior post in this series discussed the importance of quality assurance and our philosophy when it comes to executing our processes. Let’s now look at those processes in more detail, starting with the steps surrounding filesystem testing.
The first step of the quality assurance process does not actually live with the quality assurance team. Our developers are responsible for writing unit tests and ensuring they pass prior to completing their work on a feature. A unit test is a small piece of code that tests a very specific area of the actual code that executes within the application. For example, if a developer is working on a new filesystem which uses a new index structure, they will write a unit test that uses a sample index structure with known values to ensure that the code reading it returns the correct results. A typical function in the application might require several different unit tests to test different conditions.
In the above example, a unit test might be created with a corrupted index structure to ensure our code is handling that specific situation as we would expect. Since so many different situations and conditions often exist, it isn’t uncommon for the size (in terms of lines of code) of unit tests to exceed the actual code by several times.
The next step in our process involves automated filesystem testing. This functionality has existed in various forms since the early 1.x days, but has expanded in both the number of systems it supports as well as the depth of testing. This process is designed to test a specific filesystem with a specific forensic image where we know the expected results. It sounds a bit like the unit testing we just covered, but in this situation, we are testing how the various functions of the application work together in a cohesive fashion.
For this reason, these tests are often referred to as “end-to-end tests” because they are attempting to test the user experience from the beginning to the end. Our end-to-end tests automatically load a forensic image, detect it, scan it, save the case, reload the case, preview clips, and export clips. At each step in this process, the results are reviewed against the known “truth” and if a discrepancy exists, we manually review the results and address any issues identified.
As you can imagine, this level of automated testing requires IT infrastructure to operate in any reasonable amount of time. This system used to run on a single desktop which would run these tests on our test recording data before releasing any update to DVR Examiner. Last year, we distributed the system to 5 computers, and even with the added capacity, testing every filesystem took close to 24 hours. Earlier this year, we made the latest investment in our IT infrastructure to better support our QA team and goals. We have expanded our network storage capability to host nearly half a petabyte (500 TB) of forensic images of DVR hard drives.
We also introduced a dedicated server just for running these automated tests. This server runs multiple docker containers which can each execute any series of tests. The storage and server are connected via 10 gigabit ethernet which allows us to easily move and access large sets of data on the network tremendously fast. With this new infrastructure in place, we can now execute all end-to-end tests against our test recording data on a nightly basis. This will have a great impact on our ability to respond quickly to potential issues as they arise.
This battery of filesystem testing won’t prevent every issue that users might encounter, because we simply cannot account for every permutation of DVR, hard drive, configuration settings, and recording history out there; but this system will enable us to address many preventable issues which frees up our time to respond quickly to any issues encountered by users. Believe it or not, none of the above processes use a user interface. So how do we test the buttons you click in DVR Examiner and the workflows you choose? There are processes that address the entire front-end experience of the application, which we’ll discuss in our next post.
Remember to subscribe to our blog & monthly newsletter to stay up to date with with all DVR Examiner developments/updates, as well as all DME Forensics news. Until next time, thanks for reading and stay safe and healthy!