TL;DR – We fixed a weird bug in DVR Examiner 2.9.1. It is automatically fixed if you are internet-connected when you install DVR Examiner 2.9.1. If you are an offline user and just want the fix, click here. Continue reading for a deep-dive into this bug and our processes if you’re into that kind of thing.
Since we released DVR Examiner 1.0 in 2013, we have seen our fair share of bugs, but some stand out to us more than others. Sometimes it is because of a larger impact to users, other times it is because of just how obscure they were. Typically, the more obscure the bug, the harder it is for us to reproduce it and then develop a solution. For example, way back in the beginning, one of our first users switched computers and suddenly couldn’t start DVR Examiner. No window, no error message, no log, just would not start. Other computers did not give him (or us) this problem. In the end? That computer was missing a common font, so the user interface wouldn’t load. I don’t think we’ll ever forget that bug.
In this article, we are going to take a look at a bug that was recently resolved with the release of DVR Examiner 2.9.1, as well as some thoughts on our process in general. As is the case with most weird bugs, this bug ended up being something rather simple, but the path to getting there is a story worth telling in my opinion.
The Problem
Several months ago, we started receiving a trickle of reports from users that could not preview or export from DVR Examiner on a particular computer. This seemed to affect any data they used DVR Examiner with, so it did not appear to be limited to a specific piece or type of data. While we investigated the issue, we recommended they try the same data on another computer with DVR Examiner. The overwhelming majority of the time, it worked just fine. Uninstalling and reinstalling DVR Examiner on the non-working computer did not resolve the issue. The only hint we had was an error in our logs that read (in part) “System.DllNotFoundException: Unable to load DLL ‘cvextern’: The specified module could not be found.”. And if the cvextern DLL actually wasn’t present, we could end this bug breakdown right here. Naturally, the file did exist on each user’s machine in the correct location.
Our General Process
We tend to see bugs that fall into one of two categories:
- They affect a single user with a single piece of data
- They affect all (or very, very many) users with a particular type of data (a hard drive/image that utilizes the DHFS_41 filesystem, for example) or all users who utilize a particular process/feature (automatically exporting after scan, for example).
In the first situation, sometimes it is because the system is being misidentified. This typically happens when a new DVR system is released with a small tweak to something we already support. Another possibility is that there is something specific in this one user’s data that is unique and we either haven’t seen it before or wouldn’t expect to. A classic example of that was a 1 TB hard drive that, out of all the frames on the disk, contained a single frame that contained metadata indicating that the timestamp of that frame was 23:59:60. Anyone who knows how a clock works will say “that isn’t possible”, and they would be right. Except the DVR might have “hiccupped” in that moment and did the impossible. We spend more time trying to predict realistic errors than we do predicting the impossible, and rightfully so. But whatever the reason for the metadata, we needed to handle the possibility of that situation more gracefully than we were.
In the second situation, this is the type of bug that is a double-edged sword for us. On one hand, it is affecting many users, and we don’t want that for obvious reasons. However, it also generally leaves us with more information and data to troubleshoot and reproduce the problem. If it is happening to everyone, then reproducing the problem in-house is often much easier than the first situation. In the case of a particular type of data, it is still possible that a new system was released and many users are now encountering it at the same time, but it is also entirely possible that we just simply made a mistake. We try to avoid those, but we are human.
The cvextern Bug First Steps
Which of the above two situations does our bug of interest fall into? Well, neither really. We had a handful of reports of this problem, but it was still a rather small percentage of our users overall. We suspected that it was more widespread than we knew, simply because so often switching to another computer would “solve” the issue, it is possible that some people did just that and did not contact us.
As an aside, never hesitate to reach out to us to provide assistance. We want our users to use the equipment, workflow, and methods that they are comfortable with, and while it might be necessary to work around those in some situations temporarily, we want to get you back to fully functional as soon as possible.
Back to this bug, we started trying to find commonalities between the users and computers that were experiencing this issue. We found that there were no consistencies between:
- The operating system in use (Windows 7 and Windows 10 both had reported instances, and on more than one occasion users switched from a non-working Windows 7 machine to a working Windows 10 machine and vice-versa).
- The other software installed on the machine (at least one instance reported experiencing this issue with a fresh Windows installation).
- The location of the users (neighboring agencies were not experiencing this issue).
- Any recent hardware or software changes.
To make matters more difficult, despite trying just about every machine we could get our hands on, we couldn’t reproduce the issue. Until we could.
Reproducing the cvextern Bug
We recently invested in a new server for automated testing and deployments. Little did we know rolling out this process would lead us to finally reproducing this issue. The server was configured using lightweight virtualized Windows instances that we can spin up and down as needed to run automated tests on a “clean” environment. We had automated testing in place previously, but those tests were run on traditional hardware with full Windows instances, etc. When we migrated those tests to the new server, suddenly some of those tests started failing. A quick look at the logs showed us why: “System.DllNotFoundException: Unable to load DLL ‘cvextern’: The specified module could not be found.”. Like our users, the cvextern DLL was in fact in the proper location in the virtualized Windows instances, but for the first time, we were able to reproduce the error message on a system that we controlled and could test on.
Fixing the cvextern Bug
By now, you may be asking yourself “what is ‘cvextern’ anyways?”. In short, the “cv” comes from OpenCV – an open source computer vision and machine learning software library. Like many companies, we utilize libraries like this one to speed development and avoid “reinventing the wheel” wherever possible. The trick is that libraries (especially large ones like OpenCV) often ALSO rely on other libraries. When those other libraries don’t exist, bad things can happen. In our case, we use another third-party library to access OpenCV more easily, and that library made a change to the libraries it relied on.
So why wasn’t this a more widespread issue (and therefore easier to reproduce)? It turns out, the library that recently became a requirement is included in a VERY large percentage of Windows installations – but not all of them. There are editions of both Windows 10 and Windows 7 (named Windows K, Windows N, or Windows KN) that do not include certain media feature packs. When our libraries started relying on these media feature packs (which is usually a good bet given the wide distribution), everything broke for our users that happened to be using Windows K/N (as well as for our new automated testing server which utilizes a very lightweight instance of Windows which does not include these feature packs).
It does make sense that we would see some Windows K/N users given that K/N editions are often deployed in volume licensing situations (which many agencies use to license their computers). Many forensic users often have specialized machines that are purchased outside of volume licensing agreements and it stands to reason that those installations are usually going to be traditional Windows editions and not the K/N variety. This is why most users were able to switch to another to work around the problem.
The Solution – Internet Connected Users
Good news – you don’t have to do anything special. Just install DVR Examiner 2.9.1 and if you don’t have the required feature packs, we’ll grab them from Microsoft during the installation process and install them for you.
The Solution – Offline Users
Like most forensic software companies, many of our users are offline a significant portion of the time. If you fall into this category, don’t worry, there is still a simple fix. Unfortunately, we can’t distribute the feature packs directly by including them in our installer download, but you can download them directly from Microsoft and install them prior to installing DVR Examiner 2.9.1.
For Windows 10 Users:
Microsoft Article (with download link):
https://support.microsoft.com/en-us/help/4016817/media-feature-pack-for-n-edition-of-windows-10-version-1703-april-2017
Direct Download Link:
http://download.microsoft.com/download/9/E/C/9EC0EC2C-8151-468A-95A3-2F2AD6A2AC13/Microsoft-Windows-MediaFeaturePack-OOB-Package.msu
For Windows 7 Users:
Microsoft Article (with download link):
https://www.microsoft.com/en-us/download/details.aspx?id=16546
Direct Download Link:
https://www.microsoft.com/en-us/download/confirmation.aspx?id=16546&6B49FDFB-8E5B-4B07-BC31-15695C5A2143=1
Last Words
This was a bit longer than I originally intended, but I hope that it gives you some insight into our thought processes both for resolving this issue and issues in general. To the users who experienced this problem, thank you for your patience and assistance in troubleshooting this issue. To each of our users, as I stated earlier, please don’t ever hesitate to reach out if we can be of assistance. Whether it is a simple issue or something a bit more in-depth like this one, we are here to help get it resolved for you. Until next time, be safe out there!