Opinion: When Digital Forensics Vendors Hire Research Talent, Where Does It Leave Research?

by Christa Miller

In the second half of 2019, a set of hirings made some waves in the digital forensics community. First, in July, Cellebrite hired well-known SANS Senior Instructor Heather Mahalik. Then in August, Mike Williamson joined Jessica Hyde, Christopher Vance, and others at Magnet Forensics. In December, the set completed when BlackBag Technologies hired likewise well-known SANS Senior Instructor Sarah Edwards.

“Name” researchers going to work for vendors is nothing new, of course. Amber Schroader founded Paraben in 1999; Lee Reiber took over as Oxygen Forensics’ Chief Operations Officer in 2015, while Edwards’ transition to BlackBag put her in the already well-established research powerhouse of Vico Marziale and Joe Sylve.

Then Cellebrite acquired BlackBag, consolidating that powerhouse together with Mahalik and a formidable R&D team. With that, the talent acquisition process began to feel more like a research ring match, with Cellebrite and Magnet Forensics trying to knock each other’s blocks off. Where does that leave research itself?

The tradeoff between resources and profit

By now the industry takes it as a given that no one — neither vendors nor independent researchers — can keep up with the new devices, apps and app versions, or operating systems and their versions. Near-constant changes to these elements affect the way they store data, and in turn the way forensic tools acquire and parse that data.

Get The Latest DFIR News

Join the Forensic Focus newsletter for the best DFIR articles in your inbox every month.

Unsubscribe any time. We respect your privacy - read our privacy policy.

Researchers need two things to attempt to keep pace: time and funding. It’s rare when research can be done for its own sake. Most is done in conjunction with casework or coursework and involves a specific device make / model, or a specific app.

Arguably, vendors are investing in research “deep work” that can ultimately make their tools stronger and serve a wider range of forensic examiners with highly relevant acquisition and analysis capabilities.

On the other hand, no private entity invests in anything without anticipating a return. Community goodwill is valuable, but only if it results in additional sales. Whereas independent research has always been about solving interesting problems and sharing the results with the community in the hopes that it will help, vendor tool development focuses on the most critical needs — identified by the community, yes, but prioritized by how frequent the feedback is.

That means the price of investing in “deep work” may be the kind of research that solves interesting problems. The concern isn’t so much that it will become the vendor’s intellectual property — Edwards’ APOLLO remains on GitHub in addition to being a BlackLight plugin, for instance — as it is the research’s focus. The really “interesting” problems may well be paywalled behind the labs that some vendors now run.

A closed, black-box competitive advantage

Again, proprietary research is undisputedly part of the business and has been for a long time. But the risks of too much concentration in the vendor realm were highlighted in 2011’s Digital Forensics with Open Source Tools, where Cory Altheide and Harlan Carvey wrote of their “experiences where proprietary forensic products have produced demonstrably erroneous results, but these tests were performed in a ‘black box’ scenario.”

This problem has persisted and accelerated, as new app or operating system versions are known to “break” both proprietary and open source tools — showing incorrectly parsed data, for example, that can lead to erroneous interpretations.

In those cases, vendors tend to patch quietly. Whether out of fear of embarrassment in a tightly competitive market, or out of broader admissibility implications, it’s difficult to say. (Imagine that a judge, misunderstanding how easy these kinds of errors are, could call into question every result from that brand, as if all the variables were monolithic.)

It isn’t as if researchers’ efforts will disappear behind a veil of secrecy, as numerous blogs, presentations, podcast episodes, and the like already show. On the other hand, their research, written up for official vendor blogs, wouldn’t necessarily include what a vendor wanted to hold back. Often, what goes unsaid is as important as what goes on the record.

One other risk of more competitive research: collaboration. Researchers who might once have freely shared with one another and built on each other’s research are less at liberty to do so now. “Coopetition” might allow a kind of uneasy truce-forging between vendors in the name of, say, a webinar or topical lecture at a conference. However, just as monopolies limit innovation by limiting competition, researchers may be less willing to share the results of their work if transparency means limiting their employers’ competitive advantage.

It might be compared to a vendor’s outright purchase of intellectual property that leaves no open source alternative. By trading source code for scale and an easy-to-use interface, the tool developer limits its visibility. The work that went into the original research could conceivably be replicated, but it would be unnecessarily duplicative — and besides, limiting examiners’ options for testing and validation when they are already pressed for time doesn’t serve justice.

Of course, just because the admissibility of digital evidence isn’t frequently challenged doesn’t mean it won’t be. While it’s understood that proprietary methods are protected during court proceedings, worth noting is that one of the foundations of admissibility is whether a theory or tool has been tested. Another is whether the results are reproducible and repeatable.

Experienced professional forensic examiners like Mahalik, Edwards, and others know this. That’s why it’s encouraging to see Mahalik, in her recent post, call for additional research to be shared with the community — a springboard of sorts. By using her platform to encourage this kind of work, she’s accomplishing two things:

  1. Promoting independent research and tool development ultimately enables examiners to validate the results of tools like UFED Physical Analyzer.
  2. Actually highlighting the research and independent analysts who conduct it.

The commoditization of relationships and the community

Part of the reason researchers got to where they are is their strong, consistently shared research. Blog posts, Twitter feeds, podcasts, and other media offer a transparent means of showcasing digital forensic research.

Open-source community projects notwithstanding, however, the concentration of so much top talent behind vendors’ closed doors effectively plays off one against another — and commoditizes the relationships they have with community members.

Certainly, researchers who have strong relationships with the community, as well as strong reputations for being hitherto vendor-neutral, stand a better chance than, say, an average product manager in asking for suggestions or feedback on trends and the tools themselves.

But communication is part speaking, part listening. Vendors are driven by business strategy, and their viewpoints can be affected by business goals and needs. Just as the bottom line drives R&D, these filters might inadvertently result in key insights or trends going unrealized, key content never being written or recorded, and so forth.

Certainly, responses to polls and questions on social media can benefit more than one vendor, as well as independent researchers. Likewise, examiners who request research from one vendor aren’t placed under an NDA that would stop them from asking the same of another vendor.

In fact, this might be considered a duty, particularly when using more than one tool. Asking multiple vendors for the same feature can only improve examiners’ ability to test and validate its results — assuming, again, that vendors choose to put their resources into that particular feature.

Towards stronger community-based research

It might be wise to redefine “research” to encourage more people to conduct it. Not everyone can write their own parsing scripts — or enjoys writing blog posts — but Edwards has stated that basic verification can be performed in as little as five minutes, and screenshots can accompany tweets or messages posted to the Forensic Focus forums, the digital forensics Discord server, and other venues.

Meanwhile, researchers like Josh Hickman are posting test images that the community — as well as vendors — can use to validate their tools, and populating test data for given apps can be gamified between researchers or even as a student project.

Another good-faith way for vendors and researchers to build community is by replicating the APOLLO approach, in much the same way Cellebrite and Belkasoft did when they recently included the Checkm8 solution in their tools. The original Checkra1n exploit remains free and available for testing to validate extraction results — on test devices, naturally — but the tools offer a forensically sound data acquisition method. (Bonus when the vendors are willing to give credit where it’s due to the researchers who created the tools.)

As for whether enough independent blogs exist to balance proprietary research, this might be a case in which less is assuredly not more. As independent researcher and blogger Alexis Brignoni observed in a recent blog post: 

“… as expected, a new OS version will break previously [known] good artifact parsers for both third party apps and native files. It is our job to figure out where the known but now lost items are as well as finding new artifacts we weren’t aware of. This is how toolmakers can focus effectively on what is needed to be done, by [us] doing the work and telling them it is important to us….

“As examiners we own the data we are tasked with processing and it is our responsibility to verify that any inferences gathered from it are exact and backed up by the source. We are uniquely positioned to identify gaps in knowledge, to work in filling them up, and sharing that knowledge with others that can automate the process to the benefit of the greater community of practitioners…. Your perspective is needed, your expertise is essential. Make it known.”

1 thought on “Opinion: When Digital Forensics Vendors Hire Research Talent, Where Does It Leave Research?”

  1. Great article. I think it makes good points, but I believe the situation is better than most believe. Yes, some of the major players in the research arena have moved to private tool vendors, but they are sharing research just as they were before, though some of that research may not be as obvious as before (which is one of the points of the article).

    Repeatability and reproduction are important. When vendors complete a piece of R&D they will roll the results of that R&D into their tools. If a tool vendor has said their product can do X, then that tool should be tested to see if it consistently (repeatability) gets the expected results (reproducibility). The best way to accomplish this is to run a known data set against the the tool. This is one of the many reasons why I release Android images with accompanying documentation, and there are plenty of test data sets out there that can also be used for the same purpose. DFIR.training, Digital Corpora, and NIST all have documented data sets that can be used for tool testing. By running a tool against a known data set, does it not confirm/disprove the underlying research that went into that particular tool feature?

    Yes, this can be time consuming and have a negative operational impact when an organization has few people and high case loads, but both the vendor and the customer are best served by doing so. A vendor will know that there may be a problem with the tool’s implementation of the R&D (or the R&D itself), and the customer gets a tool they know works. I would argue that there is more risk in NOT testing the tool(s) before use. All it takes is one bad examination to ruin the reputation of and examiner, a laboratory, or a vendor tool. More importantly, however, it could cause an innocent person to lose their freedom, or allow a guilty person to keep theirs.

    Independent researchers can be viewed as an augmentation of vendor R&D, which results in the digital forensic community being better served by open source and vendor tools. The communication is still there between vendors and independent researchers. As an example, last summer I published a blog post detailing research I had conducted on the Wickr messaging application. It took a few months, and eventually I was at a standstill due to some encryption issues that I was unable to resolve. A few emails to Cellebrite’s Heather Mahalik and Or Begam resulted in some collaboration that ended up resolving some of the issues I was facing with the Android version of Wickr. For iOS, I ended up collaborating with Magnet’s Mike Williamson. The results of the collaboration with both Cellebrite and Magnet ended up in Physical Analyzer and Axiom. The DFIR community now has methods to examine Wickr messaging data because two competing tool vendors collaborated with an independent researcher.

    I do not think the consolidation of researchers will have a negative impact. Even if the researchers were to stop sharing research tomorrow, the DFIR community would continue to improvise, adapt and overcome as it always has.

    Call me an optimist. 🙂

Leave a Comment

Latest Videos

This error message is only visible to WordPress admins

Important: No API Key Entered.

Many features are not available without adding an API Key. Please go to the YouTube Feeds settings page to add an API key after following these instructions.

Latest Articles