Digital forensic tools rely on the National Institute of Standards and Technology (NIST)’s National Software Reference Library (NSRL), the standard reference dataset, to filter out known software and images from unknown ones. Forensic examiners, judges, and attorneys rely on NIST’s Computer Forensics Tool Testing (CFTT) Project to ascertain whether a tool has met a performance baseline to determine its admissibility.
Two newer NIST projects have been introduced to further strengthen these foundations. Forensic Focus spoke with Barbara Guttman, leader of NIST’s Software Quality Group, about how a black box test and a federated CFTT rely on the digital forensics community to contribute to the broader industry.
The Black Box Test
Anonymized research into the general state of digital forensic practice the black box study is designed to work in conjunction with a series of scientific foundation reviews for forensic science — what Guttman described as “studies that document the body of literature and scientific [information] that actually exists to support the work in a given discipline.”
The point is to show the degree to which digital forensics is “on a firm foundation grounded in science and technology,” Guttman explained, bridging between academic and practical realms. The test should be able to show whether practitioners understand not only what tools and techniques do, but also what they don’t do. “We’re hoping a lot of people participate,” she said, “so we can get a sense of [whether they’re] providing the kind of quality that the customers of the work expect.”
The test includes the kinds of questions digital examiners would answer in reports or on the stand. “They’re not ‘gimmes,’ and they’re not ‘gotchas,’” Guttman said, “so they should give a reasonable assessment of the state of practice.” For example, the mobile test asks questions like, “The drug deal was discovered in which of these apps?”
So far, said Guttman, NIST has distributed 460 multiple-choice tests. “To sign up, you do have to give an email address to verify your lab demographic,” she added. Some of the basic criteria include lab size, general region, law enforcement vs. not law enforcement — and how the non law enforcement labs are divided — practitioner years of experience, certifications, and whether the lab is accredited, although Guttman stressed the test doesn’t ask participants to spend a lot of time on this type of data. “We didn’t ask people 100 questions before they even took the test,” she explained.
As a black-box study, the test focuses on process, not tools, using two fake cases: one mobile and one a Windows PC. “We don’t collect information about what tools the participants use,” said Guttman, “[though] we did check that you could get the answers using several different tools.”
NIST expects the black box test to stay open for 3-4 months. It will take time to analyze and review, “to make sure that what we say is truly defensible by the data we collected, that it means what people will think it means,” said Guttman, adding:
“That’s actually remarkably hard to do. People want to latch onto little nuggets. You can’t completely avoid that, but we will make an effort to present things clearly so people will get the gist of the study.”
The Federated Cyber Forensic Tool Testing Project
Most people in the industry are familiar with NIST’s Computer Forensic Tool Testing (CFTT) Project. On an ongoing basis since 2002, NIST researchers release the results of what Guttman called “a straightforward process” to identify whether vendor tools “do what they think they do.”
“When you’re using a tool… in a perfect world, the tool would work perfectly and everything would be great,” she added. “But we don’t live in a perfect world. So people need to understand what their tool’s limitations are, so they can use them wisely… especially in the mobile space where things move so fast.”
The CFTT tests tools by central functionality: disk imaging, string searching, mobile acquisitions, and so forth. The project is part of developing specifications, said Guttman, “because if you want to test string searching, you should be able to find words and other strings in all different character representations [which] gets kind of technical.” For example, languages like Chinese are logographic and syllabic, rather than alphabetical.
When the CFTT started, the evaluation itself was straightforward because only a few tools existed in the space, and computers hadn’t yet given way to mobile devices on a mass scale. Now, with the digital floodgates open, Guttman said, far more tool testing needs to be done than the NIST lab alone can handle.
That’s because of the implications for admissibility. Scientific methods that are used to acquire or examine evidence introduced as exhibits in court are subject to legal tests that help determine whether the evidence can be authenticated.
Enter the federated CFTT, announced in 2017. “We took the NIST testing and we [put] it in a box… so you can do this level testing in the comfort of your own lab,” said Guttman. Labs can use it to support their own internal validation work, or share their test support — either among people they know, or with NIST. The other benefit: by relying on NIST test standards, it gives people more confidence in other people’s testing.
Community testing has two other parameters. It wouldn’t be operationally efficient, said Guttman, for 10,000 people to test the same tool. Further, balance is important. “Any time you run a testing program there’s a balance to be made…. [Y]ou could test forever and keep finding flaws or anomalies,” she said, “[versus testing] for the majority of major problems.”
NIST has already collected the results from 27 community tests from such disparate sources as defense counsel in Missouri, a northeastern transit police department, and university programs invested in the federated test as a student project.
Guttman added that some vendors want to adapt the project for their own products. “We’re really excited… to move out into the community, and it’s growing slowly,” she said. “One thing I like about working at NIST is that everything is just free and open… all the test procedures are online, so people can do this freely.”
Eventually the federated tool test results will be in a searchable database, which NIST is in the process of building. That website will enable searchers to learn whether a tool was tested against a given app, or how it parses sender data.
“People can post their [own] report so other people can use it too,” said Guttman. “You can search just for reports that came out of NIST, or just reports that came out of the federated testing program, or all reports.”
The estimated date of delivery for the federated CFTT’s portal beta version will be sometime later in 2020, in conjunction with a new Computer Forensic Reference Dataset (CFReDS) portal. “[We figured], we’re building all these datasets for our tool testing, why don’t we share it with more people?” Guttman said. “It seemed like a natural thing to do.”
Guttman said the new portal would be an improvement over the existing website-based table, enabling even more people to share their own reference data. The CFReDS portal will share a taxonomy with the federated CFTT — as well as the Computer Forensics Tools & Techniques Catalog — so it will be possible to use the same terms to search, as well as to coordinate between the projects.
Asked whether it would be possible to cross reference the black box test with the federated testing project, Guttman said eventually, depending on the number of responses to the black box test, “It could lead to a thousand more future studies.”
To enroll in the black box test project, find the Google Form here.
To participate in the federated CFTT, visit the project page here.