NIST Looks into the Black Box of Computer and Mobile Phone Forensics

According to Barbara Guttman, leader of NIST’s digital forensics research program, “We want to understand the state of the practice. Can experts produce accurate and reliable information when extracting data from a digital device?”

en flag
nl flag
et flag
fi flag
fr flag
de flag
pt flag
ru flag
es flag

Editor’s Note: Taken directly from the recent NIST media release highlighting the first large-scale “black box” study to test the accuracy of computer and mobile forensics, the following information is provided to describe the new digital forensics research initiative from NIST which, according to Barbara Guttman, leader of NIST’s digital forensics research program, is designed to help better understand the state of the practice and measure how well digital forensics experts do their job.

NIST Media Announcement

NIST to Digital Forensics Experts: Show Us What You Got

Digital forensics experts often extract data from computers and mobile phones that may contain evidence of a crime. Now, researchers at the National Institute of Standards and Technology (NIST) will conduct the first large-scale study to measure how well those experts do their job. But rather than testing the proficiency of individual experts, the study aims to measure the performance of the digital forensics community overall.

In this study, to be conducted online, participants will examine simulated digital evidence, then answer questions that might arise in a real criminal investigation. The exercise should take about two hours, and participation is voluntary. Enrollment is now open, and the online test will be available for approximately three months.

“We want to understand the state of the practice,” said Barbara Guttman, leader of NIST’s digital forensics research program. “Can experts produce accurate and reliable information when examining data from a digital device?”

In any forensic discipline, experts can encounter difficult cases. Fingerprints can be smudged and distorted. DNA can be degraded. One challenge with digital evidence is that it can often be difficult to find key bits of evidence among large volumes of data. Also, technology changes so quickly that it can be difficult to keep up.

“Forensics experts can’t extract data perfectly in every possible scenario,” Guttman said. “Phones change. Apps change. The world just moves too fast.”

While no forensic method works perfectly all the time, researchers can measure performance within a discipline by testing the experts. For instance, researchers might show fingerprint experts a series of prints and ask whether they do or don’t match. The study designers know the correct answers, and by combining the results from many experts, they can gain insight into the reliability of the method overall.

These studies only determine whether the expert gave the correct answer, without concern for how they reached it. In other words, they treat the expert as a black box — something you cannot see inside. Researchers use black box studies to assess the reliability of methods that rely on human judgment.

For the NIST black box study, participants will download simulated evidence from the NIST website in the form of one virtual mobile phone and one virtual computer. Such virtual devices, called “forensic images,” are commonly used in digital forensics, and study participants will be able to connect to them using the same software tools they use when working on real cases.

The forensic images created for this study simulate imagined but realistic scenarios involving a potential homicide and a potential theft of intellectual property. Study participants will download the images, examine them using whatever forensic software tools they choose, and answer a series of questions. For instance:

  • What software program was used to discuss a potentially illegal transaction?
  • What was the VIN number of the vehicle that connected to the phone via Bluetooth?
  • What location information can be gleaned from the photo of a black Labrador found on this device?

The study is open to all public and private sector digital examiners who conduct hard drive or mobile phone examinations as part of their official duties. NIST will not calculate the performance of any specific expert or laboratory. Instead, NIST will publish anonymized and aggregated results that show the overall performance for the expert community and different sectors within that community.

This study will fulfill a critical need identified in a landmark 2009 report by the National Academy of Sciences. Titled Strengthening Forensic Science in the United States: A Path Forward, that report called for black box studies to measure the reliability of forensic methods that rely on human judgment. Courts and jurors can then consider the results of those studies when weighing evidence. The results of this study will also provide strategic direction for future research.

This black box study is part of a larger effort to evaluate the scientific foundations of digital forensic methods. NIST is also conducting scientific foundation reviews for DNA mixtures, firearms identification and bitemark analysis.

For more information and to enroll, visit the NIST Blackbox Study for Digital Examiners webpage.

Read the original media announcement at NIST to Digital Forensics Experts: Show Us What You Got

Additional Reading

Source: ComplexDiscovery

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

ComplexDiscovery is an online publication that highlights data and legal discovery insight and intelligence ranging from original research to aggregated news for use by business, information technology, and legal professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding data and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of data and legal discovery organizations. Registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world, ComplexDiscovery OÜ operates virtually worldwide to deliver marketing consulting and services.

A (Brand) New Approach? Considering the Framework and Structure of eDiscovery Offerings

Today’s eDiscovery providers may benefit from the lessons learned in the creation of the Sgt. Pepper’s Lonely Hearts Club Band album by creating a concept for branding and packaging their offerings within that brand in a connected, theme-based way that represents the offerings’ promise and capability in a way that is easy to understand and remember.

This fictionalized branding approach was developed from the intellectual exercise of trying to figure out a reasonable and memorable way to descriptively highlight the promise and capabilities of offerings typically delivered by full-service eDiscovery providers. It may not be completely comprehensive or fully normalized. However, the hope of sharing this branding example is that it might help those involved in the branding and communication of eDiscovery provider services and solutions.

eDiscovery Mergers, Acquisitions, and Investments in 2020

Since beginning to track the number of publicly highlighted merger, acquisition,...

Relativity Acquires VerQu

According to Relativity CEO Mike Gamson, "It's imperative that the legal...

eDiscovery Mergers, Acquisitions, and Investments in Q4 2020

From Nuix and DISCO to Exterro and AccessData, the following findings,...

DISCO Closes Funding Round of $100 Million

According to DISCO CEO Kiwi Camara, “Legaltech is booming now, and...

A New Era in eDiscovery? Framing Market Growth Through the Lens of Six Eras

There are many excellent resources for considering chronological and historiographical approaches...

An eDiscovery Market Size Mashup: 2020-2025 Worldwide Software and Services Overview

While the Compound Annual Growth Rate (CAGR) for worldwide eDiscovery software...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

Five Great Reads on eDiscovery for December 2020

May the peace and joy of the holiday season be with...

Five Great Reads on eDiscovery for November 2020

From market sizing and cyber law to industry investments and customer...

Five Great Reads on eDiscovery for October 2020

From business confidence and captive ALSPs to digital republics and mass...

Five Great Reads on eDiscovery for September 2020

From cloud forensics and cyber defense to social media and surveys,...

It’s a Match! Focusing on the Total Cost of eDiscovery Review with ReviewRight Match

As a leader in remote legal document review, HaystackID provides clients...

From Proactive Detection to Data Breach Reviews: Sensitive Data Discovery and Extraction with Ascema

A steady rise in the number of sensitive data discovery requirements...

A Running List: Top 100+ eDiscovery Providers

Based on a compilation of research from analyst firms and industry...

The eDisclosure Systems Buyers Guide – 2020 Edition (Andrew Haslam)

Authored by industry expert Andrew Haslam, the eDisclosure Buyers Guide continues...

A Lifting of the Fog? Winter 2021 eDiscovery Business Confidence Survey Results

This is the twenty-first quarterly eDiscovery Business Confidence Survey conducted by...

Orion Nebula
A Nebulous Outcome? The Winter 2021 eDiscovery Business Confidence Survey

The eDiscovery Business Confidence Survey is a nonscientific quarterly survey designed...

High Five? An Aggregate Overview of Five Semi-Annual eDiscovery Pricing Surveys

As we are in the midst of a pandemic that has...

Balancing Relevance and Reality? Winter 2021 eDiscovery Pricing Survey Results

Based on the complexity of data and legal discovery, it is...