TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Even with a large pool of participants, ample time, and the ability to hone search queries based on instant feedback, nobody was able to generate a better production than Technology-Assisted Review (TAR) when the same amount of review effort was expended. It seems fair to say that keyword search often requires twice as much document review to achieve a production that is as good as what you would get TAR.

en flag
fr flag
de flag
pt flag
es flag

An extract from an article by eDiscovery expert Bill Dimm

This was by far the most significant iteration of the ongoing exercise where I challenge an audience to produce a keyword search that works better than technology-assisted review (also known as predictive coding or supervised machine learning).  There were far more participants than previous rounds, and a structural change in the challenge allowed participants to get immediate feedback on the performance of their queries so they could iteratively improve them.  A total of 1,924 queries were submitted by 42 participants (an average of 45.8 queries per person) and higher recall levels were achieved than in any prior version of the challenge, but the audience still couldn’t beat TAR.

In previous versions of the experiment, the audience submitted search queries on paper or through a web form using their phones, and I evaluated a few of them live on stage to see whether the audience was able to achieve higher recall than TAR.  Because the number of live evaluations was so small, the audience had very little opportunity to use the results to improve their queries.  In the latest iteration, participants each had their own computer in the lab at the 2019 Ipro Tech Show, and the web form evaluated the query and gave the user feedback on the recall achieved immediately.  Furthermore, it displayed the relevance and important keywords for each of the top 100 documents matching the query, so participants could quickly discover useful new search terms to tweak their queries.  This gave participants a significant advantage over a normal e-discovery scenario, since they could try an unlimited number of queries without incurring any cost to make relevance determinations on the retrieved documents in order to decide which keywords would improve the queries.  The number of participants was significantly larger than any of the previous iterations, and they had a full 20 minutes to try as many queries as they wanted.  It was the best chance an audience has ever had of beating TAR. They failed.

Read the complete article at TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Additional Reading

Source: ComplexDiscovery

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

ComplexDiscovery is an online publication that highlights data and legal discovery insight and intelligence ranging from original research to aggregated news for use by business, information technology, and legal professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding data and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of data and legal discovery organizations. Registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world, ComplexDiscovery OÜ operates virtually worldwide to deliver marketing consulting and services.

A (Brand) New Approach? Considering the Framework and Structure of eDiscovery Offerings

Today’s eDiscovery providers may benefit from the lessons learned in the creation of the Sgt. Pepper’s Lonely Hearts Club Band album by creating a concept for branding and packaging their offerings within that brand in a connected, theme-based way that represents the offerings’ promise and capability in a way that is easy to understand and remember.



Check Out the New Approach Now!

Interested in Contributing?

ComplexDiscovery combines original industry research with curated expert articles to create an informational resource that helps legal, business, and information technology professionals better understand the business and practice of data discovery and legal discovery.

All contributions are invested to support the development and distribution of ComplexDiscovery content. Contributors can make as many article contributions as they like, but will not be asked to register and pay until their contribution reaches $5.

Reynen Court Secures Additional Funding

According to the media release, Reynen Court has secured $4.5 million...

From Proactive Detection to Data Breach Reviews: Sensitive Data Discovery and Extraction with Ascema

A steady rise in the number of sensitive data discovery requirements...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

New from NIST: Integrating Cybersecurity and Enterprise Risk Management (ERM)

NIST has released NISTIR 8286, Integrating Cybersecurity and Enterprise Risk Management...

A Running List: Top 100+ eDiscovery Providers

Based on a compilation of research from analyst firms and industry...

The eDisclosure Systems Buyers Guide – 2020 Edition (Andrew Haslam)

Authored by industry expert Andrew Haslam, the eDisclosure Buyers Guide continues...

The Race to the Starting Line? Recent Secure Remote Review Announcements

Not all secure remote review offerings are equal as the apparent...

Enabling Remote eDiscovery? A Snapshot of DaaS

Desktop as a Service (DaaS) providers are becoming important contributors to...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

Revisions and Decisions? New Considerations for eDiscovery Secure Remote Reviews

One of the key revision and decision areas that business, legal,...

A Macro Look at Past and Projected eDiscovery Market Size from 2012 to 2024

From a macro look at past estimations of eDiscovery market size...

A Season of Change? Eighteen Observations on eDiscovery Business Confidence in the Fall of 2020

In the fall of 2020, 77.2% of eDiscovery Business Confidence Survey...

The Continuing Case of Budgetary Constraints in the Business of eDiscovery

In the fall of 2020, 49.4% of respondents viewed budgetary constraints...

Outstanding Accounts? eDiscovery Operational Metrics in the Fall of 2020

In the fall of 2020, eDiscovery Business Confidence Survey more...

Holding the Rudder? Fall 2020 eDiscovery Business Confidence Survey Results

This is the twentieth quarterly eDiscovery Business Confidence Survey conducted by...

Reynen Court Secures Additional Funding

According to the media release, Reynen Court has secured $4.5 million...

DISCO Raises $60 Million

According to the media release, DISCO will use this investment to...

Rampiva and the RYABI Group Merge

According to today's announcement, the RYABI Group merger is Rampiva's first...

eDiscovery Mergers, Acquisitions, and Investments in Q3 2020

From HaystackID and NightOwl Global to Reveal Data and NexLP, the...

Five Great Reads on eDiscovery for October 2020

From business confidence and captive ALSPs to digital republics and mass...

Five Great Reads on eDiscovery for September 2020

From cloud forensics and cyber defense to social media and surveys,...

Five Great Reads on eDiscovery for August 2020

From predictive coding and artificial intelligence to antitrust investigations and malware,...

Five Great Reads on eDiscovery for July 2020

From business confidence and operational metrics to data protection and privacy...