TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Even with a large pool of participants, ample time, and the ability to hone search queries based on instant feedback, nobody was able to generate a better production than Technology-Assisted Review (TAR) when the same amount of review effort was expended. It seems fair to say that keyword search often requires twice as much document review to achieve a production that is as good as what you would get TAR.

en flag
fr flag
de flag
pt flag
es flag

An extract from an article by eDiscovery expert Bill Dimm

This was by far the most significant iteration of the ongoing exercise where I challenge an audience to produce a keyword search that works better than technology-assisted review (also known as predictive coding or supervised machine learning).  There were far more participants than previous rounds, and a structural change in the challenge allowed participants to get immediate feedback on the performance of their queries so they could iteratively improve them.  A total of 1,924 queries were submitted by 42 participants (an average of 45.8 queries per person) and higher recall levels were achieved than in any prior version of the challenge, but the audience still couldn’t beat TAR.

In previous versions of the experiment, the audience submitted search queries on paper or through a web form using their phones, and I evaluated a few of them live on stage to see whether the audience was able to achieve higher recall than TAR.  Because the number of live evaluations was so small, the audience had very little opportunity to use the results to improve their queries.  In the latest iteration, participants each had their own computer in the lab at the 2019 Ipro Tech Show, and the web form evaluated the query and gave the user feedback on the recall achieved immediately.  Furthermore, it displayed the relevance and important keywords for each of the top 100 documents matching the query, so participants could quickly discover useful new search terms to tweak their queries.  This gave participants a significant advantage over a normal e-discovery scenario, since they could try an unlimited number of queries without incurring any cost to make relevance determinations on the retrieved documents in order to decide which keywords would improve the queries.  The number of participants was significantly larger than any of the previous iterations, and they had a full 20 minutes to try as many queries as they wanted.  It was the best chance an audience has ever had of beating TAR. They failed.

Read the complete article at TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Additional Reading

Source: ComplexDiscovery

Business as Unusual? Eighteen Observations on eDiscovery Business Confidence in the Summer of 2020

The results of the recent Summer 2020 eDiscovery Business Confidence Survey present the unfortunate and continuing impact of COVID-19 on the business of eDiscovery. However, for these pandemic-driven results to be fully understood, they should be viewed through the contextual lens of the results of all nineteen surveys that have been administered to eDiscovery professionals since the inception of the eDiscovery Business Confidence Survey in early 2016.



Check Out the Observations Now!

ComplexDiscovery combines original industry research with curated expert articles to create an informational resource that helps legal, business, and information technology professionals better understand the business and practice of data discovery and legal discovery.

All contributions are invested to support the development and distribution of ComplexDiscovery content. Contributors can make as many article contributions as they like, but will not be asked to register and pay until their contribution reaches $5.

Reveal Acquires NexLP

According to Jay Leib, Co-Founder and CEO of NexLP, "We chose...

Predictive Coding Technologies and Protocols: Fall 2020 Survey

The Predictive Coding Technologies and Protocols Survey is a non-scientific semi-annual...

Sharing is Caring? ayfie Group Lists on Merkur Market of Oslo Stock Exchange

According to Johannes Stiehler, CEO of ayfie Group, in a July...

XDD Acquires Anexsys

According to David Moran, XDD President and COO, “Complementing our recent...

A Running List: Top 100+ eDiscovery Providers

Based on a compilation of research from analyst firms and industry...

The eDisclosure Systems Buyers Guide – 2020 Edition (Andrew Haslam)

Authored by industry expert Andrew Haslam, the eDisclosure Buyers Guide continues...

The Race to the Starting Line? Recent Secure Remote Review Announcements

Not all secure remote review offerings are equal as the apparent...

Enabling Remote eDiscovery? A Snapshot of DaaS

Desktop as a Service (DaaS) providers are becoming important contributors to...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

Revisions and Decisions? New Considerations for eDiscovery Secure Remote Reviews

One of the key revision and decision areas that business, legal,...

A Macro Look at Past and Projected eDiscovery Market Size from 2012 to 2024

From a macro look at past estimations of eDiscovery market size...

An eDiscovery Market Size Mashup: 2019-2024 Worldwide Software and Services Overview

While the Compound Annual Growth Rate (CAGR) for worldwide eDiscovery software...

Predictive Coding Technologies and Protocols: Fall 2020 Survey

The Predictive Coding Technologies and Protocols Survey is a non-scientific semi-annual...

Business as Unusual? Eighteen Observations on eDiscovery Business Confidence in the Summer of 2020

Based on the aggregate results of nineteen past eDiscovery Business Confidence...

A Growing Concern? Budgetary Constraints and the Business of eDiscovery

In the summer of 2020, 56% of respondents viewed budgetary constraints...

A Change in Tempo? eDiscovery Operational Metrics in the Summer of 2020

In the summer of 2020, 91 eDiscovery Business Confidence Survey participants...

Reveal Acquires NexLP

According to Jay Leib, Co-Founder and CEO of NexLP, "We chose...

Sharing is Caring? ayfie Group Lists on Merkur Market of Oslo Stock Exchange

According to Johannes Stiehler, CEO of ayfie Group, in a July...

XDD Acquires Anexsys

According to David Moran, XDD President and COO, “Complementing our recent...

HaystackID and NightOwl Global Merge

According to today's announcement, the NightOwl merger is HaystackID's fourth major...

Five Great Reads on eDiscovery for July 2020

From business confidence and operational metrics to data protection and privacy...

Five Great Reads on eDiscovery for June 2020

From collection market size updates to cloud outsourcing guidelines, the June...

Five Great Reads on eDiscovery for May 2020

From review market sizing revisions to pandemeconomic pricing, the May 2020...

Five Great Reads on eDiscovery for April 2020

From business confidence to the boom of Zoom, the April 2020...

[New Survey]
[New Survey]