TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Even with a large pool of participants, ample time, and the ability to hone search queries based on instant feedback, nobody was able to generate a better production than Technology-Assisted Review (TAR) when the same amount of review effort was expended. It seems fair to say that keyword search often requires twice as much document review to achieve a production that is as good as what you would get TAR.

en flag
fr flag
de flag
pt flag
es flag

An extract from an article by eDiscovery expert Bill Dimm

This was by far the most significant iteration of the ongoing exercise where I challenge an audience to produce a keyword search that works better than technology-assisted review (also known as predictive coding or supervised machine learning).  There were far more participants than previous rounds, and a structural change in the challenge allowed participants to get immediate feedback on the performance of their queries so they could iteratively improve them.  A total of 1,924 queries were submitted by 42 participants (an average of 45.8 queries per person) and higher recall levels were achieved than in any prior version of the challenge, but the audience still couldn’t beat TAR.

In previous versions of the experiment, the audience submitted search queries on paper or through a web form using their phones, and I evaluated a few of them live on stage to see whether the audience was able to achieve higher recall than TAR.  Because the number of live evaluations was so small, the audience had very little opportunity to use the results to improve their queries.  In the latest iteration, participants each had their own computer in the lab at the 2019 Ipro Tech Show, and the web form evaluated the query and gave the user feedback on the recall achieved immediately.  Furthermore, it displayed the relevance and important keywords for each of the top 100 documents matching the query, so participants could quickly discover useful new search terms to tweak their queries.  This gave participants a significant advantage over a normal e-discovery scenario, since they could try an unlimited number of queries without incurring any cost to make relevance determinations on the retrieved documents in order to decide which keywords would improve the queries.  The number of participants was significantly larger than any of the previous iterations, and they had a full 20 minutes to try as many queries as they wanted.  It was the best chance an audience has ever had of beating TAR. They failed.

Read the complete article at TAR vs. Keyword Search Challenge, Round 6 (Instant Feedback)

Additional Reading

Source: ComplexDiscovery

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

ComplexDiscovery is an online publication that highlights data and legal discovery insight and intelligence ranging from original research to aggregated news for use by business, information technology, and legal professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding data and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of data and legal discovery organizations. Registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world, ComplexDiscovery OÜ operates virtually worldwide to deliver marketing consulting and services.

A (Brand) New Approach? Considering the Framework and Structure of eDiscovery Offerings

Today’s eDiscovery providers may benefit from the lessons learned in the creation of the Sgt. Pepper’s Lonely Hearts Club Band album by creating a concept for branding and packaging their offerings within that brand in a connected, theme-based way that represents the offerings’ promise and capability in a way that is easy to understand and remember.

This fictionalized branding approach was developed from the intellectual exercise of trying to figure out a reasonable and memorable way to descriptively highlight the promise and capabilities of offerings typically delivered by full-service eDiscovery providers. It may not be completely comprehensive or fully normalized. However, the hope of sharing this branding example is that it might help those involved in the branding and communication of eDiscovery provider services and solutions.

First Legal Acquires eDiscovery Provider Redpoint Technologies

According to Alex Martinez, CEO of First Legal, “Both First Legal...

Veristar Acquires Planet Data

According to Veristar company founder, CEO, and president Rick Avers, “We...

Questel Acquires doeLEGAL

doeLEGAL today announced that it has been acquired by intellectual property...

Following the Money? Mike Bryant Provides a SOLID Look at Legal Tech Merger and Acquisition Activity

From seed and venture capital investments to private equity and Special...

A New Era in eDiscovery? Framing Market Growth Through the Lens of Six Eras

There are many excellent resources for considering chronological and historiographical approaches...

An eDiscovery Market Size Mashup: 2020-2025 Worldwide Software and Services Overview

While the Compound Annual Growth Rate (CAGR) for worldwide eDiscovery software...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

Five Great Reads on eDiscovery for February 2021

From litigation trends and legal tech investing to facial recognition and...

Five Great Reads on eDiscovery for January 2021

From eDiscovery business confidence and operational metrics to merger and acquisition...

Five Great Reads on eDiscovery for December 2020

May the peace and joy of the holiday season be with...

Five Great Reads on eDiscovery for November 2020

From market sizing and cyber law to industry investments and customer...

HaystackID Recognized in IDC MarketScape for eDiscovery Services

According to HaystackID CEO Hal Brooks, “We are proud to once...

A Generational View of Remote Security? HaystackID™ Releases 3.0 Security Enhancements to Review Technology

According to HaystackID's Senior Vice President and General Manager for Review...

Only a Matter of Time? HaystackID Launches New Service for Data Breach Discovery and Review

According to HaystackID's Chief Innovation Officer and President of Global Investigations,...

It’s a Match! Focusing on the Total Cost of eDiscovery Review with ReviewRight Match

As a leader in remote legal document review, HaystackID provides clients...

Cold Weather Catch? Predictive Coding Technologies and Protocols Survey – Spring 2021 Results

The Predictive Coding Technologies and Protocols Survey is a non-scientific semi-annual...

Out of the Woods? Eighteen Observations on eDiscovery Business Confidence in the Winter of 2021

In the winter of 2021, 85.0% of eDiscovery Business Confidence Survey...

Issues Impacting eDiscovery Business Performance: A Winter 2021 Overview

In the winter of 2021, 43.3% of respondents viewed budgetary constraints...

Not So Outstanding? eDiscovery Operational Metrics in the Winter of 2021

In the winter of 2021, eDiscovery Business Confidence Survey more...