Technology-Assisted Review

Bill Dimm on the TAR vs. Keyword Search Challenge

Bill Dimm on the TAR vs. Keyword Search Challenge

During my [Bill Dimm] presentation at the NorCal eDiscovery & IG Retreat, I challenged the audience to create keyword searches that would work better than technology-assisted review (predictive coding) for two topics.  Half of the room was tasked with finding articles about biology (science-oriented articles, excluding medical treatment) and the other half searched for articles about current law (excluding proposed laws or politics).  TAR beat keyword search across the board for both tasks. 


TAR Course Expands Again: Standardized Best Practice for Technology Assisted Review

TAR Course Expands Again: Standardized Best Practice for Technology Assisted Review

Ralph Losey’s TAR Course has a new class, the Seventeenth Class: Another “Player’s View” of the Workflow. Several other parts of the course have also been updated and edited. The TAR course now has eighteen classes.


SuperCALifragilisticexpialidocious: An Update on One Technology-Assisted Review Protocol’s Terminology

SuperCALifragilisticexpialidocious: An Update on One Technology-Assisted Review Protocol's Terminology

With recent eDiscovery provider announcements that highlight the use of the terms “Continuous Active Learning” and “CAL”, provided below is a quick review and current update on Recommind’s (Recommind, Inc.) opposition to the trademark/service mark application by Maura Grossman and Gordan V. Cormack for trademarking CONTINUOUS ACTIVE LEARNING and CAL.


TAR for Smart Chickens

TAR for Smart Chickens

Special Master Maura Grossman recently issued an Order crafting a new validation protocol In Re Broiler Chicken Antitrust Litigation, (Jan. 3, 2018), which is currently pending in the Northern District of Illinois.


Iterated Four-Step Work Flow for Active Machine Training to Help Attorneys Locate Relevant Evidence

Iterated Four-Step Work Flow for Active Machine Training to Help Attorneys Locate Relevant Evidence

In each round of training, including the first, document reviewers are only looking at a few hundred, to at most, a few thousand documents. A typical complex project takes about ten rounds of machine training to complete by finding all of the relevant documents required.


Comparing 179 Machine Learning Categorizers on 121 Data Sets

Comparing 179 Machine Learning Categorizers on 121 Data Sets

In a monumental study, Fernández-Delgado and colleagues tested 179 machine learning categorizers on 121 data sets. They found that a large majority of them, were essentially identical in their accuracy.


Analysis of Important New Case on Predictive Coding by a Rising New Judicial Star: “Winfield v. City of New York”

Analysis of Important New Case on Predictive Coding by a Rising New Judicial Star: “Winfield v. City of New York”

Judge Parker’s Order in Winfield v. City of New York (15-CV-05236) resolved a discovery dispute, which, among other things, challenged the Defendant City’s predictive coding process, in other words, it’s machine learning search.


‘Document Tsunami’ Driving Lawyers to Upskill in TAR

‘Document Tsunami’ Driving Lawyers to Upskill in TAR

A prominent Victorian judge has highlighted the growing need for lawyers to expand their knowledge on technology-assisted review (TAR) workflows, noting that if they don’t they’re going to be deficient in their capacity to service clients.


Predictive Coding and the Stop Decision

Predictive Coding and the Stop Decision

The stop decision is the most difficult decision in predictive coding. The decision must be made in all types of predictive coding methods, not just our Predictive Coding 4.0.


Confirming Recall Adequacy With Unbiased Multi-Stage Acceptance Testing

Confirming Recall Adequacy With Unbiased Multi-Stage Acceptance Testing

This paper by Bill Dimm of Hot Neuron proposes a multi-stage acceptance testing procedure that avoids bias and actually works in practice. The amount of document review required with the new method depends on the level of recall actually achieved.


The Negative Consequences of ‘Da SIlva Moore’

The Negative Consequences of ‘Da SIlva Moore’

Predictive coding methods have come a long way since Judge Peck first approved predictive coding in our Da Silva Moore case. This failure to move on past the Predictive Coding 1.0 methods of Da Silva Moore, is, I suspect, one of the major reasons that predictive coding has never really caught on. In fact, the most successful document review software developers since 2012 have ignored predictive coding altogether.


Calling an End to Culling: Predictive Coding and the New Federal Rules of Civil Procedure

Calling an End to Culling: Predictive Coding and the New Federal Rules of Civil Procedure

Published in the Richmond Journal of Law & Technology, the paper Calling an End to Culling: Predictive Coding and the New Federal Rules of Civil Procedure by Stephanie Serhan provides cogent considerations and conclusions as to the of use predictive coding and the timing of that use in relation to keyword culling.


Ask Catalyst: A User’s Guide to TAR (For Smart People)

Ask Catalyst: A User's Guide to TAR (For Smart People)

Ask Catalyst: A User’s Guide to TAR provides a cogent overview of the effective use of TAR 2.0 in litigation and investigations. It is loaded with practical advice for practitioners at all levels about how this great technology can save time, money and effort.


e-Discovery Team’s 2016 TREC Report

e-Discovery Team’s 2016 TREC Report

One of the core purposes of all of the Tracks is to demonstrate the robustness of core retrieval technology. Moreover, one of the primary goals of TREC is: [T]o speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems.


The Twenty-Fifth Text REtrieval Conference (TREC 2016) Proceedings

The Twenty-Fifth Text REtrieval Conference (TREC 2016) Proceedings

The proceedings of the TREC Total Recall Track have been published by the National Institute of Standards and Technology. The purpose of track was to investigate methods and technologies to find, as nearly as possible, all documents in a collection that satisfy specific criteria, with reasonable effort.


The Top Twenty-Two Most Interesting e-Discovery Opinions of 2016

The Top Twenty-Two Most Interesting e-Discovery Opinions of 2016

The e-Discovery Team’s top twenty-two most interesting cases in 2016. We provide an analysis and key quotes of each, lessons learned and, where appropriate, practice pointers. We also explain why we find these opinions interesting.


Technology-Assisted Review in Australia: Discovering The Future

Technology-Assisted Review in Australia: Discovering The Future

The Victorian Supreme Court will issue a practice note about the use of TAR on 1 January 2017. We understand it will be the first court in Australia to do so. We expect that other Australian courts will follow suit in issuing a practice note, and it will be interesting to follow the approaches taken by other Australian courts.


Combination of Technology and Humans Work Best in E-Discovery Review, Study Finds

Combination of Technology and Humans Work Best in E-Discovery Review, Study Finds

The biggest takeaway of the joint research project by nonprofit Electronic Discovery Institute and tech giant Oracle Corp. is that TAR is often faster and cheaper when identifying relevant documents. But when it comes to isolating privileged or sensitive information, human reviewers outperformed machines.


Predictive Coding 4.0 – Nine Key Points of Legal Document Review – Part Six

Predictive Coding 4.0 – Nine Key Points of Legal Document Review – Part Six

Although presented as sequential steps for pedantic purposes, Predictive Coding 4.0 is highly adaptive to circumstances and does not necessarily follow a rigid linear order.


F1 Scores and Technology-Assisted Review: Automation in eDiscovery

F1 Scores and Technology-Assisted Review: Automation in eDiscovery

Provided for your review and use is the complete presentation slide deck from the recent Masters Conference keynote panel on automation in eDiscovery.


Predictive Coding 4.0 – Nine Key Points of Legal Document Review – Part Five

Predictive Coding 4.0 – Nine Key Points of Legal Document Review – Part Five

The predictive coding method can fail spectacularly with a poor expert, but so can keyword search.


CAL Trademark Challenged

CAL Trademark Challenged

After abandoning its claim of trademark over “predictive coding” in 2011, Recommind is now challenging the trademark of “continuous active learning” and its acronym, “CAL”.


Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow – Part Four

Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow – Part Four

I cannot tell you how many times I see the word “complaint” in their keyword list. The guessing involved reminds me of the child’s game of Go Fish.


Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow – Part Three

Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow – Part Three

The goal of the pro-machine approach of Professors Cormack and Grossman, and others, is to minimize human judgments, no matter how skilled, and thereby reduce as much as possible the influence of human error and outright fraud.