ARCHIVED CONTENT
You are viewing ARCHIVED CONTENT released online between 1 April 2010 and 24 August 2018 or content that has been selectively archived and is no longer active. Content in this archive is NOT UPDATED, and links may not function.A Challenge with Technology-Assisted Review
Provided for your review is a short but important comment from a recent LinkedIn Group discussion on the evaluation of machine learning protocols for technology-assisted review. The discussion originated with the availability of a new paper published by well known technology-assisted review experts, Attorney Maura Grossman and Professor Gordon Cormack. The paper, “Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, was prepared for The 37th Annual ACM SIGIR Conference on Research and Development in Information Retrieval.
The comment shared (see below) addressed key points in the paper and advanced the discussion beyond the evaluation of machine learning techniques into the questioning of a greater issue with technology-assisted review. That issue, as stated in the comment, is the fact that current technology-assisted review systems are limited to text.
This comment is worthy of careful consideration as it:
- Highlights the fact that this limitation calls into question the true effectiveness of text-based technology-assisted review tools.
- Begs the question of what classification tools are the best choice for technology-assisted review if almost all current tools are in fact limited to text.
Comment by Joe Howie of BeyondRecognition*
Interesting – Cormack and Grossman define the “TAR Problem” as not knowing about a data set at the outset (p. 2); point out that simple key term searching to select the seed set improves the performance of all TAR protocols (p. 7); and indicate that some protocols have difficulty with low prevalence or low richness collections (p.9). Visual classification impacts those issues plus addresses the implicit issue that TAR is text-restricted (i.e., should actually be called TR-TAR).
Visual classification classifies visually similar documents whether or not text is contained in them, and reviewing one document per classification provides an awareness of a collection’s content that overcomes a possible initial lack of knowledge. The classifications also provide a far more granular and meaningful tool for selecting documents than simple text searching. By eliminating plainly unresponsive document types, reviewers can increase the richness of the remaining documents, and may in fact be able to identify which specific attributes which make documents responsive, thereby short cutting the need to “predict” which documents are responsive.
Since the last century, text analysis has been the primary tool used to classify documents, but its durability as the tool of choice doesn’t mean that it remains the best choice.
* The comment was used by express permission of its contributor, Joe Howie of BeyondRecognition.
Future Consideration
Like many, I find the discussion and commentary on this issue quite interesting and certainly expect that the issue raised in the comment will spark relevant discussion among providers, consumers and commentators on the topic of technology-assisted review.