Sun. Jun 26th, 2022
    2010-2018 ARCHIVED CONTENT
    You are viewing ARCHIVED CONTENT released online from 1 April, 2010 to August 24, 2018. Content in this archive site is NOT UPDATED, and links may not function. For current information, go to ComplexDiscovery.com.

    By Ralph Losey

    Scientific Experiments on Inconsistencies of Relevance Determinations in Large Scale Document Reviews

    The base work in this area was done by Ellen M. Voorhees: Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 Info. Processing & Mgmt  697 (2000). The second study of interest to lawyers on this subject came ten years later by Herbert L. Roitblat, Anne Kershaw and Patrick Oot, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, Journal of the American Society for Information Science and Technology, 61 (2010) (draft found at Clearwell Systems). The next study with significant new data on inconsistent relevance review determinations was by Maura Grossman and  Gordon Cormack in 2012: Inconsistent Assessment of Responsiveness in E-Discovery: Difference of Opinion or Human Error?, 32 Pace L. Rev. 267 (2012); also see Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual ReviewXVII RICH. J.L. & TECH. 11 (2011). The fourth and last study on the subject with new data is my own review experiment done in 2012 and 2013. A Modest Contribution to the Science of Search: Report and Analysis of Inconsistent Classifications in Two Predictive Coding Reviews of 699,082 Enron Documents (2013)

     

    Have a Request?

    If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

    ComplexDiscovery is an online publication that highlights cyber, data, and legal discovery insight and intelligence ranging from original research to aggregated news for use by cybersecurity, information governance, and eDiscovery professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding cyber, data, and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

    ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of cyber, data, and legal discovery organizations. Focused primarily on supporting the ComplexDiscovery publication, the company is registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world. The company operates virtually worldwide to deliver marketing consulting and services.

    Early Lessons from the Cyber War: A New Microsoft Report on Defending Ukraine

    According to a new report from Microsoft, the Russian invasion relies...

    From Continuity to Culture? Preserving and Securing Ukrainian Public and Private Sector Data

    Highlighted by ComplexDiscovery prior to the start of the current Ukrainian...

    Considering Access Control Policy Models? Blockchain for Access Control Systems (NIST)

    As current information systems and network architectures evolve to be more...

    Friends in Low Places? The 2022 Data Breach Investigations Report from Verizon

    The 15th Annual Data Breach Investigations Report (DBIR) from Verizon looked...

    TCDI to Acquire Aon’s eDiscovery Practice

    According to TCDI Founder and CEO Bill Johnson, “For 30 years,...

    Smarsh to Acquire TeleMessage

    “As in many other service industries, mobile communication is ubiquitous in...

    A Milestone Quarter? DISCO Announces First Quarter 2022 Financial Results

    According to Kiwi Camara, Co-Founder and CEO of DISCO, “This quarter...

    New from Nuix? Macquarie Australia Conference 2022 Presentation and Trading Update

    From a rebalanced leadership team to three concurrent horizons to drive...

    On the Move? 2022 eDiscovery Market Kinetics: Five Areas of Interest

    Recently ComplexDiscovery was provided an opportunity to share with the eDiscovery...

    Trusting the Process? 2021 eDiscovery Processing Task, Spend, and Cost Data Points

    Based on the complexity of cybersecurity, information governance, and legal discovery,...

    The Year in Review? 2021 eDiscovery Review Task, Spend, and Cost Data Points

    Based on the complexity of cybersecurity, information governance, and legal discovery,...

    A 2021 Look at eDiscovery Collection: Task, Spend, and Cost Data Points

    Based on the complexity of cybersecurity, information governance, and legal discovery,...

    Five Great Reads on Cyber, Data, and Legal Discovery for June 2022

    From eDiscovery ecosystem players and pricing to data breach investigations and...

    Five Great Reads on Cyber, Data, and Legal Discovery for May 2022

    From eDiscovery pricing and buyers to cyberattacks and incident response, the...

    Five Great Reads on Cyber, Data, and Legal Discovery for April 2022

    From cyber attack statistics and frameworks to eDiscovery investments and providers,...

    Five Great Reads on Cyber, Data, and Legal Discovery for March 2022

    From new privacy frameworks and disinformation to business confidence and the...

    Hot or Not? Summer 2022 eDiscovery Business Confidence Survey

    Since January 2016, 2,701 individual responses to twenty-six quarterly eDiscovery Business...

    Inflection or Deflection? An Aggregate Overview of Eight Semi-Annual eDiscovery Pricing Surveys

    Initiated in the winter of 2019 and conducted eight times with...

    Feeding the Frenzy? Summer 2022 eDiscovery Pricing Survey Results

    Initiated in the winter of 2019 and conducted eight times with...

    Surge or Splurge? Eighteen Observations on eDiscovery Business Confidence in the Spring of 2022

    In the spring of 2022, 63.5% of survey respondents felt that...