Objectifying the Subjective: Evaluating eDiscovery Vendor Viability

This updated article highlights one approach to evaluating the selection of eDiscovery vendors that may help decision makers by providing an objective framework useful for the competitive comparison and contrasting of subjective evaluation elements. While the components and ratings used in the approach may be adjusted and weighted based on the experiences and preference of evaluators, the framework is beneficial for ensuring a holistic evaluation process for determining vendor viability.

en flag
fr flag
de flag
pt flag
es flag

Definition: Viable – capable of success or continuing effectiveness. (The Free Dictionary)

eDiscovery is rife with risk. Multiple parties, many jurisdictions, numerous ESI formats, various technologies, and competing vendors are all elements that can contribute to potential risk in an eDiscovery matter. With the inherent risk of eDiscovery influenced by these elements, it is essential that organizational eDiscovery decision makers both understand the risk of these elements and seek to mitigate as much risk as possible as early as possible.

One area that has the potential to impact the risk of an eDiscovery matter is an organization’s selection of an eDiscovery provider. There are many providers with many technologies and many ways in which they deliver their people, processes, and products to clients. In considering vendor selection, much is often shared about the specific capability and cost of technology and people to solve client challenges. This sharing on specific capability and pricing is usually presented in a manner that:

  1. Speaks to the capability of a vendor’s technology and people to perform specific tasks as needed by the client.
  2. Specifies the pricing for the delivery of the technology and talent to support the client’s stated need.

This focus on the technology and talent elements of a vendor’s capability is undoubtedly warranted as these elements ultimately provide the cutting edge for the knife of eDiscovery task execution. However, just as there is much more to the utility of a knife than its edge (especially if you want to use it more than once), there are additional areas worthy of consideration in vendor selection if one is considering the long term strategic utility and viability of a vendor.

Strategically Comparing Vendors

To help eDiscovery decision makers evaluate and compare both the immediate and long term success potential of electronic discovery vendors, the following considerations, shared in a formulaic manner to allow for objective comparison of subjective evaluation criteria, are provided for your review. The author is well aware that there are many considerations required to comprehensively evaluate electronic discovery vendors and the following considerations represent only one view of how one can evaluate and compare vendors beyond just the “feature, function and pricing” of services and products.

Baseline Definitions and Rating Criteria:

Capability Criteria

  • Technology = Demonstrated ability of product/service offering’s technical contribution to solving specific electronic discovery challenges. Simplistically rated in this model as below market expectations = 0, at market expectations = 2, or exceeding market expectations = 4.
  • Security = Demonstrated historical and continuing commitment to security and privacy based audits, investigations, certifications, and attestations. Simplistically rated in this model as below market expectations = 0, at market expectations = 2, or exceeding market expectations = 4.
  • Domain Knowledge = Demonstrated ability of an organization to utilize technology to solve specific electronic discovery challenges. Simplistically rated in this model as below market expectations = 0, at market expectations = 2, or exceeding market expectations = 4.

Communication Criteria

  • Reputation = The confidence level customers have in the actual or perceived ability of an organization to solve electronic discovery challenges viewed in relation to other organizations. Simplistically rated in this model as having a reputation for not satisfactorily meeting electronic discovery challenges = 0, having a reputation for sometimes satisfactorily meeting electronic discovery challenges = 2, or having a reputation for satisfactorily accomplishing electronic discovery challenges a majority of the time = 4.
  • Integrity: Demonstrated honest interaction and market messaging based on topical due diligence of organizational leadership, leadership assertions, and market messaging based on searches of business and legal databases for incongruencies between representation and reality. Simplistically rated in this model as representations do not represent reality = (-2), no real evidence congruency representations and reality = 0,  or representations are congruent with reality = 2.
  • Awareness = An organization’s mindshare in the eyes of the customer in relation to other organizations seeking to solve similar electronic discovery challenges. Simplistically rated in this model as awareness not prevalent = 0, awareness among the majority of legal and corporate law professionals on a regional level = 1, or awareness among the majority of legal and corporate law professionals on a national level = 2.

Commerce Criteria

  • Customers = The number of active entities that have paid for electronic discovery products/services in the current calendar year. Simplistically rated in this model as having less than 20 customers = (-4), having between 20 and 100 customers = 0, and having greater than 100 customers = 4.
  • Free Cash Flow = Net Income + Depreciation/Amortization – Changes In Working Capital – Capital Expenditure. Simplistically rated in this model as negative free cash flow = (-2), as no free cash flow = 0, and as positive free cash flow = 2.

Authenticity Criteria

  • Motive = The stimulus causing an organization to determine product/service strategies and tactics. Simplistically rated in this model as continuing to provide products/services based on purely financial reasons = (-1), continuing to provide products/services based purely on senior leadership interest in solving electronic discovery challenges = 0, or continuing to provide products/services based on combination of financial reasons and senior leadership interest in solving electronic discovery challenges = 1
  • Sincerity = Congruence or lack thereof between an organization’s stated market desires and actual leadership actions. Simplistically rated in this model as a lack of agreement between internal messaging to senior leadership and messaging to market = (-1) or an agreement between internal messaging to senior leadership and messaging to market = 1.
  • Employee Turnover = The rate at which an organization gains or loses staff. Simplistically rated in this model as a turnover rate higher than 50% annually = (-1), and turnover rate between 25% and 49% annually = 0, or a turnover rate less than 25% annually = 1.

Combined Ratings From Baseline Definitions:

  • Total Success Predictor Rating (TSPR):  The rating of a vendor’s potential for long term viability.  (TSPR = SPR1 + SPR2 + SP3) This can be used to compare a vendor’s potential for long term viability.
  • Success Predictor Rating (SPR):  The rating of a vendor’s capability, communication, commerce and authenticity for a specific time frame.  (SPR = C1 + C2 + C3) This can be used to compare a vendor’s performance (past or current) or potential performance (future) for a specified annual period.
  • Capability Rating (C1): The rating of a vendor’s capability based on technology, security, and domain knowledge.   (C1 = Technology Rating + Security + Domain Knowledge Rating) This can be used to compare a vendor’s technology, security, and domain expertise in relation to market expectations.
  • Communication Rating (C2): The rating of a vendor’s capability based on reputation, integrity, and awareness. (C2 = Reputation Rating + Integrity Rating + Awareness Rating) This can be used to compare a vendor’s ability to share its capability with others and to fulfill on the promise of such sharing.
  • Commerce Rating (C3): The rating of a vendor’s financial stability based on customer activity and free cash flow.  (C3 = Customer Rating + Free Cash Flow Rating) This can be used to compare a vendor’s financial stability through the lens of cash flow and customer base.
  • Authenticity Rating (A1):  The rating of a vendor’s authenticity based on motive, sincerity and employee turnover.  (A1 = Motive Rating + Sincerity Rating + Employee Turnover Rating)  This can be used to compare the alignment of a vendor’s vision with its internal and external actions.

Comparison Formula Quick Look

  • TSPR + SPR1 + SPR2 + SPR3
  • SPR1 = (C1 + C2 + C3 + A1) Past Calendar Year
  • SPR2 = (C1 + C2 + C3 + A1) Current Calendar Year
  • SPR3 = (C1 + C2 + C3 + A1) Next Calendar Year (Forecast)
  • C1 = Capability = Technology + Security + Domain Knowledge
  • C2 = Communication = Reputation + Integrity + Awareness
  • C3 = Commerce = Customers + Free Cash Flow
  • A1 = Authenticity = Motive + Sincerity + Employee Turnover

Comparison In Action:  Practical Comparison

This example represents an application of the aforementioned comparison formulas and any similarity to specific electronic discovery vendors are purely coincidental. Context for this specific evaluation/comparison is a client’s need to select a viable vendor for the completion of a time limited immediate electronic discovery task.

Vendor #1 Description: A Top National Electronic Discovery Vendor is being compared with other electronic discovery vendors for the execution of a time-limited immediate task. This vendor has the technology, security, and domain knowledge that meets market expectations, has strong national awareness and a reputation for satisfactorily meeting electronic discovery challenges, and vendor messaging appears to be in align with actual capabilities. The vendor has more than 100 customers but is experiencing negative cash flow. Vendor’s motive is continuing to provide products and services purely for financial reasons, and the vendor does not have congruence between stated market desires and actual leadership actions. Additionally, this vendor has an employee turnover rate between 25% and 49% annually.

Vendor #1 Success Predictor Rating:

  • Time Limited Immediate Task = SPR2
  • Equation Choice: SPR2 = (C1 + C2 + C3 + A1) Current Calendar Year
  • Equation Components: Technology = 2, Security = 2, Domain Knowledge = 2, Reputation = 4, Integrity = 0, Awareness = 2, Customers = 4, Free Cash Flow = (-2), Motive (-1), Sincerity (-1), Employee Turnover = 0
  • SPR2 Rating: (2+2+2) + (4+0+2) + (4 + -2) + (-1 + -1 + 0) = 12

Vendor #2 Description: A Leading Regional Vendor is also being compared with other electronic discovery vendors for the execution of a time-limited immediate task. This vendor has the technology, security, and domain knowledge that meets market expectations and has a regional awareness and a reputation for satisfactorily accomplishing electronic discovery challenges a majority of the time. However, some past assertions on service features by a key senior leader may be questionable. The vendor has positive free cash flow with more than 20 and less than 100 customers. Vendor’s motive is to provide products and services based on a combination of financial reasons and senior leadership interest in solving electronic discovery challenges, and the vendor has congruence between stated market desires and actual leadership actions. Additionally, this vendor has an employee turnover rate of less than 25%.

Vendor #2 Success Predictor Rating:

  • Time Limited Immediate Task = SPR2
  • Equation Choice: SPR2 = (C1 + C2 + C3 + A1) Current Calendar Year
  • Equation Components: Technology = 2, Security = 2, Domain Knowledge = 2, Reputation = 2, Integrity -1, Awareness = 1, Customers = 2, Free Cash Flow = 2, Motive = 1 Sincerity = 1 Employee Turnover = 1
  • SPR2 Rating: (2+2+2) + (2+-1+1) + (2+2) + (1 + 1 + 1) = 15

Assessment:

Vendor #2 in comparison to Vendor #1 appears to have the capability to execute the task however does not have quite as good a reputation, integrity, or the quantity of customers rating as Vendor #1. This reputation and integrity characteristic however may be offset by the stability of Vendor #2 based on its commerce and authenticity ratings. As Vendor #2 has over 20 customers and Vendor #1 has less than 100 customers, it may be assumed that customer volume differential between Vendor #2 and Vendor #1 may be inconsequential.

Recommended Choice:

For the execution of a time-limited immediate task = Vendor #2. If the desire was to compare vendors for potential partnerships and long term tasks, one would need to work through the Total Success Predictor Rating formula (Success Predictor Ratings For Past, Present, and Future (Forecast)) to determine ratings for comparison/decision.

Additional Considerations

The aforementioned formulaic approach for objectively comparing subjective elements should be viewed as one of many tools available to help determine appropriate choices in eDiscovery vendor selection.  Within this approach, specific areas can be broken down into further sub-components (i.e., technology can be broken down into collections, analytics, processing, review, managed review, etc.) and weighted appropriately to provide further detail in vendor comparisons/evaluations.

References

  • “Viability”. Thefreedictionary.Com, https://www.thefreedictionary.com/viability. Accessed 20 Apr 2019.
  • “Free Cash Flow”. En.Wikipedia.Org, https://en.wikipedia.org/wiki/Free_cash_flow. Accessed 21 Apr 2019.
  • “EDRM”. Edrm.Net, https://www.edrm.net/. Accessed 21 Apr 2019.

Additional Reading

Source: ComplexDiscovery

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

ComplexDiscovery is an online publication that highlights data and legal discovery insight and intelligence ranging from original research to aggregated news for use by business, information technology, and legal professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding data and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of data and legal discovery organizations. Registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world, ComplexDiscovery OÜ operates virtually worldwide to deliver marketing consulting and services.

A (Brand) New Approach? Considering the Framework and Structure of eDiscovery Offerings

Today’s eDiscovery providers may benefit from the lessons learned in the creation of the Sgt. Pepper’s Lonely Hearts Club Band album by creating a concept for branding and packaging their offerings within that brand in a connected, theme-based way that represents the offerings’ promise and capability in a way that is easy to understand and remember.



Check Out the New Approach Now!

Interested in Contributing?

ComplexDiscovery combines original industry research with curated expert articles to create an informational resource that helps legal, business, and information technology professionals better understand the business and practice of data discovery and legal discovery.

All contributions are invested to support the development and distribution of ComplexDiscovery content. Contributors can make as many article contributions as they like, but will not be asked to register and pay until their contribution reaches $5.

Reynen Court Secures Additional Funding

According to the media release, Reynen Court has secured $4.5 million...

From Proactive Detection to Data Breach Reviews: Sensitive Data Discovery and Extraction with Ascema

A steady rise in the number of sensitive data discovery requirements...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

New from NIST: Integrating Cybersecurity and Enterprise Risk Management (ERM)

NIST has released NISTIR 8286, Integrating Cybersecurity and Enterprise Risk Management...

A Running List: Top 100+ eDiscovery Providers

Based on a compilation of research from analyst firms and industry...

The eDisclosure Systems Buyers Guide – 2020 Edition (Andrew Haslam)

Authored by industry expert Andrew Haslam, the eDisclosure Buyers Guide continues...

The Race to the Starting Line? Recent Secure Remote Review Announcements

Not all secure remote review offerings are equal as the apparent...

Enabling Remote eDiscovery? A Snapshot of DaaS

Desktop as a Service (DaaS) providers are becoming important contributors to...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

Revisions and Decisions? New Considerations for eDiscovery Secure Remote Reviews

One of the key revision and decision areas that business, legal,...

A Macro Look at Past and Projected eDiscovery Market Size from 2012 to 2024

From a macro look at past estimations of eDiscovery market size...

A Season of Change? Eighteen Observations on eDiscovery Business Confidence in the Fall of 2020

In the fall of 2020, 77.2% of eDiscovery Business Confidence Survey...

The Continuing Case of Budgetary Constraints in the Business of eDiscovery

In the fall of 2020, 49.4% of respondents viewed budgetary constraints...

Outstanding Accounts? eDiscovery Operational Metrics in the Fall of 2020

In the fall of 2020, eDiscovery Business Confidence Survey more...

Holding the Rudder? Fall 2020 eDiscovery Business Confidence Survey Results

This is the twentieth quarterly eDiscovery Business Confidence Survey conducted by...

Reynen Court Secures Additional Funding

According to the media release, Reynen Court has secured $4.5 million...

DISCO Raises $60 Million

According to the media release, DISCO will use this investment to...

Rampiva and the RYABI Group Merge

According to today's announcement, the RYABI Group merger is Rampiva's first...

eDiscovery Mergers, Acquisitions, and Investments in Q3 2020

From HaystackID and NightOwl Global to Reveal Data and NexLP, the...

Five Great Reads on eDiscovery for October 2020

From business confidence and captive ALSPs to digital republics and mass...

Five Great Reads on eDiscovery for September 2020

From cloud forensics and cyber defense to social media and surveys,...

Five Great Reads on eDiscovery for August 2020

From predictive coding and artificial intelligence to antitrust investigations and malware,...

Five Great Reads on eDiscovery for July 2020

From business confidence and operational metrics to data protection and privacy...