As I recall...

We at Valora are occasionally called upon to evaluate the results of traditional document review (read: manual, doc-by-doc review efforts), as a sort of post-project audit.  We use the usual precision and recall metrics to determine whether a document should have been tagged at all and whether it was done so correctly.  While both these measures are extremely important, the truth is it’s really all about the recall.  Let me explain.

If you are supposed to mark 1,000 documents as privileged, and your team only tags 200, does their precision on those 200 really matter?  In layman’s terms:  Do you really care how well those 200 docs scored on accuracy when the reviewers missed 800 documents in the first place?

This is not to suggest that there isn’t a crucial role for precision scoring.  There is!  But, not if you blow it on recall first.  In our above example, if the review team indeed found 1,000 documents and marked them privileged, you would surely wish to score the accuracy of those markings.  But, only if the recall fell within 20% +/- of the intended target.  Well below (or above) this mark, precision is useless info.  So, until as an industry we can demonstrate reliable recall on a regular basis, precision will just have to wait its turn.

Automated Review Reaches National Economic (& Comedic) Proportions

As a big fan of both The Daily Show and Fareed  Zakaria, imagine my surprise and delight when Jon Stewart’s guest, Zakaria, recently told the live studio audience that Automation was changing the way legal work, specifically discovery, was being conducted today!  Perhaps the most interesting quote was

 “…machines can do things that people used to. There’s now computer programs that can do stuff that lawyers used to be able to do, discovery and things like that.  May not be such a bad thing!”

Well, AutoReveiw has now officially reached national, international really, proportions for both economic analysis and comedic value.  Sounds like we’ve about “crossed the chasm” to me.  For the full June 7, 2011 Stewart-Zakaria interview clip, please see Valora’s website, here.

[source]

"I Can See Clearly Now..."

Many years ago, early in my career, I worked at software powerhouse, Symantec. As they recently announced their intent to acquire Clearwell, I thought I would pass along my insider’s 2 cents on the topic. I became a "Symantian" in 1996 after they acquired a smaller company with a very hot product in a growing market. At the time the company was called Delrina and they made WinFax. You likely had a copy at some point. It was the innovator in using modems to send faxes and make VOIP calls. (I worked on the telephony end - the cooler part!)

In retrospect, the CW deal sounds very similar to the Delrina one and a good example of how Symantec operates. They bought Delrina at the height of its popularity, paying top dollar. Then they decimated the division, laying off more than half the workforce. (I was not laid off, but asked instead to become GM of a division.) They basically never put another dime into development, QA or marketing and rode the product's margins all the way into the sunset until it was obsolete. I suspect they made a fortune as there were few costs beyond the initial acquisition. They picked up the entire user base and locked out all competition. Within 3 years WinFax as a product was gone and it was completely embedded with other Symantec suite products.

Within 12 months, no one will be charging $300/GB to cull with Clearwell. Every client will have it embedded in their enterprise suite and simply perform the culling themselves. 3rd party vendors beware. The Clearwell party is over and the Relativity party is next.

All Aboard the ALSP Train!

Next year I will have the distinct pleasure of serving on the national board of the Association of Litigation Support Professionals (ALSP). Although I have been on several boards in the past (including Valora’s, of course) and I have been involved in countless litigation industry associations, I have never had the option to combine those two, and I am eager to give it a go.

Serving on a board is a strange mix of honor and stress, much like parenting. You give your best ideas, efforts (and often money) to the organization on whose board you sit and you represent that honorable group to the community. You are a walking embodiment of the organization itself. Its goals become your badges of honor and its achievements become your glories.

At the same time, you absorb the pain and stress that comes from non-profit organizations struggling to achieve impossible things, with impossible budgets and ever-competing mindshare among constituents. Just like above, the organization’s troubles become your troubles and you come to represent its unseen drama and turmoil.

As for the litigation industry, ALSP holds a unique position. There are lots of regional and job classification groups, but none that span the gamut like ALSP. ALSP includes anyone involved in litigation, be they vendors, attorneys, paralegals, litigation support personnel, government agencies and corporate GC’s. All it takes to be a part of ALSP is a willingness to belong and to contribute to the common good. There is no exclusivity and no hierarchy. Simply an egalitarian group aiming to better the profession.

I look forward to serving on the board of such an honorable and class-free institution. Here’s to you, ALSP! Thanks for having me aboard.

Stuck in the Past

Remember rotary phones? Black & white tv? Vinyl albums? While these have all been long ago replaced by better solutions, there are still many of them in actual use today. It’s hard to imagine, but did you know that the old touch phone (last updated in 1968) is still being produced and sold today1?

I am mentioning this because it is analogous to what is going on with electronic discovery and duplicate files. Up to 10% of electronic discovery populations are never de-duped at all, according to a recent survey of top EDD providers. Valora was fortunate to participate in the survey and see early access to its results, but they weren’t pretty. It seems that nearly half of all electronic discovery populations get, at best, a simple within custodian de-dupe effort, while the results of cross-custodian de-duping are clearly spelled out in financial terms.

Why is this? Is it just that some customers are obstinate and refuse to try anything new? I doubt it. I think it is the same reason that people still use their rotary phones and listen to records. They are comfortable, familiar, safe and well understood. Simple removal of duplicates inside a single custodian, or no removal of duplicates is comfortable and easy to understand. Furthermore, it cements the idea that all files or documents need to be assessed individually – a belief that the legal community seems terrified to let go, even though all signs point to its eventual demise. Those who would cling to no de-duping or inferior de-duping are the same people who will avoid population analytics and automated review. Be glad they practice law and not medicine.

PS. In case you were wondering: Almost 1 billion vinyl albums were sold in 2007, up 15% from 20062.

-Sandy

[1] Source: Wikipedia http://en.wikipedia.org/wiki/Model_500_telephone

[2] Source: Time magazine http://www.time.com/time/magazine/article/0,9171,1702369,00.html