Stuck in the Past

Remember rotary phones? Black & white tv? Vinyl albums? While these have all been long ago replaced by better solutions, there are still many of them in actual use today. It’s hard to imagine, but did you know that the old touch phone (last updated in 1968) is still being produced and sold today1?

I am mentioning this because it is analogous to what is going on with electronic discovery and duplicate files. Up to 10% of electronic discovery populations are never de-duped at all, according to a recent survey of top EDD providers. Valora was fortunate to participate in the survey and see early access to its results, but they weren’t pretty. It seems that nearly half of all electronic discovery populations get, at best, a simple within custodian de-dupe effort, while the results of cross-custodian de-duping are clearly spelled out in financial terms.

Why is this? Is it just that some customers are obstinate and refuse to try anything new? I doubt it. I think it is the same reason that people still use their rotary phones and listen to records. They are comfortable, familiar, safe and well understood. Simple removal of duplicates inside a single custodian, or no removal of duplicates is comfortable and easy to understand. Furthermore, it cements the idea that all files or documents need to be assessed individually – a belief that the legal community seems terrified to let go, even though all signs point to its eventual demise. Those who would cling to no de-duping or inferior de-duping are the same people who will avoid population analytics and automated review. Be glad they practice law and not medicine.

PS. In case you were wondering: Almost 1 billion vinyl albums were sold in 2007, up 15% from 20062.

-Sandy

[1] Source: Wikipedia http://en.wikipedia.org/wiki/Model_500_telephone

[2] Source: Time magazine http://www.time.com/time/magazine/article/0,9171,1702369,00.html

Doc Review Metamorphosis

Most people today communicate via email far more than they do via telephone (which in turn people use far more than letter-writing). It wasn’t always this way, of course, but most people do not realize that email has been around in one form or another since 1984! It took effectively 25 years for email to come to dominate as the preferred communication method, particularly for business.

How long will the Doc Review industry take to evolve to true non-linear[1] activity? Probably in less time than you think. For comparison, the transition from letters to phone dominance took more or less 60 years (1900-1960). The transition from phone to email took 25. And now as we evolve from relatively static email to far more dynamic Twitter, texting and live data/video feeds, the evolution will be even shorter, generally expected at 5-7 years. As a society we are becoming faster at adapting to change. The most sophisticated review teams are just now experimenting with non-linear review. The rest will be there within 3-4 years.

Already we see signs of the evolution to non-linear document review. Most people are at least aware of the simplest form on non-linear review: near duplicate detection. With near & exact duplicates grouped together into “Dupe Groups,” the very first level of non-linear review is taking place. With Near Dupe, savvy reviewers actually look at a group of documents together, rather than one by one. Documents are usually grouped together by content or attribute similarity, with a group “captain” embodying either the fullest set of content or a logical start and end-point to the logic chain grouping the documents together.

A similar technique is called Email Thread Grouping (ETG), where pieces (“stringlets”) of email conversation threads that might be resident in pockets of document storage mechanisms are brought together logically. Because typical ESI collection involves documents from several custodian sources, often who have important communication relationships with one another, the incidence rate of disassociated email conversations is extremely high. ETG bring together groups of documents from Inboxes, Outboxes, folders, different email storage systems and even different collection sites! By grouping the conversations together from all custodial sources, the review has taken a second step toward non-linear review.

Finally, the courts and the industry are waking up. It’s not going to be a “doc-by-doc” world much longer. Will you be ready? Where can you turn for smart, unbiased information?

The Electronic Discovery Institute is a non-profit organization dedicated to resolving electronic discovery challenges by conducting studies of litigation processes that incorporate modern technologies. They recently conducted a survey which broaches the subject of electronic document deduplication; the beginnings of non-linear review. Have a look and let us know what you think.


-Sandy


[1] Non-linear review is the concept of reviewing documents in bulk fashion, rather than one by one in sequential order.

Name That Service!

Remember the game show “Name That Tune[1]”? Contestants had to listen to the first few strains of a song and name the title. “I can name that tune in 4 notes!” they would say. Well, now it’s time to name something else – Valora’s new service offering!

After years of “black box” services provision, Valora is finally opening up its interface to our customers. For those of you who have been to Valora’s Processing Center or seen one of our FirstLook Population Analysis Reports, the interface will have a familiar feel.

To those who are new to such services, we are creating a combination Early Case Assessment Tool & Non-Linear Review interface.

The idea is really very simple: show our customers easily and intuitively what we here at Valora already know about their documents. For years, we have been identifying and documenting every possible attribute known to mankind[2] about each and every document that passes our threshold. More recently, we began to use that knowledge in a cross-functional, population analytics way to help diagnose what populations contain and how best to stretch processing dollars accordingly.

Now we are using the same information in a predictive, pre-emptive way to automatically accomplish much of the tasks that today take place manually. Similar to how Valora solved the expensive, inefficient manual coding problem a few years ago, we are now solving the problem of getting both document processing and document review down to a few cents per GB.

While we anticipate public release of our system in January, 2010, we are opening up our Beta program to three select matters late in the summer. Two of the three Beta partners have already been selected, but there is room for one more. If you are interested, contact our Marketing Department at 781.229.2265, mktg@valoratech.com for more information about Beta requirements and expectations.

And finally, the contest! You may have heard via email, and it’s true, that I will personally come to your home (especially if you live in Hawaii, Bermuda, south of France…) and cook you and your family a gourmet dinner! I’m a pretty good cook and I take requests. So, think about Valora, think about our new ECA-NLR platform and most importantly, think about a name! Official submission rules are on our website: Contest Rules! May the best name win!

-Sandy

Name That Service!

---

[1] For a modern-day (and incredibly time-wasting, though hilarious,) version of Name That Tune, visit: http://www.namemytune.com/nmt.asp.

[2] Today, Valora captures over 60 “fields” of information about each and every document in our systems.

Why are we so excited about our GSA Award?

Well, for starters, we are able to sell all our services at the top echelons of local, state and federal government. We are now part of a select list of “ok to use” government suppliers. This means Valora will be actively solicited by government agencies across the nation.

But, what it really means is that we have garnered an impressive seal of approval on our collective lapel. The Government Supply Agency (GSA) really does its homework on who may become an official supplier of goods and services. The goal is for government agencies to streamline their purchasing by using a pre-vetted supply source. But the upshot for everyone else is that Valora has earned this special position by performing admirably over a number of years. Here’s what Valora had to show in order to become a GSA supplier:

1. That we have several years of outstanding and honorable service delivery and business dealings. GSA surveys a minimum of twenty (!) references to create an “OpenRatings” score for each GSA applicant. We are extremely proud of our OpenRatings score, having scored a whopping 92/100 total possible points!
2. We must establish fair pricing and justify each and every cost item. We have to indicate every discount, every bundling variation, and every custom software/services configuration.
3. Our staff had to undergo strong background checks, including financial, criminal and employment verification. We had to prove our company employs American workers, pays fair wages, holds appropriate insurance policies, is up to date in its federal and state tax payments and so on.
   

In short, we had to prove we are an organization in good standing, that provides strong-quality work products at good prices, and is comprised of decent, hardworking and upstanding people.

Valora received our certification in under 6 months – an outstanding accomplishment, which I think speaks very much for itself. Well done, Valora team! Well done!

-Sandy

Electronic Files Rehashed

This month’s blog entry is provided by guest blogger, Aaron Goodisman. Aaron is Valora’s Chief Technology Officer and Vice President of Engineering.

To a lot of people the word “hash” conjures up visions of leftover corned beef and onions (or, in some parts of the country, barbecued pork) – delicious. In the world of computers and electronic files, however, it’s less about chopped up meat and vegetables, and more about chopped up files and documents.

A “hash” (or more completely a “hash code”) is a kind of electronic signature of a computer file. The data bytes that make up the file are processed through a hash function (chopped up) to produce a short, fixed-length snippet of data that can be used to refer to the original file. Since the resulting hash code is short, it’s easy to store a lot of them. It’s also quick to send them across a network or compare them to each other – much quicker than doing the same thing with the whole file.

Web browsers and web servers use hash codes to decide which web pages to refresh. If the browser has a copy of a file stored locally on your computers hard disk (e.g., the header graphic on this page), it sends the hash code of that to the server to ask if the file has changed. The server compares that hash code to the hash code of its version of the file. If they’re different, the server transmits the new header graphic file; otherwise, it tells the browser to go ahead and use its local version. Sending just the hash code takes much less network bandwidth than sending the whole graphic, making pages load faster and reducing the overall load on the network.

The Litigation Support and Computer Forensics industries use hash codes, too. Rather than processing every file on a custodian’s hard drive, savvy practitioners skip over duplicate files, useless files or malicious files by computing the hash code of each file on the disk and comparing it against lists of hash codes of files known to be useless, malicious or already processed. Because the hash codes are small, the lists are easy to manipulate and fast to search.

For all this to work properly, the hash function needs to have several important characteristics:

• It needs to be fast to compute the hash code of a file (otherwise it would defeat the purpose of being able to do quick comparisons)
• The hash code of a file needs to change if you change the file, even a little bit (otherwise the web browser wouldn’t know to download the new version)
• It needs to be extremely difficult to create a file with a specific hash or to create two files with the same hash (called a “collision”).
  

The last of those is particularly important for electronic file processing, because it wouldn’t do for the critical document in a case to be removed because it accidentally happened to have the same hash code as a standard Windows library file or some non-critical document already processed. Neither would we want a virus writer to be able to manipulate a file to contain a virus, but have the hash code of a different file known to be safe.

Fortunately, over the years a lot of hash functions have been developed and thoroughly tested. One of these hash functions is called MD5, developed in 1991 by MIT professor Ron Rivest. This hash function became extremely popular with the growth of the internet, the proliferation of electronic files, and the widespread use of hashing techniques for various purposes. In fact MD5 became so pervasive that some people in the Litigation Support industry use it as a synonym for term hash code.

Unfortunately, things change. Computers get faster and MD5 has reached the end of its useful life in this industry. In 1996 it was shown that it was theoretically possible to create two files with identical MD5 hash values (a collision) and in 2007 a group of researchers described how to do it. A recent article in Technology Review magazine describes one example of this.

The good news, of course, is that there are many other hash functions to choose from. Several years ago Valora switched to the SHA-1 hash function, designed by the National Security Agency and published as FIPS 180. SHA-1 is similar to MD5, but uses a longer hash code and is orders of magnitude less vulnerable to collisions. Those orders of magnitude don’t mean SHA-1 will be useful forever. Indeed, it’s been shown that SHA-1 is vulnerable to attacks similar to the ones used to bring down MD5, but it will be a while before the necessary computing resources are available. By then we’ll have moved on to other hash functions appropriate to the computing power of the day. But the industry needs to keep moving forward with technology, so when the time comes to switch functions, this discussion doesn’t need to be rehashed.


-Aaron


Aaron Goodisman is a software industry veteran with over 20 years experience in engineering management, software architecture, and product development. Prior to founding Valora Technologies, Mr. Goodisman served as Vice President of Engineering at SilverStream Software, acting as both manager and visionary for this award-winning application server product and its associated development and deployment tools.

Mr. Goodisman received his undergraduate and Master's degrees in Computer Science from MIT and is considered a world expert in Java industry standards and UI design. He is a frequent industry speaker and has authored several articles for industry publications. Mr. Goodisman is named as the inventor on several U.S. patents and currently pending patent applications.