Why Information Governance is Eclipsing eDiscovery

Everywhere you look right now, Information Governance, “IG,” has taken center stage.  I have had the pleasure of speaking twice in a week on the topic – once to records managers at ARMA’s Northeast Regional Conference and once to litigators at McDermott’s Technology in the Law Symposium.  Why is IG so hot and why is it overtaking the discussion on eDiscovery?

IG is hot, hot, hot

IG is the Next Big Thing because it is a catch-all concept that covers a lot of currently important ground:  compliance, data ethics and breaches, data management and intelligence, workflow, visualization and analytics.  All of these elements play a key part in IG, and always have.  So, why is it hot now?  The biggest factor is the Target data breach of over 70 million customers’ personal data.  The scope of the breach (nearly 25% of all American citizens were affected) and the wall-to-wall media coverage has helped propel responsible data management into the forefront of society’s concerns.  In fact, a Pew Research study in January found that over 50% of Americans are “worried about the amount of personal information available about them…”  The IG train of responsible data management has left the proverbial station and is speeding its way through the legal system, through Wall St., and through consumers’ concerns and buying behavior.  Nothing speaks louder than consumers and their wallets.  For Target, “Satisfactionwith the overall shopping experience was down almost 2 percentage points inMarch, with declines “most acute” among middle-and-upper-income shoppers” as late as April, 2014 -four months after the breach was announced.

Why is the IG discussion eclipsing the eDiscovery discussion?

For starters, eDiscovery is old news.  The earliest uses of the phrase stem from 2004, nearly a decade ago, and well before the FRCP changes in late 2006.  Today most litigants, and certainly their outside counsel & advisors are very familiar with its concepts.  In fact, most service providers in legal, lit support or eDiscovery already have a wealth of tools and solutions to choose from.  Need Early Case Assessment?  ESI Processing?  Predictive Coding for Doc Review?  There are a plethora of solutions, all heavily vying for your attention.  The truth is, it’s just not that complicated anymore and the solutions have decreased so much in cost that almost all solutions are accessible to almost all matters.  In short, eDiscovery has become as exciting as word processing or scanning.

But the eDiscovery blahs are only half the reason for the decline in discussion.  The other half is that intelligent IG encompasses eDiscovery.  eDiscovery is subsumed by smart DDC (data, document & content) management, right along with litigation holds, retention policies, workflow routing, exception handling, data breach response and investigations.  Today, eDiscovery is but one of any number of critical activities undergone by major corporations all the time.  It’s just not the fire drill it used to be, and those implementing IG will see to eDiscovery’s needs along the way.  

So, where does that leave us?

Unfortunately, this leaves us woefully and inadequately prepared to handle IG.  The passel of eDiscovery tools do little to solve problems that are much larger than typical litigation matters every imagined.  The the records management side of the house is of little help with their diminished budgets, and dearth of tools available for large-scale data mining and management.  Thus there is promising opportunity for IG-oriented solutions that take the best of both worlds, with an eye towards intelligent DDC management from the outset.  Stay tuned, blog readers, and see where Valora heads next...

IBM Watson Runs a Food Truck?!

What to do after winning Jeopardy against the world’s best players?  Open a food truck, of course!  Yes, that Watson is now running a food truck, and apparently it creates some truly delicious dishes!  Confused?  Don’t be.  The intelligence behind the Watson engine that successfully answered hundreds of randomized Jeopardy questions is now the creative engine behind a gourmet food truck.  IBM is endeavoring to show that predictive analytics have uses in the most unusual of places!

As with most predictive analytics, there is still an important role for humans to play in providing balance, judgment and expertise.  Watson does  the data-crunching heavy lifting to find interesting and appealing flavor combinations, faster (better?) than human beings can do on their own, and then trained chefs implement the Watson directions.

This hybrid approach should have a familiar ring to it.  Let the software do the hard, data-intensive number-crunching and then marry that output with human skill and finesse.  It’s a winning combination and one that we employ here at Valora every day.  We utilize our analytics, indexing, and rules platform, PowerHouse, to organize, catalog and find relationships in content for us and then we add the human skill, the expertise, to refine the output and do custom things for specific projects.  

Here’s an example:  We run 50,000 emails and attachments through PowerHouse, which quickly finds well over 150 attributes about each item.  Then we ask PH to find important relationships and insights, such as trend data or topic clusters.  From there, we adapt the rules programming to customize the output so it yields middle initials, or zip + 4, or the top 3 issues per document, or whatever it is that any particular customer needs.  Load it up to BlackCat for easy, online review (often by the client’s workforce) and we’re done.  Predictive analytics mastery!

Now, if you’ll excuse me, I think pork belly moussaka sounds amazing!

What Do Self-Driving Cars and Documents Have in Common?

In case you missed it, Google had an announcement earlier this week about the rapidly improving reliability of their self-driving cars.  The cars now automatically recognize, pedestrians, trucks, and construction areas and even when a cyclist suddenly veers in front of the car.  Having logged more than 700,000 accident-free miles, it’s an impressive demonstration of a potentially society-altering capability.

So, why mention it here, other than its super-coolness factor?  Because of how it works.  Google’s self-driving cars function because they are taught to recognize patterns.  Patterns of behavior, actions, appearance, movement and trends.  Once a pattern is recognized, the cars’ on-board computer systems run a series of rapid-fire statistical algorithms to determine what is happening (context), and thus what actions the car should take.  Sound familiar?  It should.  This is the same kind of technology behind IBM’s Watson and Valora’s PowerHouse.

Much like Google teaching its cars to recognize a stroller in a crosswalk, Valora teaches PowerHouse to recognize a patent application in Chinese or a break in privilege from an email string.  Google’s vehicles accurately assess and predict traffic behavior patterns almost 100% of the time, much better than human beings.  Valora’s PowerHouse sees similar marks for accuracy and prediction. 

In addition to one day allowing us to text messages or read an e-book while we “drive,” the autonomous vehicles have another enormous advantage:  they better utilize roads, gas and electricity.  These types of benefits have broad-reaching impact beyond whether any one person is using or not using the self-driving car.  The same holds true for autonomous data mining.  Once the document house is in order, everyone benefits from easy, organized search, to intuitive data visualization to automated notifications of significant events.

Too bad I couldn't write this blog entry while on my way to work this morning…

Interesting Predictions about Data Analytics from Gartner

By 2017, analytic applications offered by software vendors will be indistinguishable from analytic applications offered by service providers.

"Traditional vendors of analytic platforms recognize that in order to expand their reach beyond traditional power users, they must deliver packaged domain expertise and applications to enable self-service by a wider range of users. Service providers are seeking to turn custom project work and domain expertise into repeatable solutions that can be adopted by other organizations more easily.

The result is that end-user organizations selecting analytic applications will have a significantly wider variety of possible providers to evaluate. Organizations evaluating software vendors will almost always find a SaaS version of their packaged applications, and the similarity of product concepts will shift the emphasis of competition to the domain expertise embedded by the vendors into the application. Software vendors will increasingly face a co-opetition situation with their traditional service provider channels, forcing them to augment their own professional service capabilities. Service providers will use packaged applications as an integral part of their customer relationships, implying that there is a greater specialization in the services that they provide."

-Gartner Press Release 12.16.2013

Data Vs. Document Vs. Content

Remember letters?  Typeset documents on official-looking letterhead?  When we communicated primarily via letters, no one wondered what to call the media transmitting information.  It was a Document, plain and simple.  Then came the Internet and websites and eyeballs, and suddenly it was all about Content.  Keeping your content fresh, managing your content, re-using content.  Now it is all about the Data – Big Data, of course.  So, what’s the difference?  Data vs. content vs. document – is there a difference?  In theory, not much, but in practice, yes there is.

Let’s start with Documents.  Documents can be physical or virtual, but they typically have a defined start and end, often delineated by page.  Documents have a specific purpose: they were created by someone, for someone, and they are meant to convey information.  Documents carry with them an air of significance, importance and validity.  That’s why we have phrases like, “Legal documents, financial documents and immigration documents.”  Good examples of documents:  Your tax form, your birth certificate, a receipt from a purchase, your boarding pass.

Content is amorphous.  Though it too can be physical or virtual, it is generally thought of as virtual/electronic in nature only.  Content may or may not have a specific purpose.  It may be written by someone, or sometimes auto-generated.  Content is often not meant to stand on its own, but rather be a supporting player.  Content can be ephemeral, biased and taken out of context.  Because of this, content is not always trusted and carries less validity than documents.  Good examples of content:  blog entries, news, chapters in a book.

Data is virtual.  It is reported, stored or derived from other systems and carries with it a factual and scientific nature.  Data is meant to be bias-free and exist for measurement or tracking purposes.  Good examples of data:  your height and weight, stock prices, bank account balances.

To call information data is to expand on the original intent of what we understand data to be.  However, because our information today is generated and stored electronically, it feels like data, and we (or savvy marketers) have started calling it data.  Thus stored information has becomes data, with all the attached concepts typically assigned to data (factual, bias-free, etc.).  Data, therefore, feels trustworthy and valid – a strong case for managing its exposure.

For more information on the difference between Data and Records, see my article in this month's ARMA newsletter.  When is Data A Record?  (See pages 23-25)

Data Analytics Steal the Show at DC Technology in the Law Symposium

I was delighted to serve as a panelist at the Technology in the Law Symposium, held earlier this month by the DC Bar Association and McDermott Will & Emery, LLP.  Three panels spoke on the use of predictive technologies and analytics and their use in the courtroom, for eDiscovery and well beyond.  Panelists ranged from outside counsel litigators, to DOJ government attorneys, to service providers and consultants, with Hon. Judge John Facciola presenting the keynote.  The lively, and at times contentious event featured three broad topics:  What the Courts are Saying About Predictive Coding, Predictive Coding Pessimists v. Optimists, and the Use of Data Analytics in Other Areas.

“There’s no greater compliance training than grand jury subpoena.” John Kocoras, Partner, McDermott, Will & Emery

While much of the early discussion around Predictive Coding was, well, somewhat predictable by now, (Key messages:  PC is good, it is proven for doc review, courts are onboard, don’t be reckless with it), the panelists and audience really became animated about the uses of predictive analytics beyond simple relevance review.  Panelists John Kocoras (MWE), Kristian Werling (MWE), Sandra Serkes (Valora) and Kurt Michel (Content Analyst) described scenarios where predictive technologies were being used in multiple corporate settings to assess M&A documents, contracts or financial statements and seek out areas of corporate compliance exposure.  At one point, moderator Jason R. Baron (Drinker Biddle) jokingly asked the panel whether such technologies could accurately predict legal case outcomes.  Answer:  Yes, within reasonable margins of error.

In all, the inaugural Symposium was a roaring success for the 150+ attendees from all over Washington, DC and beyond.  MWE is hoping to expand on their initial success and present several more related symposia in 2014.  Well worth the free attendance!

Why ARMA?

Some people might wonder why a “lit support vendor” would be attending the ARMA National Conference in Las Vegas.  Truth is, Valora’s capabilities have been exceeding “lit support” for a long time.  We find kindred spirits in ARMA, because we are looking at the larger world of corporate documents – for lots of purposes, litigation being just one of them.  In the last 18 months, we have seen tremendous convergence between traditional litigation and eDiscovery with Records Information Management and Information Governance.  In fact, last month I gave a presentation to the NYC chapter of ARMA on “5 Things Litigation Can Teach Records Management and 5 Things You Can Teach Them.”  (Let me know if you’d like a copy of the slides.)  We are going to see more and more of this kind global information management, where litigation is but one use of an organized and controlled data governance strategy.  Watch this space for more on this topic in the weeks to come.

25 Cool Valora Things

I am often asked, “what’s the coolest thing Valora has ever done?”  That’s a toughie because Valora does a lot of cool things and I would be hard-pressed to pick just one.  Having just gotten yet another totally awesome request yesterday, I decided to compile a list of The 25 Coolest Things Valora Has Ever Done.  

If you think we forgot some, email me at: sserkes@valoratech.com.  And, if you’d like to learn more about any of the real-world scenarios on that list, just email or call.  We’d be happy to share our stories with you (to the extent we are able).

And, finally, here's a bonus cool thing:  an Auto-Generated Word Cloud for the content on this page.

1. Re-orient and AutoCode documents presented in "mirror writing"
2. Capture the Japanese "Showa" Date off of documents
3. Assess long distance spending habits by analyzing multiple years of corporate phone records
4. Create metadata for (paper) documents from 1901 - 1925, including Near Dupes
5. Analyze credit card receipts to determine his & hers spending habits for a high profile divorce
6. Automatically determine if documents are Classified
7. Identify buildings by address, building number or building name (e.g., "Trump Tower")
8. Index video files, with generated stills that correspond to key phrases & topics
9. Uncover an "inappropriate relationship" within standard business communications
10. Identify likely missing documents from email chains, custodians and shared drives
11. Code work product documents that included Valora invoices and emails in them (talk about recursive self-reference!)
12. Determine which applicants were lying on their hiring application
13. Translate documents to/from Japanese, German, French & English to each of the other 3 languages
14. Identify bodies of water in documents
15. Analyze shipping records to identify unusual purchasing behavior
16. Select "best" versions from multiple reports and coverage of the same event
17. Audit the results of Onshore Doc Review vs. Offshore Doc Review vs. AutoReview
18. Determine what type of information was likely underneath document redactions (blackouts)
19. Identify the cell phone of an NBA player
20. Match 25,000 index cards with appropriate database records
21. Redact out ages of minors (no redactions for 21+)
22. Review documents for 162 unique "Issues"
23. AutoUnitize a 300,000 page PDF into "logical" documents
24. Graph potential smuggling routes based on email traffic and news reporting
25. Index 30 million records in 3 months (that's over 300,000 records every 24 hours)

Specialized Knowledge, Skill, Training and Education

This entry is provided by guest blogger, Aaron Goodisman, Valora’s Chief Technology Officer.

Oh, I feel for D4; I really do. Let me explain:

Law Technology News reports on a case in which D4 Discovery acted as litigation support vendor for both defendant and plaintiff, albeit at different times and performing different functions. Naturally, when defendants Nixon Peabody (working for Kaleida Health) found out, they objected to U.S. Magistrate Judge Leslie Foschio, but he refused to disqualify D4 as a vendor for the plaintiffs.

Sounds like a win for D4, no? As a vendor with many clients in the litigation support space, Valora doesn't like to turn away work any more than the next guy. And, as professionals with over a decade of experience in the legal field, I'm confident that we could maintain appropriate walls of confidentiality between project teams, as D4 asserts they have done.

The problem lies in judge Foschio's rationale for the refusal to disqualify. What the judge essentially said is that D4's scanning and objective coding for Nixon Peabody does not include expertise or consulting, and that it did not expose D4 to any confidential information about the case or Nixon Peabody's case strategy. As an experience scanning and coding provider, this is simply incorrect.

“Objective” coding refers to tagging documents with information that can be objectively determined, without rendering any kind of opinion. In this regard, at least, judge Foschio's rationale makes some sense. That type of information is sufficiently objective that Valora uses software to determine it for most documents. No opinions there.

On the other hand, the design of a scanning and coding project is absolutely a consulting activity: which information is captured for which types of documents, which collections get extra information tagged, which are fast-tracked, which get an extra quality control pass. How the various containment and attachment relationships are captured among documents, folders, binders, boxes. To an experienced litigation support person, those decisions speak volumes about the case.

For proper and accurate scanning and coding to have occurred, D4 had to have access to, and indeed looked at, every single one of the documents in the case, including any that Nixon Peabody later withheld as privileged.

Again, I have no reason to believe that D4 violated their confidentiality responsibilities to either party, nor does it appear that Nixon Peabody is claiming that. Rather, what's happening here is that a judge has said that the services D4 provides do not require “specialized knowledge, skill, training or education.” That's just wrong.

Valora's clients come to us precisely because we provide those things. Kaleida continues to maintain that D4 should have been disqualified from working with the plaintiffs. I'm sure it's standard legal practice, but it feels like somebody's defending the value of such services, at least a little.

Technology-Assisted Essay Grading

The NY Times recently reported on the growing use of automated essay grading systems, what we in the legal & records space might call "Technology-Assisted Grading," or "TAG." This is yet another instance of the rest of the world utilizing predictive technologies in conjunction with statistical pattern-matching to create an ultimately subjective judgment of the content of a document. Even more interesting than the fact that MIT & Harvard are making this technology available for free via edX (my alumni donations at work??), is that the higher education community is having the same heated arguments that are occurring right now in the legal arena. Here is the best comment from the piece:

"Although automated grading systems for multiple-choice and true-false tests are now widespread, the use of artificial intelligence technology to grade essay answers has not yet received widespread endorsement by educators and has many critics.."

Sound familiar? It should. This is exactly the argument raging now by outside counsel (playing the part of professors in the article) attempting to hold onto their turf, once considered "untouchable" by technology. While it is true that computers can't "read" either student essays or litigation emails, they can be trained to recognize the salient elements that make the essay strong or the litigation email privileged. Those traits are easily describable as Rules. Either a document fits the Rules, or it doesn't. Nuances are accounted for with confidence scoring and sampling for accuracy (precision & recall). As long as there is sufficient auditing and exception handling, the work quality should be outstanding at a fraction of the time and expense of the purely manual method.

If recent advances have taught us anything, it is that nothing, and certainly no job function, is immutable. It doesn't matter whether the work task is rote (like tightening bolts), cerebral (like computation) or subjective (like analysis), it can all be done by the right algorithms, utilizing the proper training, feedback and statistical sampling.

Furthermore, when subjective work product is automated, society gains impartiality, consistency, speed and reduction of cost for the same services. That allows us to do more, work faster and create better results with fewer resources – the very definition of progress.

It is time to stop fighting the obvious, accept the reality, incorporate the efficiency gain and move on. I'm ready for my essay grade, please.

Sprechen sie deutsch? Parlez-vous français? You do now!

Remember in Star Trek when the "away team" would encounter a new civilization and be instantly able to communicate with the alien beings by using their handy "UniversalTranslator"? No Tower of Babel in science-fiction! Well, there needn't be one in today's document environment either! With the recent great strides in pattern recognition and content translation, we effectively have a Universal Translator for document and files written in virtually any world language. With support for 65 world languages, Google is to thank for the raw translation effort, while Valora has taken things to the next lelve by implementing the raw capabilities into complex litigation and records management workflows.

For example, on a recent matter we rapidly AutoTranslated documents from 5 foreign languages into English, where they can now be easily understood and managed by the US litigation team. The whole effort took under a week and was 1/10^th the cost of manually translating the same material! As with most Automated Solutions Valora offers, a little technology goes a long way! Learn about Valora's AutoTranslation services, by clicking here.

12 Tips To Get The Most Out of Technology-Assisted Review ("TAR")

1. Decide which TAR approach best fits your needs and how you plan to deploy the solution: Do you want the seed set, predictive coding approach or the pattern-matching, rules-based approach? Seed set is good if you don't really know what you want, or you like to "decide on the fly." Rules-based is good if you know what you're looking for and can explain it (similar to how you would train contract attorneys for a large-scale review).


2. Similarly, do you want TAR as a service or do you want to install a product? Products are typically lower-cost, but less featured or customizable to your specific needs. Services typically cost more, but usually include expert analysis and consulting as part of the package. One consideration in product vs. service is how frequently you encounter a need for TAR and how similar each instance is to the next. Higher frequencies would lead you towards a product, but low similarities across needs would lead you towards services. Remember to include both hard costs (typically dollars outlaid) and soft costs (such as training time and expenses, storage needs, platform support, etc.) in your analysis.


3. Be realistic about how much you will rely on the coding performed by tool or process and what level of QC you will require. Will you eventually have "eyes on" every document or will you only put "eyes on" subsets of the documents based on relevance or issue criteria? Understanding this early will help you to make the right decisions on pricing, implementation and staffing.


4. Get comfortable with pricing metrics conversions. Some solutions are sold per document or file, some per GB and some per hour. Here's how to translate between those metrics. Assume: ~ 6,000 docs/files per GB (post processing), and ~ 50 docs/files reviewed per person per hour. Now you can compare pricing for different solutions!


5. Be explicit about your needs. Do you want a simple yes/no answer for privilege or do you want to know which types of privilege are being invoked? Ex: attorney-client vs. work product. Same for relevance. Is it enough to know simply that a document is relevant or do you need to know why it is relevant (and/or to what degree)?


6. Map out your workflow and strategy. You (or your client) will need to defend your document production approach. For maximum defensibility, make sure your process is repeatable and transparent. Be wary of TAR solutions that do not disclose why or how propagated decisions are made. Be similarly cautious of solutions that yield different results when different people are "manning" them. Furthermore, make sure that the provider will back you up by providing tangible proof to support the defensibility of the process.


7. Understand that TAR is an iterative process. The more guidance and feedback you provide, the stronger the results will be. Do not expect the first round to be perfect. You and the systems will both get better over time. As a rough rule of thumb, expect 4-5 iterations.


8. Think about Exception Handling. Even the best TAR solutions will encounter "problematic" documents from time to time. How will you handle hand-written documents, custom application files or documents written in foreign languages? A good TAR solution should be able to easily identify the docs/files it cannot handle and remove them from the automated processing queue. In other words, don't pay twice for documents that will ultimately need manual processing.


9. Make good use of Issue Codes. Most sophisticated TAR solutions can handle multiple Issue Codes, providing very helpful tagging and organizational information for Hot or Responsive documents. A consultative TAR solution provider can help you maximize your Issue Codes protocol so that it complements and enhances the production.


10. Be aware of potential privacy concerns. Many document collections have sensitive or personally identifying information (PII) in their contents that cannot be openly shared. Sophisticated TAR techniques identify, cull and/or automatically redact this information prior to production. TAR approaches can save many hours of manual effort to cleanse data for production.


11. Choose your solution carefully. Expect that your needs will change over time, both in general and across a single matter. Ideally, the solution provider has full, unfettered access to the TAR engines, so that they can be custom-tailored to your (or your client's) exact circumstances. Be wary of "one size fits all" solutions.


12. TAR Beyond Document Productions. TAR has uses far beyond review for responsive and privilege. Consider utilizing the techniques when you (or your client) are on the receiving end of a large volume of data. TAR processes can be extremely cost-effective at organizing, cataloging and indentifying trends and data threads in incoming material.

Redaction? There’s an App for That..

Raise your hand if you've either participated in or managed a group of contract attorneys sitting in rows of cubicles (or "stations") redacting documents for production. Whether your tool of choice was a black marker or a cursor, I'm betting it was a thankless, tedious, pain-staking job and you hated it. Wouldn’t it have been nice if you could've taught the computer how to recognize the patterns of PII (private identification information) and have it make the redactions itself?

Well, guess what, Valora heard your anguished cries and we've built an AutoRedaction engine that rivals any group of manual redactors. With blazing speeds, impressive accuracy and astounding savings, our PowerHouse^TM system literally autoredacts paper and ESI documents in seconds.

Capitalizing on our extensive experience with pattern-matching technology^[1], Valora has built a custom software program that automatically determines the presence of sensitive PII, confidential or privileged information, and then redacts out that information on the image. AutoRedaction takes the form of a black block, with or without a representative stamp, such as "Redacted" or "Employee 123." Redactions can be made permanent, such as for production purposes, or kept temporary, with a technique for "lift and peek," when desired. Redactions can also be made to the underlying text, or on both text and image, if desired.

---

^[1] For more on Probabilistic Hierarchical Context-Free Grammars, see this link on Google Scholar.

Statistical Pattern Matching Accurately Predicts Presidential Winners and Electoral College Counts, Why Not Privilege and Responsiveness in Litigation?

The technology utilized by political statisticians is finally getting the attention it deserves.  Not because it is partisan, but because it is accurate.  The excellent article in today’s LA Times explains how mathematical models predicted the election outcome well before the first polls had opened. How? By taking the information from numerous sample sets and re-modeling over and over again with different assumptions and weightings. If this sounds a lot like statistical sampling and pattern-matching, then you have been paying attention! The techniques used by the Nate Silvers of the world to classify and label voting patterns are being used right now in litigation to “predict” (or diagnose, if you prefer) for privilege, responsiveness and issues.

At Valora, we call this technique Probabilistic Hierarchical Context-Free Grammars, but others have shortened it to Statistical Pattern Matching, which works just fine. The point is that information about documents (or voter behavior or music choices) has been available for a long time. The only missing piece is the human comfort level with statistics and probabilistic systems.

If the statisticians can call elections, baseball winners and consumer preferences, isn’t it time we let them loose onto document analysis and review? If you’d like a primer on or a demonstration of Probabilistic Hierarchical Context-Free Grammars in litigation, contact us at valoratech.com.

Valora Technologies CEO, Sandra Serkes, Responds to Craig Ball’s LTN Article on “Next Level” Technology Assisted Review

Original article: Imagining the Evidence

I am pleased to inform both Mr. Ball and the world that the “next level” of TAR, meaning the use of whole documents and populations, rather than selected seed sets, is already here and doing fine.  Rules-Based approaches to TAR are not constrained by the need to create and perfect the selection of a seed set.  Instead, they apply their algorithms and iterations across the entire population, at once, each time.  There is no need for any exemplar document, as the exemplar is the rule itself – thus any document can be evaluated for its “exemplary-ness” and to what degree, where and when.

Furthermore, Mr. Ball discusses the thorny issue of self-interested collection and seed set tagging.  He suggests the opposing party should be the one to set the seed set tags into motion.  This is a step in the right direction.  But, the best approach would be to have both producing and opposing working together to determine relevance – an option easily afforded by a Rules-Based approach.  Rather than having any one party have to sit down and hand-craft a seed set, both sides can agree on the RULES of responsiveness, rather than on whether this document or that one is the better exemplar.  With agreed-upon rules in place, documents are easily assessed not just for yes/no relevance, but also to what degree.

Finally, the notion of “imagining” the documents is very much alive and well in the field of Data Visualization.  We often use this technique in a descriptive way (here’s what your data shows), but it can also very much be used in a proscriptive way (is there anything that looks like this?  How close?).  This concept is very much connected to the current practice of iterating for performance optimization (aka: trading off precision and recall).  TAR systems that utilize the notion of DocType or Attribute templates already have the concept of a “generic” or “iconic” version, essentially an exemplar.  It is trivial to create more templates and use them in a hierarchical manner to test how much a potential document matches the generic exemplars, by relevance priority.

Valora Technologies CEO, Sandra Serkes, Invited to Speak to ILTA South Pacific Region About Technology Assisted Review

ILTA Program to Cover "Exploring Predictive Coding and Technology Assisted Review: Valora Technologies' Approach"

July 25, 2012, 6:00pm EST

During her presentation, Ms. Serkes will layout the Technology-Assisted review (TAR) landscape and Valora Technologies' overall approach.  With Document Review the most costly phase of ediscovery, there are considerable savings to be achieved when it's possible to automate some or most of that review process in a reliable and defensible manner.  This theory is the motivation behind the hype about predictive coding and TAR. Unlike most of the solutions on offer in this market space, Valora Technologies' approach is Rule-Based and transparent.  Come learn about the developing TAR landscape and one provider's unique vision for this space.

Remote & physical presentation sign-up now open.

3 Drawbacks To Predictive Coding

Valora’s Response to LTN article: Take Two: Reactions to 'Da Silva Moore' Predictive Coding Order

What is missing there, and elsewhere, is a discussion of the specific weaknesses of the overall Predictive Coding technique.  Here are just three drawbacks of the technique: 

1. PC tagging algorithms are not transparent.  No one really knows why the PC engine "chose" the documents it did.  Typically, the “choosing” algorithm is hidden and not disclosed.  All we know is that somehow the document recognized is a lot like another tagged.  
2. PC has no checks or balances on the skill set, education, consistency or motivations of the seed set coder(s).  The entire Predictive Coding approach assumes that the seed set coder(s) know what they are doing, and that they are correct, consistent and honest.  Would you defend that position, particularly given that the “human being as gold standard" concept has been roundly deflated (see Blair & Maron, Grossman, TREC, etc.)?
3. Typically, seed set creation and audit sampling for PC use a random sampling technique, the weakest of all types. 

Other sampling techniques (stratified, cluster, panel, etc.) are aware of document attributes and utilize intelligent groupings to create a much stronger, more representative sample for seed set coding and auditing purposes.

Since at present, all Predictive Coding solutions are products, which means they have limited functionality and flexibility for specific case matters, perhaps we should be thinking about the broader picture of Technology-Assisted Review (TAR) as a service – customizable, measurable and transparent.

What’s the Difference Between Automated Review and Predictive Coding?

Automated review and predictive coding are often mentioned in the same breath, as synonyms for each other. They are actually different concepts. Predictive coding, in which a topic-expert manually codes a "seed set" of documents (and the software follows suit) is a type of automated review. There are 2 other types.

A second approach to automated review is called Rules-Based Coding, in which a set of rules is created to direct how documents should be coded, very similar to a Coding Manual or a Review Memo that might be prepared for a group of on- or off-shore contract attorneys. The preparation of the Ruleset is typically done by some combination of topic experts, attorneys and technologists. The rules are run on the document population and it is evaluated, tweaked and run again until all parties are satisfied.

The third approach to automated review is called Present & Direct, in which software takes a first, unprompted assessment of the documents and puts forth a graphical representation (pretty charts and diagrams) of what the data contains. This is sometimes called Early Case Assessment or Data Visualization. Once data analysis is presented, the reviewer "informs" the software what he/she wants by batch-tagging key document groupings.

All of these techniques are variations of one another and each has its strengths and weaknesses for use in different types of matters and circumstances. (A topic for a future blog post, clearly!) The point here it is to recognize that Predictive Coding DOES NOT EQUAL Automated Review; it is simply one of several techniques to accomplish it.

Valora's position on Judge Peck's Commentary in (Da Silva v. Moore) Transcript

The transcript, articles and subsequent brouhaha are overblown. It is obvious that the parties are not particularly debating the efficacy of predictive coding, but rather the proper way to implement it, particularly when there are changes to the population volume or to the relevancy specifications. What is interesting to all of us is this:

Why is there so much wrangling over the size of the seed set and the need to re-seed it with changes?

How should such TAR systems and workflow handle changes? What are the differences between changes to document volume (count) and changes to review specifications?

Why has this ignited the blogosphere, to the point of blatant fact distortion?

There is considerable wrangling over the seed set because of 2 reasons: control and comfort. Judge Peck alludes to the second notion with his distinctions between a statistical sample and a "comfort sample." (His words.) The client service folks in the audience are laughing at this notion as they know full well the mistrust their clients have in "the numbers," while the statisticians present are utterly confounded. Surely, the statistical sample is in fact the most comfortable sample, anything else would be downright uncomfortable!

And, so we get to the second issue here: control. By registering vague, ill-described discomfort with the numbers, the attorneys regain some control by playing on fear. What don't we see? What didn't we get? As Judge Peck points out, it's not about the miniscule "misses," but rather the overall "sure-fires" and the ability to build a case around those.

One of the big problems in this matter is the conflating of two separate types of changes: changes in population size (adding documents) and changes in relevancy scope. These changes are completely independent of one another and should be managed separately. The size change affects the random sampling and so it should be regenerated each time new docs are added. The scope change affects the seed set coding and so it should be regenerated as well to reflect current, up-to-date specifications. In fact, the seed set coding should never be out of step with current specs, or it is obsolete. A far better TAR workflow design than random sampling to generate a seed set and fixed tagging of it is to assume change is part of the litigation and build in mechanisms to manage it. For starters, the seed set should be stratified per the attributes of the then-current document population, rather than random. It should be regenerated every time docs are added or removed. Next, a transparent, easy to edit ruleset should be used to track spec changes and show those over time as documents change their tags. Finally, TAR systems should be priced so that there is no disincentive to make such changes easily.

And finally, why all the hoopla? Because we all know that deep down this is where it is all going. In some ways we are rooting for Recommind (even if we would consider ourselves a competitor) because TAR is the better solution: lower cost, faster and more accurate. Now, if we could be smart about sampling, seed sets and inevitable changes to spec and volume, we'd really have something to shout about.

As I recall...

We at Valora are occasionally called upon to evaluate the results of traditional document review (read: manual, doc-by-doc review efforts), as a sort of post-project audit.  We use the usual precision and recall metrics to determine whether a document should have been tagged at all and whether it was done so correctly.  While both these measures are extremely important, the truth is it’s really all about the recall.  Let me explain.

If you are supposed to mark 1,000 documents as privileged, and your team only tags 200, does their precision on those 200 really matter?  In layman’s terms:  Do you really care how well those 200 docs scored on accuracy when the reviewers missed 800 documents in the first place?

This is not to suggest that there isn’t a crucial role for precision scoring.  There is!  But, not if you blow it on recall first.  In our above example, if the review team indeed found 1,000 documents and marked them privileged, you would surely wish to score the accuracy of those markings.  But, only if the recall fell within 20% +/- of the intended target.  Well below (or above) this mark, precision is useless info.  So, until as an industry we can demonstrate reliable recall on a regular basis, precision will just have to wait its turn.