Technology Assisted Review

Technology Assisted Review (“TAR”)

by Dr Allison Stanfield

TAR in Australia

The Federal Court of Australia now recognises the use of TAR in electronic discovery, in its updated practice note Federal Court of Australia Technology and the Court Practice Note (GPN-TECH).  Similarly, the Supreme Court of Victoria has released a new practice note, also recognising the use of TAR:  Supreme Court of Victoria, Practice Note SC Gen 5 Technology in Civil Litigation.

While many Australian law firms have been using this technology to assist in review of electronic documents for discovery for some years, and other jurisdictions have been using TAR, endorsed by the courts, it is only recently that Australian courts have accepted the use of TAR.  In the Federal Court of Australia on 7 November 2016, Murphy J made orders in Money Max Int Pty Ltd v QBE Insurance Group Ltd[1] that the applicant provide a report from its e.discovery provider ‘describing with particularity the manner in which the respondent has applied technology assisted review (TAR) for the purposes of giving discovery’. In particular, the report was to set out:[2]

  • the nature and technical parameters of the TAR algorithm used;
  • the process for selecting and coding the training set of documents;
  • the process for selecting and coding the validation set of documents;
  • the process for training the algorithm to identify relevant documents for production, including the level of relevance applied;
  • the process for validation and testing, including disclosure of analyses relating to the accuracy, validation or quality of documents produced;
  • the number of documents in the complete data set identified as relevant and
  • irrelevant following the application of TAR and, with respect to the relevant
  • documents, the number of documents withheld on the basis of privilege;
  • the search terms applied in conjunction with TAR; and
  • the process followed with respect to potentially privileged documents.

In McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors[3], Vickery J made orders that the use of TAR, stating that such orders fell within the overarching purpose of Civil Procedure Act 2010 (Vic) s 7.

Vickery J noted that TAR has been recognised and ‘endorsed’ in other jurisdictions[4] and referred to the case of Pyrrho Investments Limited v MWB Property Limited[5], where the High Court of the United Kingdom set out how the TAR process works.  In that case, Master Matthews stated that predictive coding is just as accurate, if not more so than a manual review using keyword searches, and also estimated that predictive coding would offer significant cost savings in this particular case and that the possible disclosure of over two million documents done via traditional manual review would be disproportionate and ‘unreasonable’.

In His Honour also referred to the case of Irish Bank Resolution Corporation Ltd & Ors v Quinn & Ors[6]  and quoted the following excerpts from that judgment:[7]

‘66.  The evidence establishes, that in discovery of large data sets, technology assisted review using predictive coding is at least as accurate as, and, probably more accurate than, the manual or linear method in identifying relevant documents. Furthermore, the plaintiff’s expert, Mr. Crowley exhibits a number of studies which have examined the effectiveness of a purely manual review of documents compared to using TAR and predictive coding. One such study, by Grossman and Cormack, highlighted that manual review results in less relevant documents being identified. The level of recall in this study was found to range between 20% and 83%. A further study, as part of the 2009 Text Retrieval Conference, found the average recall and precision to be 59.3% and 31.7% respectively using manual review, compared to 76.7% and 84.7% when using TAR. What is clear, and accepted by Mr. Crowley, is that no method of identification is guaranteed to return all relevant documents.

67.  If one were to assume that TAR will only be equally as effective, but no more effective, than a manual review ,the fact remains that using TAR will still allow for a more expeditious and economical discovery process …

69.  Pursuant to the legal authorities which I have cited supra, and with particular reference to the albeit limited Irish jurisprudence on the topic, I am satisfied that, provided the process has sufficient transparency, Technology Assisted Review using predictive coding discharges a party’s discovery obligations under Order 31, rule.12.’

Vickery J made reference to ‘TEC SOP 5 [TAR]’ which was to be an interim measure and apply for use in the TEC List pending Practice Note SC Gen 5 Technology in Civil Litigation.

TAR:  What is it?

TAR is being lauded as a way to locate relevant documents for discovery, and of course it is only relevant documents that can be admitted as evidence.  TAR helps lawyers find relevant documents in a much more efficient and cost effective manner compared to traditional linear review which often meant relying on junior lawyers who may not have fully understood the case, and who were faced with a long and tedious process of reviewing hundreds of documents during an eight (or more) hour shift.  Technology on the other hand, is not subject to fatigue, hangovers, gossip or being ill-informed.  These tools use every word in every document to assign relevance as determined by the senior lawyer on the matter.

TAR is burgeoning artificial intelligence, which can include a number of different ‘clever’ technologies, and research is ongoing in order to find even more clever ways of finding what lawyers seek in a repository of documents.  These technologies include ‘clustering’, ‘concept searching’, ‘email threading’, ‘near de-duplication’ and ‘predictive coding’.  In-built features, such as predictive coding are being celebrated as the answer to help curtail ever-increasing litigation costs for both in-house and external counsel.

Clustering technology

Clustering technology can be used to group together emails and other electronic documents that relate to the same topic.  Clustering relies on statistical relationships which result in documents with similar words being clustered together.  The clustering software compares each document in a set to a ‘pivot’ document which has already identified as relevant.  The more words a document has in common with the pivot document, the more likely it is to be about the same topic and therefore relevant.  The clustering software ranks documents based on their statistical similarity to the pivot document.  Clustering can be used as a helpful tool for initial categorisation.  The algorithms in the software analyse the actual content of individual documents- allowing them to be sorted into related ‘clusters’ or groups.  The solution scans the content of each document and, by cross-referencing against a specialised index, identifies recurring key concepts. Documents dealing with discrete concepts can then be batched to individual reviewers, again so documents of a similar concept can be reviewed together.[8]

Concept searching

Concept searching allows the technology to determine relevance by associating words with particular concepts.  For example, if the term ‘Java’ is being searched, then the concept search engine would be able to identify whether it is ‘Java’ the Indonesian island, ‘Java’ the scripting language or ‘Java’ coffee beans are more relevant to the user.  The concept search engine will still locate the other concepts, but will order them lower in relevance ranking than the relevant concepts.  When using a tool such as ‘concept searching’ a reviewer’s workflow can be set so that the reviewer can review documents that may be associated with a particular issue or concept, so that they are reviewing documents that are similar in nature.  In a traditional linear review, two different reviewers may review documents that are of similar concept, but this correlation may be missed because the two documents are reviewed in context with each other.  By utilising the power of the technology, the efficiency of the review increases enormously.  Each reviewer would then see all of the documents related to a particular concept and this approach gives the reviewer additional context and enables him or her to quickly move through each conceptual batch, coding with more accuracy and consistency.  By the time he or she finishes a particular batch, a reviewer should be an ‘expert’ on whatever concept was grouped into that batch.  Through conceptual batching there are advantages to be made where teams can structure the review to better meet the team’s priorities. While the conceptual groups are generally software-created, once generated, a quick check of each cluster allows the case team to select those that are most relevant or most interesting for priority review. Likewise, conceptual clusters that are clearly irrelevant can be de-prioritised or bulk-tagged as such.[9]

Email threading

‘Email Threading’[10] is another example of where technology assisted review really increases productivity.  Email Threading allows the reviewer to simply review the email that is last in the email thread; that email will include the whole conversation and the reviewer can determine if the whole thread is relevant or not.  Therefore, instead of reviewing a number of related documents, or again seeing the documents out of context with one another, one document is reviewed to determine the relevance of many documents that are related.  In a traditional review, no single reviewer is likely to see the entire thread and therefore misses out on the whole conversation.

Near de-duplication

Near de-duplication allows documents that are similar, but not identical, to be identified and grouped together, based on a certain percentage similarly, which is set by the user when conducting the near de-duplicate search.[11]  A pivot document is selected against which similar documents are compared, and then highlighted to the user.  Differences between each similar document as compared to the pivot document are marked up so that the user can review these to determine if such documents are indeed duplicates for the purposes of the review, or for example, a different version of the pivot document.  The differences are highlighted in much the same way that differences are highlighted using the ‘compare’ function in MS Word.

Predictive coding

Predictive coding is a method where the user can ‘train’ the system to recognise documents that are relevant.  A senior lawyer will be presented with a random set of, say, 500 documents from the repository which the lawyer will then mark as ‘relevant’ or ‘not relevant’.  The technology will then determine, from the words in each of the relevant documents, what other documents are relevant.  The lawyer can review further randomly presented sets of documents, until the system learns what is relevant.  There are two primary terms in predictive coding; precision and recall.  Precision is the percentage of documents that lawyers review that are actually relevant. It is a measure of how efficient the reviewers are, and how much time is wasted reviewing non-relevant documents.  The higher the precision rate percentage, arguably the more efficient and cost effective the review.  Recall is an illustration of how many documents are being missed and are not reviewed at all.  In a perfect world with a reviewer who never makes a mistake, he or she would review every document in the document repository and would have 100% recall. The lower the recall rate the more relevant documents are missing.

To compare the effectiveness of predictive coding with other review methods such as traditional linear (or manual) review or keyword searching or predictive coding, the results can be measured by the levels of precision and recall.  Judge Cote in the New York District Court has confirmed that ‘predictive coding had a better track record in the production of responsive documents than human review’.[12]  Her Honour went on to say that although both predictive coding and human review fell short of identifying for production all of the documents the parties in litigation might wish to see, ‘no one should expect perfection for this process’[13].  Her Honour made the point that parties in litigation are required to act in good faith during discovery and that production of documents can be a herculean undertaking often requiring clients to pay vast sums of money.  All that can be expected, said her Honour, was that ‘good faith, diligent commitment to produce all responsive documents uncovered when following the protocols to which the parties have agreed, or which a court has ordered’.  The point of this case is to highlight that the use of technology such as predictive coding is becoming an accepted method of review during discovery and that indeed, can be more accurate than human review.  The court made reference to an article published by Grossman and Cormack in Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review,[14] where the authors compared the results of a review by humans against a review done using predictive coding; the results showed that predictive coding was more accurate and efficient.

Decisions in the USA

In the United States of America, in the case of Da Silva Moore v Publicis Groupe,[15] Magistrate Andrew J Peck, issued the first decision in a court in the United States of America, specifically addressing the use of predictive coding as a replacement for traditional linear document review.  During argument, the plaintiffs expressed concerns about the accuracy of the original coding and the possibility that the software would overlook relevant documents.  Judge Peck stated that while many lawyers have embraced the technology, several are reluctant to because of the risk of legal sanction. With the order, Judge Peck has now removed that risk. As the court noted, ‘statistics clearly show that computerized searches are at least as accurate, if not more so, than manual review.’[16]  Citing a recent study, Judge Peck claimed that technology-assisted review is more accurate and fifty times more economical than exhaustive manual review. The ruling concluded with Judge Peck reasoning that ‘the use of predictive coding was appropriate considering…the superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches)’.[17]

Judge Peck’s decision exemplifies the changing nature of discovery for lawyers. In his ruling, Judge Peck stated his long held position the legal industry needs to embrace predicative coding and other technological processes as they continue to play an increasingly useful and relevant role in the justice system.  Addressing lawyers, Judge Peck stated:

‘What the bar should take away from this opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review.’[18]

In the subsequent case of Rio Tinto PLC v. Vale S.A.,[19] Judge Peck, after providing a brief history of cases where courts have allowed technology assisted review (TAR) where the parties agreed, Judge Peck stated that ‘it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it’.[20]  Judge Peck noted that though the extent to which adverse parties must cooperate in sharing TAR training documents is unsettled, the parties may choose to cooperate, as they did in this case, and should be encouraged to do so.  Finally, Judge Peck stressed that ‘it is inappropriate to hold TAR to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review’.[21]

With predictive coding, instead of using keywords to find documents, entire documents are indexed and the system is ‘taught’ which documents are relevant and which are not relevant, by having a lawyer review a random set of documents, and the system then uses algorithms to ‘learn’ what is relevant from the relevant documents selected.  The system then finds documents that are conceptually similar to the relevant documents.  Through rounds of teaching the system, say 1,000 documents at a time, the system is able to keep increasing the recall percentage until the high standard established by Judge Peck is achieved.  Although the algorithms are advanced and not transparent to a lay user, the concept is not totally foreign, as anyone using a Google search has experienced advanced algorithms finding the webpages they intend (not simply which words appear in the websites).

In a typical review undertaken by paralegals, a document set of 35,000 documents might achieve a recall rate of about 50%, in other words, half of the relevant documents may be missed.  By contrast, if a senior associate reviews 4,500 utilising the random review process set out above, she might also achieve a recall rate of 50%.  However, if the senior associate reviews 10,000 documents, the recall rate can be increased to 80%, which is the standard that Judge Peck advocates.


The EDRM now includes a standard for Technology Assisted Review (which the EDRM names ‘computer assisted review’).[22]

Figure 1:  Computer Assisted Review Reference Model

Academic studies of TAR

Cormack and Grossman recently conducted a review of the best way in which the use of TAR should be conducted.  The study looked at three types of TAR tools:  Continuous Active Learning (‘CAL’), Simple Active Learning (‘SAL’) and Simple Passive Learning (‘SPL’).[23]  Essentially, all three use TAR to assist in ‘training’ the system to find relevant documents based on which documents the legal team code as ‘relevant’.  Each method uses a process whereby a set of documents (the ‘training set’), say 1,000 documents, is coded by a senior lawyer as ‘relevant’ or ‘not relevant’ which the system then uses to ‘learn’ which other documents might be relevant as well.  This process is repeated several times until the review team is satisfied that a sufficient level of relevant documents have been found.  The difference between the three processes is whether randomly selected documents are used, or whether the set of documents has been located via a non-random method such as using basic keyword searching.  In the CAL method, the 1,000 documents are selected using keyword searches and then the documents that are coded by the lawyer are used to train a learning algorithm, which scores each document in the collection by the likelihood of it being relevant.  In SAL, the set of documents can be selected randomly or non-randomly, but then subsequent document sets for coding by the reviewer are selected based on those about which the learning algorithm is least certain.  With SPL, the document set is selected randomly and relies on the review team to work on an iterative basis until there is some certainty that the review set is ‘adequate’.  The study concluded that when keyword searches are used to select all of the training sets, the result was superior to that achieved when a random selection is used, and summed up that ‘random training tends to be biased in favour of commonly occurring types of relevant documents, at the expense of rare types.  Non-random training can counter this bias by uncovering relevant examples of rare types of documents that would be unlikely to appear in a random sample’.  Such studies are extremely valuable in learning how best to use this technology, however, further guidelines and endorsement from the courts would be welcome.


These search technologies are crucial in assisting lawyers to find electronic evidence that is relevant, since any documents that are not relevant will not be admissible.  Further, it is only relevant documents that must be authenticated and which would be subject to any of the exclusionary rules of evidence.  Therefore, it is vital that relevant documents are located, and also any documents to which privilege applies so that these are not inadvertently discovered.  However, the key to lawyers taking up the use of such technology, is through education, both at an undergraduate level, and for practitioners.

[1] Federal Court of Australia (Murphy J) VID 513/2015.

[2] Ibid at [5].

[3] (No 1) [2016] VSC 734.

[4] Ibid at [18].

[5] [2016] EWHC 256 (Ch).

[6] [2015] IEHC 175.

[7] McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors above n 353 at [23].

[8] John Jay College of Criminal Justice, Towards Scalable E-discovery Using Content-based Hierarchical File Clustering (John Jay College of Criminal Justice, 2013), 23.

[9] See further:  EDRM Website on Search Methodologies:  <> at 11 January 2017.

[10] See further:  EDRM Website definition of email threading: at 11 January 2017.

[11] See further:  EDRM Website definition of near duplicate detection:  <> at 11 January 2017.

[12] Federal Housing Finance Agency v HSBC North America Holdings Inc, et al 2014 WL 584300, 3.

[13] Ibid.

[14] Maura R. Grossman & Gordon V. Cormack, ‘Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review’ (2011) 17(3) Richmond Journal of Law and Technology 1, 37.

[15] 11-civ-1279 (ALC) (AJP), U.S. Dist. LEXIS 23350 (S.D.N.Y. Feb. 24, 2012).

[16] Ibid 28-29.

[17] Ibid.

[18] Ibid.

[19] 2015 WL 872294 (S.D.N.Y. Mar. 2, 2015).

[20] Ibid at 2.

[21] Ibid at 3.

[22] Electronic Discovery Reference Model website:  <> at 11 January 2017.

[23] Gordon V. Cormack and Maura R. Grossman, Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, at:  <> at 11 January 2017.

© 2017 Allison Stanfield