AUTOMATED CONTRACT REVIEW Machine Learning Comes to Corporate Law Session #133
Kingsley Martin KM Standards Amy Harvey & Michael Nogroski Chapman and Cutler SPEAKERS Julian Tsisin Google
AUTOMATED CONTRACT REVIEW Past, Present, and Future Kingsley Martin KMStandards LLC
Contract Tasks and the Contract Life Cycle One contract Pre signature Draft Review Many contracts Post signature Optimize Manage
Automation of Legal Tasks 1980+ Profile: mark relevant material 1990+ Search: find relevant material 2010+ Analyze: identify relevant elements & issues 2020+ Optimize: predict optimal outcome Optimize Analyze Search Profile 1980 1990 2000 2010 2020 2030
Evolving Technologies Machine Learning AI Expert Systems Predictive Analytics Deep Learning NLP Clustering Search Classification Structural Classification Database Data Extraction 1980 1990 2000 2010 2020 2030
Application to Contract Review source contracts Structural Classification Metadata Extraction Determines how contracts are organized what clauses they contain standard language for each clause full range of deal-specific alternatives Natural Language Identifies subject matter action object context Captures presence or absence of clauses key contract language key business terms names places dates amounts
Accuracy combined technologies MUTUAL NON-DISCLOSURE AGREEMENT This Non-Disclosure Agreement ("Agreement") is effective... 1. Definitions. When used in this Agreement: 1.1 "Affiliates" means any legal entity which a Party owns, which owns a Party, or which is under common ownership with such Party. 1.2 "Authorized Representatives" means either Party's Affiliates, directors, officers, managers, partners, employees, contractors, agents, accountants, legal counsels, advisors, and associates. 2. Purpose. The Parties have entered into this Agreement to exchange certain Confidential Information in connection with [PURPOSE OF DISCLOSURE] (the "Purpose"). 3. Exchange of Information. The Parties agree to exchange Confidential Information under the terms and conditions of this Agreement.... 11. Term. The term of this Agreement will commence on the Effective Date and continue [select: until terminated by either Party upon providing a written notice to the other Party/for period of (x) years]... 13. General Provisions 13.1 Entire Agreement. This Agreement constitutes the entire agreement between the Parties, and supersedes all prior agreements, representations and understandings of the Parties, written or oral. 13.2 Counterparts. This Agreement may be executed in counterparts, each of which will be deemed to be an original, but all of which, taken together, will constitute one and the same agreement. 13.3 Amendment. This Agreement may be amended only in writing by the Parties. 13.4 Notices. All notices permitted or required under this Agreement will be in writing and will be delivered in person or mailed by first class, registered or certified mail, postage prepaid, to the address of the Party specified in this Agreement or such other address as either Party may specify in writing. Such notice will be deemed to have been given upon receipt. 13.5 Assignment. This Agreement will not be assigned by either Party without the prior written consent of the other Party. 13.6 Governing Law. This Agreement will be governed by and construed in accordance with the laws of California, without regard to its conflict of laws rules. 13.7 Survival. The terms of this Agreement will survive the termination or expiration of this Agreement.... IN WITNESS WHEREOF, the Parties have each executed and delivered this Agreement as of the Effective Date. document article clause scope target context structure data semantics
Accuracy precision and recall CONTRACT STANDARDS GENERIC TEMPLATE This agreement is made on [AGREEMENT DATE][ (the "Effective Date")] between [PARTY ONE NAME], with its principal place of business at [PARTY ONE ADDRESS] (the "[PARTY ONE ABBREVIATION") and GENERIC LICENCE AGREEMENT [PARTY TWO NAME], [whose principal place of residence is at/with its This License Agreement ("Agreement") is effective as of [EFFECTIVE principal place of business at] [PARTY TWO ADDRESS] (the "PARTY DATE] (the "Effective Date") and is between [PARTY ONE NAME], with its TWO ABBREVIATION"). principal place of [PARTY ONE BUSINESS OR RESIDENCE] at [PARTY CONTENT LICENCE The parties AGREEMENT agree as follows: ONE ADDRESS] ("[PARTY ONE ABBR]"), and [PARTY TWO NAME], with Terms and 1. Conditions Definitions. In this agreement, in addition to the terms its principal place of [PARTY TWO BUSINESS OR RESIDENCE] at This content licence agreement defined consists above, of the the Information following Table, definitions these apply: [PARTY TWO ADDRESS] ("[PARTY TWO ABBR]") (together, the "Parties" terms and conditions and any attachments 1.1 "Affiliate" (together of any Person the "Agreement"). means, at the time the determination and each, a "Party"). 1 Definitions is made, any other Person Controlling, Controlled by, or under common 1 Definitions 1.1 In this Agreement, Control the following with, that definitions Person, whether apply unless directly or indirectly. 1.1 In this Agreement, the following definitions apply unless expressly stated otherwise. In 1.2 addition, "Business all terms Day" means in quotation a day marks other than in a Saturday, a expressly stated otherwise. In addition, all terms in quotation marks in the Information Table are Sunday, defined or terms any other in this day Agreement. on which the principal chartered banks the Information Table are defined terms in this Agreement. (a) "Confidential located Information" in [CITY means FOR REFERENCE information that BANKS] one party are not open for business. (a) "Confidential Information" means information that one (or an affiliate) discloses 1.3 to "Change the other in party Control" under means this Agreement, party (or an affiliate) discloses to the other party under this and that is marked as confidential (a) any or sale, would transfer, normally be other disposition of ownership Agreement, and that is marked as confidential or would normally considered confidential interests information [PARTY under the ONE] circumstances. that results in It a change in the Person or be considered confidential information under the circumstances. It does not include information Persons that holding the recipient the majority already voting knew, interest that does not include information that the recipient already knew, that becomes public through no fault (b) any of the merger, recipient, acquisition, that was or other reorganization of becomes public through no fault of the recipient, that was independently developed [PARTY by the ONE] recipient, that has or that the was same lawfully result as described in paragraph independently developed by the recipient, or that was lawfully given to the recipient by (a), a third or party. given to the recipient by a third party. (b) "Intellectual Property (c) Rights" any sale, means transfer, all copyright, or other moral disposition of all or (b) "Intellectual Property Rights" means all copyright, moral rights, patent rights, trade substantially marks, design all of rights, [PARTY rights ONE]'s in or assets. relating rights, patent rights, trade marks, design rights, rights in or relating to databases, rights in 1.4 or relating "Closing" to confidential means the completion information, of rights the transactions to databases, rights in or relating to confidential information, rights in relation to domain contemplated names, and under any this other agreement. intellectual property in relation to domain names, and any other intellectual property rights (registered or unregistered) throughout the world. rights (registered or unregistered) throughout the world. (c) "Licensed Content" means: (a) all content made available (c) "Licensed Content" means: (a) all content made available by Licensor to Google under this Agreement using the delivery by Licensor to Google under this Agreement using the delivery method in Attachment A; and (b) any Updates. (d) "Update(s)" means Licensor's updates, refreshes, corrections, or modifications to the Licensed Content. (e) "Users" means users of Google products and services, including users of Google APIs. recall hierarchical training sets general document type document category specific document type precision
The Future People Man && Machine > Man Machine Process Technology Content
MACHINE LEARNING AT CHAPMAN AND CUTLER Amy Harvey Michael Nogroski
Extracted Text Annotated Original Document
Provision Level Analysis Provision Title Number of times the provision appears in the document
Provision text preview
AUTOMATED CONTRACT REVIEW: MACHINE LEARNING COMES TO CORPORATE LAW Julian Tsisin
GOAL Improve visibility into agreements, decreasing risk and increasing opportunity to maximize financial and operational performance
Buy v. Build State of the art machine learning but Vendors should have domain expertise and best practices. Still developing but somewhat commoditized space (not a complete list; in alphabetical order) BrightLeaf Counselytics ebrevia Kira KM Standards LegalRobot RAVN Seal Software more...
EVALUATING ML-BASED SOFTWARE Can t test exact functionality unless pilot with many (thousands) of agreements Hosted v. On-Premise Underlying ML engine References
EVALUATING ML-BASED SOFTWARE What accuracy can we expect from humans? 4 lawyers independently labeled the same ~500 patents Met for consensus on labels for patents Individual lawyers labels vs consensus ( ground truth ):.75 precision,.68 recall average Zandor Accuracy:.78 precision /.80 recall Equal-to-human precision; better-than-human recall
LESSONS LEARNED Don't be afraid to fail Start with a small use cases; market it as Proof of Concept Find lawyers who are interested in the cool tech Training data is crucial