ContraxAwareLearning library

Core chapter 13 of 18

AI in Contract Management with Practical Guardrails

Use AI to accelerate review while preserving source verification, access controls and human accountability.

Executive summary

  • Give the machine a defined job. Reading a scan, locating a clause and deciding whether the business accepts an exception are three different jobs. Assign the decision to a person with authority.
  • Make a field carry its evidence. Keep the document version, clause location, unit and qualifier with each value people will act on. A correct number attached to the wrong date type can cost you a renewal.
  • Measure the expensive misses. Count the exceptions a reviewer should have found. Then measure how many the tool found, how many alerts wasted review time and which missed items could change a decision.
  • Keep uncertainty actionable. Give each unresolved fact a reason, an owner and a review deadline. Open questions belong in the next person's queue.

Part A · The Why

Know what the AI task actually is

OCR, extraction, retrieval, summarization and playbook comparison

Suppose a procurement manager asks, “Which of our suppliers needs written notice this month?” Several pieces of work sit inside that question. Optical character recognition turns scans into searchable characters, extraction proposes field values, and retrieval finds relevant text. Summarization shortens the passage, while a playbook comparison tests the wording against an approved company position.

An error at any of these stages can make its way into a convincing answer. Start by finding out precisely which job the tool is doing.

Trace one answer backward. The summary says the customer has thirty days to act. Did the system find the right agreement? Did it read “thirty days after receipt of invoice” from the payment section?

Did an amendment replace the original notice period? A fast summary of the wrong paragraph is still a bad basis for sending a letter. Assign an observable acceptance test to each stage, beginning with a legible source.

Set a stopping point in the request. “Return the notice clause, the stated deadline and the evidence used” gives a reviewer something concrete to evaluate. Deciding whether to renew requires different information, including service performance, alternatives and authority. Put that decision on the business owner's desk.

Missing context, ambiguous terms and conflicting documents

A portfolio contains families of documents. A master agreement sets general terms. An order may change particular terms for one service.

An amendment may change them again from a particular date. A service with two orders can have two defensible renewal answers. Capture which order, period, party and version a question covers before judging an answer.

Use distinct outcomes for an absent term, an unreadable page and competing interpretations. “No notice provision found in the three supplied documents” describes a search and its limits. “No notice is required” makes a much bigger claim.

A scan with a missing appendix gives you another outcome entirely. Request the appendix and name the person responsible for finding it.

Keep both sides of a conflict. Say the main agreement says sixty days and a later exhibit says thirty for one location. Preserve the two references, the location and the open question about scope.

The reviewer can then resolve the conflict without reconstructing how it was discovered. Until that happens, choose an internal follow-up early enough to protect both possible dates.

Model confidence versus demonstrated accuracy

Confidence describes a system's assessment of its output under its own method. Different tools may calculate or name that assessment differently. A bright green badge tells you little about how the system handles handwritten changes in your agreements. Establish the relationship between its output and independently checked results on your own mix of documents.

Give a reviewer ten fee clauses without showing the proposed extractions. Have them record currency, frequency, amount and qualification. Then compare each proposal with that record.

A result of 2,000 without “USD per month” fails the complete-field test even if the digits match. Write that acceptance rule before seeing the score.

Keep the easy and difficult sets separate. Clean Word templates help test routine work. A faxed order with handwritten initials tests something else.

When someone quotes a percentage, ask for the denominator, the document types, the fields assessed and the treatment of blanks. Those answers tell you whether the number answers your question.

Design a risk-based verification process

Field-level source evidence and human acceptance

Start with the action a field can trigger. An inaccurate industry classification might distort a chart. An inaccurate notice cutoff could put a decision beyond reach.

For each consequential field, record the proposed value, source file and version, clause location, scope, reviewer's judgment and date. Use one row per fact when two facts have different meanings.

Require the reviewer to state the unit aloud during practice. “Twenty-four thousand dollars annually” and “twenty-four thousand dollars per month” are different commitments. The same discipline applies to dates.

Record whether a date starts a term, ends it, requires dispatch or requires receipt. A calendar entry alone cannot carry all four meanings.

Acceptance should attach to the version someone actually read. If a later draft changes the paragraph, reopen the affected verification. Keep the old determination as history. This lets the next person distinguish an outdated answer that was once accurate from a mistake that needs investigation across the portfolio.

Sampling, critical-field checks and false negative costs

Group fields by consequence and reversibility. For a pilot, have a person check every notice deadline, party identity and financial commitment before anyone relies on it. Sample lower-impact descriptive fields across contract types and scan qualities. A random sample drawn from the whole pile may contain no handwritten exhibit at all.

Count missed qualifying items as well as noisy flags. If a tool wrongly flags five standard payment clauses, five people may lose time. If it misses one short nonrenewal window, the team may lose its opportunity to exit. Put a simple consequence beside every test error, such as additional review minutes, an incorrect budget assumption or a lost decision window.

Escalate by consequence even if the error is uncommon. A single mistaken interpretation of “received by” is reason to check adjacent deadline records before using them. The size of that check should follow the affected clause pattern, document population and upcoming decisions. Record what you reviewed and the basis for releasing the remainder.

Confidentiality, vendor data use and permission boundaries

List what you intend to send through each service. Contract text may contain employee information, confidential pricing or material subject to special handling. Ask who receives it, where it goes, whether it is retained or reused, which settings govern that use and how deletion works. Record the answers for the service and account configuration you actually use.

Test ordinary access with two fictional agreements. Give the procurement trainee access to one and ask for information about both. The expected result is a usable answer about the authorized agreement and no disclosure about the restricted one.

Count titles, snippets and excerpts as possible disclosures. Have the workspace administrator address unexpected visibility before expanding the pilot.

For lawyers using generative AI, the ABA discusses competence, confidentiality and supervision under the Model Rules. A legal department can use that discussion when developing its own policy with counsel. The responsible professional must determine the rules applicable in their jurisdiction. ABA Formal Opinion 512

Evaluate the tool against real failure modes

Negation, exceptions, tables and amendment precedence

Create paired examples that differ by one meaningful word or condition. “May renew” gives permission. “Automatically renews unless” introduces a default and an exception.

“Thirty days after invoice receipt” uses a different anchor from “thirty days after invoice date.” Put both versions in the set. A system that returns the same answer for both has revealed a useful weakness.

Include a fee table with separate setup and recurring columns. Put a subtotal below it. Test a page break that separates a heading from its numbers.

Then ask separately for one-time fees and annual recurring fees. The reviewer should be able to show which row and column produced each figure and whether tax, usage or a discount is included.

Add one amendment that changes the limit for a single service. A useful answer identifies the service and applicable period. If the model carries the new limit across the entire agreement, record the scope error even if it extracted the new number correctly.

Gold-standard labels, precision and recall

Create an answer key before running the tool. Two knowledgeable reviewers should independently label the difficult examples, compare results and document how disagreements were resolved. Allow “unresolved from supplied documents” as a correct outcome. If trained reviewers need a missing exhibit, a model shouldn't earn credit for guessing its contents.

In a fictional test of 100 passages, reviewers mark 20 as containing a payment exception. The system flags 18. Fifteen flags are correct and three are false alarms.

It misses five actual exceptions. Precision is fifteen correct flags divided by eighteen flags, about 83%. Recall is fifteen found exceptions divided by twenty actual exceptions, or 75%.

Those measures answer different operating questions. Precision helps estimate how much reviewer time will go toward unhelpful flags. Recall tells you how many known exceptions escaped detection.

Here 77 ordinary passages were correctly left alone, so overall accuracy is 92%. The five missed exceptions still require review. Show the underlying counts beside every percentage.

A 92% accuracy score can hide five missed exceptions

Illustrative review of 100 payment passages. A human answer key identifies 20 exceptions and 80 ordinary passages.

  1. 15 passagesCorrect exception flags

    These flag the right issue. They count toward precision and recall.

  2. 3 passagesFalse alarms

    Of 18 flags, 15 were correct. Precision is 15 ÷ 18, about 83%.

  3. 5 passagesMissed exceptions

    Only 15 of 20 expected exceptions were found. Recall is 15 ÷ 20, or 75%.

  4. 77 passagesCorrect ordinary results

    The tool correctly leaves these unflagged. Overall accuracy is (15 + 77) ÷ 100, or 92%.

Inspect the five misses before deciding how to use the tool. The same counts say little about the cost of a missed condition.

Change monitoring and an error escalation log

Give each error a short identifier, date, affected task, expected answer, actual answer, evidence and consequence. Record the product or configuration information available at the time, the source document revision, who owns the investigation and when the corrected example will be retested. This provides a reproducible starting point even when the service manages the model internally.

Keep a small fixed regression set alongside fresh material. Reuse the fixed set after changes to extraction instructions, document processing, the approved playbook or the product. Rotate additional cases to catch failure patterns the team has never rehearsed. A familiar set is useful for comparison, but eventually tells you more about the set than the portfolio.

NIST's voluntary AI Risk Management Framework and its generative AI profile provide useful context for assigning responsibility, evaluating risks and checking results in the setting where a tool is used. Give an individual the authority to pause reliance on a field until a significant error is understood. NIST AI Risk Management Framework

A fictional case of two correct dates and one wrong reminder

A regional equipment distributor pilots extraction on 24 service agreements. In one draft, February 28 ends the current term and January 29 is the last day for the counterparty to receive nonrenewal notice. The tool locates February 28 correctly.

A spreadsheet import places it in the notice column. Every character is accurate and the resulting business instruction is wrong.

The contract administrator traces the failure to a mapping that used a single field called “renewal.” She separates the term end, notice receipt deadline and planned internal decision date. Her team checks all 24 records against the relevant clauses and sends ambiguous entries to named owners. During the exercise, a second agreement turns out to require dispatch by a date rather than receipt.

The pilot's result is a test anyone can repeat. Hand a colleague the record and ask what action happens, who takes it, when the other party must receive or be sent the notice, and which version establishes the answer. If the colleague has to infer the meaning of a date from its column position, the record needs another pass.

Part B · The How

When testing AI contract management software, give it an agreement and a later amendment that changes the notice period. Ask which period applies today and have the reviewer open the passage supporting the answer.

Review extracted information in ContraxAware

Open the original beside proposed fields

Start on REVIEW & CONFIRM for the fictional Northstar Services Agreement. Read the heading and opening lines in Original document. The source calls itself an unsigned draft.

On the right, Where is it in the process? shows Draft — needs review. The two pieces of evidence agree about its stage.

  1. In Contract details, locate Overview.
  2. Compare Contract name with the title on the left.
  3. Under Vendor / company, compare Northstar Technologies with the supplier named in the original. Harbor Retail Co. is the customer.
  4. Click Show in document beneath a field when you need its source. View source · Word text is also visible under these two fields.
The original unsigned Northstar agreement sits beside its proposed draft status, contract name and vendor.
Northstar import review. The original unsigned Northstar agreement sits beside its proposed draft status, contract name and vendor.View full size

Why this matters. A field can contain the correct legal name and still represent the wrong side of the agreement. Checking the opening sentence resolves the role question. ContraxAware AI contract extraction

Validate dates, amounts, parties and source locations

Read section 2 in the left document panel. The practice term begins November 7, 2025 and ends November 6, 2026. Renewal starts November 7, 2026.

Written nonrenewal notice must be received by October 27, 2026. In your exercise log, give each event its own row and retain the word “received.”

Read section 3. It states USD 2,000 per month and USD 24,000 annually. The arithmetic agrees because twelve months multiplied by USD 2,000 gives USD 24,000.

The invoice condition says undisputed invoices are due within thirty calendar days of receipt. Entering only “net 30” would discard the trigger and the qualification.

The visible record fields in the image cover the title and vendor. Use the left-hand text as evidence for the dates and fees in your exercise log. A claim that a field already contains a fee or a deadline requires viewing that field in your own record.

Correct or leave uncertain information unresolved visibly

Click 2 import notes to check and compare each note with the source available to you. If a proposed value can be resolved from the supplied wording, record the correction and the evidence. When evidence is missing, preserve the original proposal, write the outstanding question and assign its follow-up.

For example, the record might identify the date a service report is due without establishing who internally verifies receipt. The missing verifier is a business assignment. Guessing a name in a data field creates the appearance of a completed handoff. Ask the service owner who actually performs the check and when they can confirm it.

Use Save for Later if investigation remains. Use Save & Open when the draft is ready to continue. The screen explains that nothing is sent automatically. Keep any unresolved business or legal decision visible when handing the record to its next reviewer.

Use search and guided review

Ask a bounded question and inspect the cited evidence

Try a narrow search first. Ask which wording in the Northstar Services Agreement controls receipt of nonrenewal notice. Identify the agreement explicitly.

Check the matching passage, the document version and the adjacent language about automatic renewal. A result about another Northstar document should send you back to the scope.

Next ask which document supplies a current signature date. The provided Northstar source is unsigned. A responsible response can identify that limitation.

An answer that picks the date appearing after “Version 2” treats a draft date as evidence of signing. Save that example for your training set. ContraxAware contract search

Compare selected terms with approved guidance

Choose the playbook corresponding to the agreement type and your company's role. Review a proposed departure by reading the clause and the approved position together. Watch for limitations in nearby definitions, exclusions and exhibits. The reason the company cares about a clause should be visible to the person judging the exception.

In Northstar, a proposed insurance increase belongs to draft review. Before treating USD 2 million as an operating requirement, establish which wording was ultimately accepted, by whom and in what version. If all you have is a redline, the useful output is a question for the authorized reviewer. ContraxAware AI contract review

Give the reviewer a compact handoff. Identify the clause and source version, state the proposed answer, show the competing interpretation and explain the decision that depends on it. “Please review insurance” leaves them to repeat the investigation. “Does this proposed increase require risk approval under our supplier policy?” gives them a decision.

Set a due date that protects the next business step. If a pricing assumption may change the purchase order, name the finance owner and the planned order date. Keep the question open until the identified person responds. Their response should specify scope and version so another reviewer can tell whether the next draft needs a fresh decision.

Measure and improve safely

Run a small labeled training set

Choose ten fact questions. Use nine Northstar facts and one separate fictional order that says a disputed invoice timing rule appears in a missing schedule. Prepare the expected answers independently. Record the file, version, question, source location, expected value, proposed value, human disposition and likely consequence in each row.

Use the same questions and source package when comparing two runs. Record “no answer” distinctly from a wrong answer. A justified request for the missing schedule earns credit on the ambiguous order. Forcing a numeric answer would train the team to reward unsupported certainty.

Log false positives and missed critical terms

After a run, mark every proposed exception that the human reviewer dismisses. Then inspect every expected exception the run failed to surface. This second pass needs an answer key.

The independently labeled cases reveal what the list of findings missed. Count unresolved scope disagreements separately until a qualified person settles the label.

Write one test consequence per miss. A swapped invoice trigger could shorten payment time. A missed exclusion could make the review appear complete.

Assign the person best placed to judge that consequence. They can then decide whether a manual check should remain in place for all affected records.

Repeat checks after material configuration or product changes

Reserve a clean copy of the source package and answer key. After a material change, repeat the test and compare old and new dispositions by field. Track whether an apparent improvement simply moved an error from the review queue into an unflagged result. Anyone approving wider use should see unresolved high-consequence misses.

Practice explaining a confident wrong answer

Use nine Northstar rows for supplier, customer, start, term end, notice receipt deadline, renewal start, monthly fee, annual fee and invoice payment trigger. Add the missing-schedule order as row ten. Exchange logs with a colleague. Each should be able to reproduce the other's answer or identify the missing evidence.

For a harder exercise, imagine 100 passages with one critical exception. A process reports “ordinary” on all 100. It is correct on 99 and misses the only exception.

Its critical-exception recall is zero out of one. Explain to a project sponsor why a human still needs to review that category, then propose three additional examples that could expose the same weakness.

John's Builder Note

I want the source close enough to the proposed value that checking it feels easier than explaining later why nobody checked. Before adding another score or dashboard, watch a reviewer answer one question. If they have to hunt for the right version, fix that part first.

Questions from the field

Does a blank extracted field mean a clause is absent?

It may mean the clause is absent, the file is incomplete, the scan is unreadable or the question was too broad. Check the source package and record the specific reason you can establish. Assign any remaining uncertainty to a person.

Can we use a signed contract as our answer key?

Use it as source evidence, then label the fields and exceptions independently. A signed copy can contain a negotiated concession, unusual drafting or a term later amended. In a playbook test, the company policy provides a separate comparison standard.

Optional knowledge check

Choose your answer, then open the explanation.

1. AI returns a confident renewal date. What is the next control?
Show answer

Check the source clause and relevant amendments. Confidence is not ground truth. The source and any applicable changes establish the fact.

2. A tool misses one rare but critical clause. Which metric matters?
Show answer

Error impact and critical-term recall. A high aggregate accuracy score can hide costly omissions. Evaluate the errors relevant to the risk.

3. Who owns the final decision after AI review?
Show answer

The authorized human reviewer. AI assists analysis. Accountability for legal, commercial and operational decisions remains with authorized people.

Admin self-audit checklist

Further reading