Sign inGet started

Evaluating an AI tool before you trust it with a bid

Practitioner15 min

By the end you can

  • Test a tool on a solicitation you already know.
  • Measure errors, not impressions.

Research on hallucination covers both metrics for measuring it and methods for reducing it 1.

Evaluate a tool the way that research does: with numbers you can compare across tools and over time, not a general feeling after a demo.

How Bids2Bank checks its own output

When a person checks Bids2Bank's supplier research by hand, each claimed value is marked right or wrong, and wrong means the words are not on the page or do not say that 2. A wrong value means the run must be investigated before the feature is used 3.

A value the run left as "not verified", although the page plainly shows it, is noted as a miss, not an error 4.

Check 1
In Bids2Bank's hand check of supplier research, a value left "not verified" that the page plainly shows is recorded as what?

Test on a solicitation you already know

Pick a solicitation your team has already worked. Write down 20 facts you can check against it: the due date, each Section M factor, page limits, a clause or two, the NAICS code. Ask the tool for each one and mark every answer right, wrong or missing. Then run the same 20 questions on a second solicitation.

Worked example. A capture lead asks a tool 20 questions about last year's solicitation. It answers 15 correctly, misses 2 and gets 3 wrong, one of them a page limit. That is 3 errors and 2 misses out of 20, and the team decides the tool may draft but every page limit and date must be checked by a person.

Count errors and misses separately, and write down which kinds of fact the tool got wrong. A tool that misses often but is never wrong may still save time; one that is confidently wrong about dates or limits needs a person behind it.

Ask how the tool reports its own failures

In a Bids2Bank color review, a quote that does not appear in the original solicitation text is dropped and counted 5. When library extraction drops a proposed field, the drop is counted, never returned 6. An agent run that fell back to its deterministic path is recorded as degraded, with its reason in the event log 7.

Ask any vendor the same questions: what does your tool do when it cannot support a sentence, and can you show me how often that happens?
Check 2
How does Bids2Bank treat a color-review quote that does not appear in the original solicitation text?
Progress is saved in this browser only.

Indented paragraphs are invented worked examples. Tool links open Bids2Bank and need a free account.

Next lesson: Building agents with skills and approvals