Finished Build Your First Agent last night and had a matcher running against the sample set before I went to bed. Forty minutes, most of it reading. Posting the parts that did not work, because the parts that did work are just the course.
Thing one. I wrote the quantity test as an exact match because that is how I would have described the rule out loud. It routed every partial delivery, which is about a fifth of the sample. Changing it to less than or equal took ten seconds and moved my approval rate from 51 percent to 78.
Thing two. My first tolerance was 50 dollars flat. On a 240 dollar invoice that is a 20 percent band, which is far too loose, and on a 40,000 dollar one it is nothing. The course says within 2 percent or 50 dollars whichever is smaller and I ignored it because I thought I had a better idea. I did not have a better idea.
Thing three, and this is the one I would not have found without reading a run. Two invoices resolved to three purchase orders each. The reference field in the sample data has a couple of records where the same string appears on more than one order. The agent picked one and was confident about it. Nothing in the decision was wrong. The fetch was wrong, and it is invisible unless you open the run and look at what it pulled.
Ended at 81 percent automatic with an override rate I cannot measure yet because I am the only reviewer. Next thing I want to build is the duplicate checker the course says should be its own agent. If anyone has done that one against real data I would like to know what lookback window you settled on.