Top AI models lose track when long records keep changing. Here’s how to keep your numbers right

On Tuesday, the Allen Institute for AI (Ai2), a nonprofit research lab, released StateBench, a test of how well AI models keep track of things that change over the course of a long document. Its finding matters to any business that hands records to a chatbot: top models are good at finding the latest fact, and much worse at following a long chain of changes.

What the test measured

The researchers sorted record-keeping problems into four levels, each with an everyday example:

• A value that never changes, like a birth date.
• A value that gets replaced, like a thermostat, where only the latest setting matters.
• A value where each update builds on the last, like a game score.
• Values that depend on each other, like a magician swapping cups, where finding the ball means following every swap.

As Ai2 puts it, some problems require “only finding the latest update,” while harder ones require “following a chain of changes, where one missed step can throw off the final answer.”

The team asked four frontier models, Google’s Gemini 3.8 Flash, Anthropic’s Claude Opus 5.5, and OpenAI’s GPT-6 Sol and Luna, to read long sequences of updates and give the final value of everything tracked. All four did well on the first two levels. On running totals and swaps, accuracy fell sharply as the inputs grew longer, dropping to 13% or lower on the longest problems at the hardest level. The accompanying paper reports that on both harder levels, the models were no better than chance once inputs reached somewhere between 10,000 and 64,000 tokens, the word pieces AI models read.

The tests used generated sequences, not real business records. Ai2 also notes a limit: StateBench tracks a fixed set of items, while real tasks “often introduce new entities over time.”

Why it matters for your business

Plenty of everyday records are the hard kind. Inventory changes with every sale, delivery, and return. A customer’s balance changes with every charge and payment. A job’s price changes with every change order, and a schedule changes every time two people trade shifts.

Paste a long history into a chatbot and ask where things stand, and it has to carry every change forward. That’s where the models in this test broke down.

Closer to home

Picture a feed and seed store in Enterprise that tracks its stock in a long email thread, a Dothan remodeler whose kitchen job has collected a dozen change orders, or a restaurant anywhere in the Wiregrass where servers swap shifts all week. Ask an AI assistant how many bags are left, what the job costs now, or who closes Saturday, and it may answer confidently and still be wrong.

Five ways to keep AI from losing count

1. Keep the running total in software. Let your accounting, inventory, or scheduling system do the arithmetic, and have AI read the current numbers instead of rebuilding them from history.
2. Ask for the latest, not the sum. The models handled questions about the newest value well. Point AI at the place where the current balance or count already lives.
3. Keep the history short. Accuracy fell as inputs grew. Work through long logs a week or a job at a time, and start each batch from a confirmed total.
4. Test with your real records, in their real format. Ai2 tests every problem in several formats because the format changed the results: small models that aced one version were near chance on the same problems written as spreadsheet edits. Take a past month where you know the right answers and compare. Here’s how to test AI on your own files fairly.
5. Check totals before money moves. Have a person confirm any count, balance, or price an AI produced before you order, bill, or pay.

None of this means AI can’t help with records. It means the math belongs in the systems built for it.

Where we can help

In the automations our workflow automation service builds, rules handle what’s predictable, and AI handles what used to need a person to read and decide, like messy documents and free-form email. We keep your accounting, scheduling, and inventory in step, flag what doesn’t line up across systems, and send anything uncertain to a person for a quick review. Have records that change all day? Book a free strategy call.

Sources

• Ai2: StateBench: Testing how language models track state changes over long documents, November 3, 2026
• arXiv: A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models, November 2, 2026