Methodology
Methodology: how we measure
Every number on this site can be traced back to a specific document, model output, and run cost. Here is how they are produced.
What we measure
Six everyday business tasks: extracting 13 fields from an invoice (supplier, identifiers, document number, payment reference, dates, net amount and tax by rate, total, currency, and bank account); triaging customer emails (category, priority, sentiment, customer, order and invoice numbers, amount, currency, and deadline); finding personal data in business texts, for example before they are sent to an AI service (names, email addresses, phone numbers, postal addresses, dates of birth, national IDs, and bank accounts); extracting the key terms of a contract (parties, contract type, price and billing period, start and end dates, notice period, late-payment penalty, and governing law); summarizing a document so that the summary includes the key amounts, dates, and owners, leaves out rejected amounts and superseded dates, and stays within a word limit; and answering questions about a document, including questions the document does not answer. We report the share of correct fields and the share of documents without a single error, because for a business the second number often matters more. Documents are in Czech, US English, German, French, and Spanish.
The test set
All documents are synthetic: generated from random but valid data (Czech company IDs and national ID numbers with valid checksums, bank accounts following Czech National Bank rules, EINs, and ABA routing numbers), so the correct answer is the source data, not a manual transcription, and contains no typos. No real personal or company data: email addresses use .example domains, US phone numbers use the fictional 555-01xx range, and SSNs come from a range that is never issued. Documents in different languages come in pairs with the same structure, which lets us measure the language gap.
Invoices come in eight layouts with realistic variants (supplier not VAT-registered, deposit invoice, credit note, foreign currency) and at four levels: 30% PDF text, 30% clean image, 30% scan (rotation, noise, JPEG), 10% photo (perspective, shadow). One third of every set is non-public; for that part we publish only aggregate scores, so a model cannot score well by having seen the answers.
Emails come as a single message, with a long signature, as a thread with quoted earlier messages, and as an email forwarded by a colleague; some deadlines are relative (“by Friday”). Priority follows fixed rules written in the prompt, so it is not a matter of opinion. Texts with personal data (HR records, support tickets, meeting minutes, emails) deliberately also contain data that is not personal: company names derived from surnames, company IDs, generic addresses such as info@, order numbers, and dates that are not dates of birth.
How a run works
All models are called through OpenRouter with the same prompt in the document's language, at temperature 0, each pinned to one fixed provider so the results are reproducible. If a model supports structured output (JSON schema), we use it; if the provider rejects the schema, we retry without it and record that. If the answer is not valid JSON, we ask once more and record that too. For every call we store the model, prompt and dataset versions, tokens, cost, latency, and provider. Outputs are cached, so each week we pay only for new models, documents, and prompts. New models are measured on text documents through the half-price Batch API. The answers are the same, but latency cannot be measured that way, so we measure speed on a sample of 10 documents per language with regular calls, and show the cost at the standard price so it stays comparable with the other models.
How we score
A field is correct when it matches the correct value exactly after the format is normalized: dates to YYYY-MM-DD, amounts to a number with two decimals (either separator convention), and whitespace in identifiers and account numbers is ignored. Diacritics are not normalized: “Horak” instead of “Horák” is an error. Errors are categorized (missing field, wrong value, lost diacritics, swapped dates, net instead of total, wrong rate, number format, invalid JSON). The bar for invoices is 95% correct fields.
Emails: category, priority, and sentiment must match exactly; relative deadlines must resolve to the right date. Personal data: each field is a list and is correct only when it is complete and has nothing extra; a company ID flagged as personal data counts as an extra value. The bar is 95% correct fields for every task.
Costs and the language tax
Cost is the actual token spend as returned by the API in USD, scaled to 1,000 documents. Conversion to other currencies happens only for display, at the rate shown in the footer. The language tax is measured on a parallel corpus of 50 paragraphs in Czech, German, French, Spanish, Polish, and English: we send the text with a minimal output limit, subtract the overhead of an empty prompt, and compare input tokens with English.
Real cost adds the labor of someone who has to find and fix every document with an error: 3 minutes per document at 450 CZK / 35 EUR / 40 USD per hour. Speed is the median response time (90th percentile in the tooltip); valid answers is the share of documents with valid JSON. API rate-limit errors are not counted against a model: we retry them until every document has an answer.
The real cost calculator uses measured costs where we have documents in the language. For other languages it estimates: the surcharge measured on actual documents in another language, scaled by the language tax of the chosen language (an invoice image costs the same in every language, so the plain token ratio would overstate the surcharge). Estimates are marked ≈ and show no accuracy.
The language gap
The accuracy difference between Czech and English documents is computed only on fields both versions have. US invoices have no VAT ID or tax point, so those fields are left out of the comparison.
Limitations
Synthetic documents do not cover everything a real company sees (handwritten notes, multi-page invoices, unusual layouts). With about 150 documents per language, accuracy has an uncertainty of roughly ±1 percentage point; don't treat smaller differences between models as decisive. Models and prices change, so every number shows its run date.
Every model gets the same prompt, written once per task and language and not tuned for any particular model: a single message with no system prompt and no worked examples. We measure how well a model handles a reasonable prompt on the first try, not how much can be squeezed out of it by tuning. With a prompt adapted to a specific model and your documents, results can be better, especially for format errors (how amounts or identifiers are written). So treat small differences between the top models as a tie. The prompt is shown with every example output.
Independence
Rankings and recommendations cannot be bought. No model vendor pays us for placement. If you find a mistake in one of our correct values or in the scoring, let us know; we'll fix it and note the change.
Current run: 20261010T220935Z-c30495 from October 10, 2026, dataset version 0.1.0.