Qwen
Qwen3.8 Flash
How Qwen3.8 Flash by Qwen handles realistic business documents: accuracy, real cost, speed, and where it goes wrong.
List price: $0.15 per 1M input tokens, $0.47 per 1M output tokens · qwen/qwen3.8-flash
Results
| Task · documents | Rank | Correct fieldsThe bar: the share of correct fields below which manual fixes cost more than a cheaper model saves. The black mark on the meter shows it. | Per 1,000 documents | To fix / 1,000How many of 1,000 documents have at least one wrong field and need someone to check and fix them. | Speed (median) |
|---|---|---|---|---|---|
| Contract clauses · English (US) | 13/14 | 95.13% | $0.62 | 307 | 14.1 sec |
| Contract clauses · Czech | 12/14 | 99.90% | $0.63 | 13 | 13.6 sec |
| Contract clauses · German | 9/14 | 98.00% | $0.78 | 127 | 19.7 sec |
| Contract clauses · Spanish | 4/14 | 100.00% | $0.60 | 0 | 14.2 sec |
| Contract clauses · French | 9/14 | 99.49% | $0.66 | 67 | 16.2 sec |
| Email triage · English (US) | 9/14 | 97.48% | $0.30 | 213 | 8.9 sec |
| Email triage · Czech | 5/14 | 98.52% | $0.45 | 133 | 13.0 sec |
| Email triage · German | 6/14 | 97.85% | $0.39 | 187 | 12.1 sec |
| Email triage · Spanish | 10/14 | 96.74% | $0.46 | 280 | 11.2 sec |
| Email triage · French | 10/14 | 96.44% | $0.45 | 240 | 13.2 sec |
| Invoice extraction · English (US) | 14/14 | 82.36% | $1.15 | 427 | 17.4 sec |
| Invoice extraction · Czech | 14/14 | 91.79% | $1.02 | 233 | 17.0 sec |
| Invoice extraction · German | 14/14 | 93.49% | $1.12 | 127 | 19.0 sec |
| Invoice extraction · Spanish | 14/14 | 88.87% | $1.38 | 253 | 22.0 sec |
| Invoice extraction · French | 13/14 | 90.46% | $1.28 | 167 | 19.4 sec |
| Personal data detection · English (US) | 14/14 | 99.24% | $0.28 | 13 | 7.5 sec |
| Personal data detection · Czech | 14/14 | 96.95% | $0.41 | 113 | 9.6 sec |
| Personal data detection · German | 9/14 | 99.62% | $0.36 | 7 | 12.2 sec |
| Personal data detection · Spanish | 5/14 | 99.81% | $0.38 | 13 | 10.2 sec |
| Personal data detection · French | 6/14 | 99.71% | $0.40 | 20 | 10.9 sec |
| Questions about a document · English (US) | 11/14 | 99.78% | $0.44 | 13 | 8.7 sec |
| Questions about a document · Czech | 13/14 | 94.22% | $0.71 | 153 | 14.3 sec |
| Questions about a document · German | 11/14 | 98.89% | $0.46 | 33 | 8.6 sec |
| Questions about a document · Spanish | 13/14 | 99.67% | $0.53 | 13 | 10.3 sec |
| Questions about a document · French | 13/14 | 98.78% | $0.57 | 27 | 10.4 sec |
| Document summaries · English (US) | 9/14 | 99.89% | $0.75 | 7 | 17.8 sec |
| Document summaries · Czech | 12/14 | 95.44% | $1.48 | 67 | 34.2 sec |
| Document summaries · German | 14/14 | 95.33% | $1.43 | 47 | 29.5 sec |
| Document summaries · Spanish | 11/14 | 94.44% | $1.22 | 67 | 23.9 sec |
| Document summaries · French | 10/14 | 96.33% | $1.46 | 53 | 28.1 sec |
Compare with another model
- Qwen3.8 Flash vs Claude Haiku 5.5
- Qwen3.8 Flash vs Claude Sonnet 5
- Qwen3.8 Flash vs Claude Sonnet 5.5
- Qwen3.8 Flash vs DeepSeek V4.1 Flash
- Qwen3.8 Flash vs Gemini 3.5 Flash Lite
- Qwen3.8 Flash vs Gemini 3.8 Flash
- Qwen3.8 Flash vs Gemma 4 31B
- Qwen3.8 Flash vs GLM-5.3 Flash
- Qwen3.8 Flash vs GPT-6 Luna
- Qwen3.8 Flash vs GPT-6 Sol
- Qwen3.8 Flash vs Mistral Large 4
- Qwen3.8 Flash vs Mistral Medium 3.5
- Qwen3.8 Flash vs Mistral Small 4
Where it goes wrong
The fields it gets wrong most often and the error types, across all documents in the benchmark.
Contract clauses · English (US)
Weakest fields
Provider69.33%
Errors in this field: 17 of 40 example documents. For example:
- Santiago-Harris Group Co. · Columbus Printing Inc.
- Model's answer
- Santiago-Harris Group Co., a limited liability company
- Correct value
- Santiago-Harris Group Co.
- Pittsburgh Construction Co. · Evans Electric LLC
- Model's answer
- Pittsburgh Construction Co., a limited liability company
- Correct value
- Pittsburgh Construction Co.
- Frye & Sons, Co. · Nunez-Smith Group Inc.
- Model's answer
- Frye & Sons, Co., a limited liability company
- Correct value
- Frye & Sons, Co.
- Kim Electric Co. · Ramos & Sons, Co.
- Model's answer
- Kim Electric Co., a limited liability company
- Correct value
- Kim Electric Co.
- Blue Ridge Services Corp. · Bishop-Kim Group LLC
- Model's answer
- Blue Ridge Services Corp., a Delaware corporation
- Correct value
- Blue Ridge Services Corp.
- Santiago-Harris Group Co. · Columbus Printing Inc.
Client69.33%
Errors in this field: 17 of 40 example documents. For example:
- Santiago-Harris Group Co. · Columbus Printing Inc.
- Model's answer
- Columbus Printing Inc., a Delaware corporation
- Correct value
- Columbus Printing Inc.
- Pittsburgh Construction Co. · Evans Electric LLC
- Model's answer
- Evans Electric LLC, a limited liability company
- Correct value
- Evans Electric LLC
- Frye & Sons, Co. · Nunez-Smith Group Inc.
- Model's answer
- Nunez-Smith Group Inc., a Delaware corporation
- Correct value
- Nunez-Smith Group Inc.
- Kim Electric Co. · Ramos & Sons, Co.
- Model's answer
- Ramos & Sons, Co., a limited liability company
- Correct value
- Ramos & Sons, Co.
- Blue Ridge Services Corp. · Bishop-Kim Group LLC
- Model's answer
- Bishop-Kim Group LLC, a limited liability company
- Correct value
- Bishop-Kim Group LLC
- Santiago-Harris Group Co. · Columbus Printing Inc.
Effective date99.33%
Errors in this field: 1 of 40 example documents. For example:
- Page Printing LLC · Raleigh Printing Inc.
- Model's answer
- —
- Correct value
- 2025-12-12
- Page Printing LLC · Raleigh Printing Inc.
Late-payment penalty (% per day)99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value92
- missing field3
What the error types mean
- wrong value
- The model gave a value other than the correct one.
- missing field
- The value is in the document, but the model left the field empty.
Contract clauses · Czech
Weakest fields
Contract type98.67%
Errors in this field: 1 of 40 example documents. For example:
- Interservis s.r.o. · Staněk s.r.o.
- Model's answer
- services
- Correct value
- maintenance
- Interservis s.r.o. · Staněk s.r.o.
Error types (number of fields)
- wrong value2
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Contract clauses · German
Weakest fields
Governing law91.33%
Errors in this field: 7 of 40 example documents. For example:
- Hahn & Schulz GmbH & Co. KG · Alpentec GmbH
- Model's answer
- —
- Correct value
- DE
- Hansawerk GmbH · Schreinerei Weber AG
- Model's answer
- —
- Correct value
- DE
- Maintec GmbH · Schreinerei Wagner GmbH
- Model's answer
- —
- Correct value
- DE
- Kraus Consulting GmbH · Lange Consulting GmbH & Co. KG
- Model's answer
- —
- Correct value
- DE
- Schulz Systemtechnik GmbH · Elbdata GmbH
- Model's answer
- —
- Correct value
- DE
- Hahn & Schulz GmbH & Co. KG · Alpentec GmbH
Provider ID94.67%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 2 of 40 example documents. For example:
- Wesermetall e.K. · Weserform GmbH
- Model's answer
- DE225841349
- Correct value
- — (not in the document, the field should stay empty)
- Hansawerk GmbH · Schreinerei Weber AG
- Model's answer
- DE810257236
- Correct value
- — (not in the document, the field should stay empty)
- Wesermetall e.K. · Weserform GmbH
Client ID94.67%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 2 of 40 example documents. For example:
- Wesermetall e.K. · Weserform GmbH
- Model's answer
- DE441054820
- Correct value
- — (not in the document, the field should stay empty)
- Hansawerk GmbH · Schreinerei Weber AG
- Model's answer
- DE687960979
- Correct value
- — (not in the document, the field should stay empty)
- Wesermetall e.K. · Weserform GmbH
Provider99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- extra value14
- invalid JSON13
- missing field12
What the error types mean
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
Contract clauses · Spanish
No errors on this task.
Contract clauses · French
Weakest fields
Billing period94.00%
Errors in this field: 2 of 40 example documents. For example:
- Établissements Martin SAS · Lambert et Associés SAS
- Model's answer
- —
- Correct value
- one_time
- Établissements Joly SARL · Occidata SASU
- Model's answer
- —
- Correct value
- one_time
- Établissements Martin SAS · Lambert et Associés SAS
Late-payment penalty (% per day)99.33%
Errors in this field: 1 of 40 example documents. For example:
- Lefèvre Conseil SARL · SAS Alpidata
- Model's answer
- 0.01
- Correct value
- 0.50
- Lefèvre Conseil SARL · SAS Alpidata
Error types (number of fields)
- missing field9
- number format1
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Email triage · English (US)
Weakest fields
Sentiment80.67%
Errors in this field: 6 of 40 example documents. For example:
- Jane Rubio
- Model's answer
- positive
- Correct value
- neutral
- Laurie Knight
- Model's answer
- positive
- Correct value
- neutral
- Tyler Brown
- Model's answer
- negative
- Correct value
- neutral
- Daniel Stewart
- Model's answer
- negative
- Correct value
- neutral
- Allison Floyd
- Model's answer
- negative
- Correct value
- neutral
- Jane Rubio
Priority97.33%
Errors in this field: 2 of 40 example documents. For example:
- Angela Harmon
- Model's answer
- urgent
- Correct value
- normal
- Joseph Jackson
- Model's answer
- urgent
- Correct value
- normal
- Angela Harmon
Deadline99.33%
Errors in this field: 1 of 40 example documents. For example:
- Angela Harmon
- Model's answer
- 2025-09-16
- Correct value
- — (not in the document, the field should stay empty)
- Angela Harmon
Error types (number of fields)
- wrong value33
- extra value1
What the error types mean
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Email triage · Czech
Weakest fields
Sentiment88.67%
Errors in this field: 3 of 40 example documents. For example:
- Ivo Dvořák
- Model's answer
- negative
- Correct value
- neutral
- Milan Matějka
- Model's answer
- positive
- Correct value
- neutral
- Růžena Sedláková
- Model's answer
- positive
- Correct value
- neutral
- Ivo Dvořák
Priority99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Customer name99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Order number99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value20
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · German
Weakest fields
Sentiment84.67%
Errors in this field: 6 of 40 example documents. For example:
- Rolf-Peter Metz
- Model's answer
- negative
- Correct value
- neutral
- Alexej Sauer
- Model's answer
- negative
- Correct value
- positive
- Dogan Wernecke
- Model's answer
- negative
- Correct value
- neutral
- Linda Bähr
- Model's answer
- negative
- Correct value
- neutral
- Nadja Wiek
- Model's answer
- negative
- Correct value
- neutral
- Rolf-Peter Metz
Priority96.00%
Errors in this field: 1 of 40 example documents. For example:
- Friedbert Hoffmann
- Model's answer
- urgent
- Correct value
- normal
- Friedbert Hoffmann
Error types (number of fields)
- wrong value29
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · Spanish
Weakest fields
Priority84.00%
Errors in this field: 11 of 40 example documents. For example:
- Lorenzo Giménez
- Model's answer
- urgent
- Correct value
- high
- Anna Barba
- Model's answer
- high
- Correct value
- normal
- Lucas Santiago
- Model's answer
- urgent
- Correct value
- high
- Serafina Catalán
- Model's answer
- urgent
- Correct value
- high
- Reyna Pardo
- Model's answer
- high
- Correct value
- normal
- Lorenzo Giménez
Sentiment86.67%
Errors in this field: 4 of 40 example documents. For example:
- Candelas Serra
- Model's answer
- negative
- Correct value
- neutral
- Albert Blanco
- Model's answer
- negative
- Correct value
- neutral
- Rosalva Talavera
- Model's answer
- negative
- Correct value
- neutral
- Priscila Goñi
- Model's answer
- negative
- Correct value
- neutral
- Candelas Serra
Error types (number of fields)
- wrong value44
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · French
Weakest fields
Priority83.33%
Errors in this field: 12 of 40 example documents. For example:
- Tristan Bonneau
- Model's answer
- urgent
- Correct value
- high
- Sabine Breton
- Model's answer
- normal
- Correct value
- high
- Amélie Turpin
- Model's answer
- urgent
- Correct value
- high
- Capucine Maréchal
- Model's answer
- high
- Correct value
- normal
- Michelle Riou
- Model's answer
- urgent
- Correct value
- high
- Tristan Bonneau
Sentiment90.00%
Errors in this field: 2 of 40 example documents. For example:
- Zoé Da Costa
- Model's answer
- negative
- Correct value
- neutral
- Élisabeth Gimenez
- Model's answer
- positive
- Correct value
- neutral
- Zoé Da Costa
Category98.67%
Errors in this field: 1 of 40 example documents. For example:
- Suzanne Marchal
- Model's answer
- complaint
- Correct value
- delivery
- Suzanne Marchal
Customer name99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value39
- invalid JSON9
What the error types mean
- wrong value
- The model gave a value other than the correct one.
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Invoice extraction · English (US)
Weakest fields
Net amount by tax rate70.00%
Errors in this field: 16 of 40 example documents. For example:
- Reid Office Products LLC · A-003849
- Model's answer
- —
- Correct value
- 0 %: 3700.94
- Miranda-Vega Group LLC · 4959
- Model's answer
- : %: 564.90
- Correct value
- 7.25 %: 564.90
- Derek Richardson · INV-16398
- Model's answer
- : 1333.33 %: 1333.33
- Correct value
- — (not in the document, the field should stay empty)
- Summit Services Inc. · INV-15230
- Model's answer
- —
- Correct value
- 0 %: 1191.25
- Raleigh Engineering Inc. · INV253511
- Model's answer
- —
- Correct value
- 7.25 %: 401.30
- Reid Office Products LLC · A-003849
Tax amount by tax rate76.00%
Errors in this field: 11 of 40 example documents. For example:
- Reid Office Products LLC · A-003849
- Model's answer
- —
- Correct value
- 0 %: 0.00
- Miranda-Vega Group LLC · 4959
- Model's answer
- : %: 40.96
- Correct value
- 7.25 %: 40.96
- Raleigh Engineering Inc. · INV253511
- Model's answer
- —
- Correct value
- 7.25 %: 29.09
- Liberty Systems Inc. · INV-17299
- Model's answer
- : %: 1763.15
- Correct value
- 10.35 %: 1763.15
- Hughes Janitorial Services Corp. · A-001911
- Model's answer
- —
- Correct value
- 10.35 %: 991.68
- Reid Office Products LLC · A-003849
Payment reference79.33%
Errors in this field: 6 of 40 example documents. For example:
- Reid Office Products LLC · A-003849
- Model's answer
- —
- Correct value
- A-003849
- Miranda-Vega Group LLC · 4959
- Model's answer
- PO-24910
- Correct value
- — (not in the document, the field should stay empty)
- Summit Services Inc. · INV-15230
- Model's answer
- —
- Correct value
- INV-15230
- Hughes Janitorial Services Corp. · A-001911
- Model's answer
- —
- Correct value
- A-001911
- Eagle Partners Co. · 4272
- Model's answer
- —
- Correct value
- 4272
- Reid Office Products LLC · A-003849
Bank account81.33%
Errors in this field: 6 of 40 example documents. For example:
- Reid Office Products LLC · A-003849
- Model's answer
- —
- Correct value
- 021000089 8907060439
- Summit Services Inc. · INV-15230
- Model's answer
- —
- Correct value
- 121000248 7164127789
- Raleigh Engineering Inc. · INV253511
- Model's answer
- —
- Correct value
- 021000021 434184039
- Raleigh Logistics LLC · 2025-1010
- Model's answer
- 09100022 4452711716
- Correct value
- 091000022 4452711716
- Eagle Partners Co. · 4272
- Model's answer
- —
- Correct value
- 021000089 16652093
- Reid Office Products LLC · A-003849
Error types (number of fields)
- invalid JSON154
- missing field83
- wrong rate23
- wrong value18
- extra value13
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
- wrong rate
- Net amount or tax assigned to the wrong VAT rate.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Invoice extraction · Czech
Weakest fields
Net amount by tax rate80.00%
Errors in this field: 9 of 40 example documents. For example:
- Sedláček s.r.o. · 2025/0774
- Model's answer
- —
- Correct value
- 21 %: 1553.45
- Dvořák s.r.o. · FV2501487
- Model's answer
- 21 %: 2913.87, : 2913.87, %: 2913.87
- Correct value
- 21 %: 2913.87
- Hruška spol. s r.o. · 20250328
- Model's answer
- : %: 91280.74
- Correct value
- 21 %: 91280.74
- Sedlák s.r.o. · 20250158
- Model's answer
- 12 %: 5771.52, 21 %: 18450.00
- Correct value
- 12 %: 5771.52, 21 %: 18546.00
- Stavmet s.r.o. · 20250906
- Model's answer
- : %: 65857.00
- Correct value
- 21 %: 65857.00
- Sedláček s.r.o. · 2025/0774
Tax amount by tax rate80.67%
Errors in this field: 8 of 40 example documents. For example:
- Sedláček s.r.o. · 2025/0774
- Model's answer
- —
- Correct value
- 21 %: 326.22
- Dvořák s.r.o. · FV2501487
- Model's answer
- 21 %: 611.91, : 611.91, %: 611.91
- Correct value
- 21 %: 611.91
- Hruška spol. s r.o. · 20250328
- Model's answer
- : %: 19168.96
- Correct value
- 21 %: 19168.96
- Sedlák s.r.o. · 20250158
- Model's answer
- 12 %: 692.58, 21 %: 3874.50
- Correct value
- 12 %: 692.58, 21 %: 3894.66
- Stavmet s.r.o. · 20250906
- Model's answer
- : %: 13829.97
- Correct value
- 21 %: 13829.97
- Sedláček s.r.o. · 2025/0774
Bank account90.67%
Errors in this field: 4 of 40 example documents. For example:
- Sedláček s.r.o. · 2025/0774
- Model's answer
- —
- Correct value
- CZ6520100000000078563487
- Moravská energetická spol. s r.o. · 25101373
- Model's answer
- CZ680100000008698507992
- Correct value
- CZ6801000000008698507992
- Adam Vítek · 2025/1045
- Model's answer
- 392286–346812831/2010
- Correct value
- 392286-346812831/2010
- Hrubý s.r.o. · DB2500960
- Model's answer
- CZ280600000000512736564
- Correct value
- CZ2806000000000512736564
- Sedláček s.r.o. · 2025/0774
Supplier93.33%
Errors in this field: 2 of 40 example documents. For example:
- Sedláček s.r.o. · 2025/0774
- Model's answer
- —
- Correct value
- Sedláček s.r.o.
- Sedlák s.r.o. · 20250158
- Model's answer
- Sedlá k s.r.o.
- Correct value
- Sedlák s.r.o.
- Sedláček s.r.o. · 2025/0774
Error types (number of fields)
- invalid JSON65
- missing field41
- wrong value36
- wrong rate16
- extra value1
- net instead of total1
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
- wrong value
- The model gave a value other than the correct one.
- wrong rate
- Net amount or tax assigned to the wrong VAT rate.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- net instead of total
- The amount including VAT instead of excluding it, or vice versa.
Invoice extraction · German
Weakest fields
Payment reference88.67%
Errors in this field: 5 of 40 example documents. For example:
- Hoffmann UG (haftungsbeschränkt) · 2025/1479
- Model's answer
- KD-73584
- Correct value
- — (not in the document, the field should stay empty)
- Dachdeckerei Fischer GmbH · 2025/0014
- Model's answer
- 4528059615
- Correct value
- — (not in the document, the field should stay empty)
- Bauunternehmung Schröder GmbH · 2026-0622
- Model's answer
- —
- Correct value
- 2026-0622
- Braun IT-Service GmbH · RE2500825
- Model's answer
- KD-85750
- Correct value
- — (not in the document, the field should stay empty)
- Schreinerei Ferdi Schuster · 2025/0939
- Model's answer
- —
- Correct value
- 2025/0939
- Hoffmann UG (haftungsbeschränkt) · 2025/1479
Invoice number92.67%
Errors in this field: 2 of 40 example documents. For example:
- Bauunternehmung Schröder GmbH · 2026-0622
- Model's answer
- —
- Correct value
- 2026-0622
- Schreinerei Ferdi Schuster · 2025/0939
- Model's answer
- —
- Correct value
- 2025/0939
- Bauunternehmung Schröder GmbH · 2026-0622
Supplier94.00%
Errors in this field: 2 of 40 example documents. For example:
- Bauunternehmung Schröder GmbH · 2026-0622
- Model's answer
- —
- Correct value
- Bauunternehmung Schröder GmbH
- Schreinerei Ferdi Schuster · 2025/0939
- Model's answer
- —
- Correct value
- Schreinerei Ferdi Schuster
- Bauunternehmung Schröder GmbH · 2026-0622
Supplier ID (EIN, IČO…)94.00%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 2 of 40 example documents. For example:
- Bauunternehmung Schröder GmbH · 2026-0622
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Schreinerei Ferdi Schuster · 2025/0939
- Model's answer
- —
- Correct value
- 666/427/95126
- Bauunternehmung Schröder GmbH · 2026-0622
Error types (number of fields)
- invalid JSON117
- extra value8
- wrong value2
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
Invoice extraction · Spanish
Weakest fields
Payment reference85.33%
Errors in this field: 6 of 40 example documents. For example:
- Talleres Ibérica, S.A. · PF25/1188
- Model's answer
- —
- Correct value
- PF25/1188
- Leyre Lladó Vall · F2025-1476
- Model's answer
- 4548118187
- Correct value
- — (not in the document, the field should stay empty)
- Centro de Formación Mediterránea S.L. · 2026/0876
- Model's answer
- —
- Correct value
- 2026/0876
- Aguirre Formación S.L. · 2025/0652
- Model's answer
- —
- Correct value
- 2025/0652
- Coloma Servicios Informáticos S.L.U. · PRO-2025-0066
- Model's answer
- —
- Correct value
- CLI-17661
- Talleres Ibérica, S.A. · PF25/1188
Net amount by tax rate86.00%
Errors in this field: 5 of 40 example documents. For example:
- Centro de Formación Mediterránea S.L. · 2026/0876
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Aguirre Formación S.L. · 2025/0652
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Coloma Servicios Informáticos S.L.U. · PRO-2025-0066
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Escobar Comunicación S.L. · RECT-2025-0924
- Model's answer
- —
- Correct value
- 0 %: -835.36
- Centro de Formación Centro S.L.U. · A-2025-1405
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Centro de Formación Mediterránea S.L. · 2026/0876
VAT ID87.33%
What we ask for: The supplier's VAT ID (DIČ, USt-IdNr., VAT number). Not the registration number.
Errors in this field: 6 of 40 example documents. For example:
- Talleres Ibérica, S.A. · PF25/1188
- Model's answer
- —
- Correct value
- ESA38261673
- Panadería Estrada Menendez, S.L. · 2025/1392
- Model's answer
- ESB52463130
- Correct value
- — (not in the document, the field should stay empty)
- Centro de Formación Mediterránea S.L. · 2026/0876
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Aguirre Formación S.L. · 2025/0652
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Coloma Servicios Informáticos S.L.U. · PRO-2025-0066
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Talleres Ibérica, S.A. · PF25/1188
Supplier88.00%
Errors in this field: 5 of 40 example documents. For example:
- Talleres Ibérica, S.A. · PF25/1188
- Model's answer
- —
- Correct value
- Talleres Ibérica, S.A.
- Centro de Formación Mediterránea S.L. · 2026/0876
- Model's answer
- —
- Correct value
- Centro de Formación Mediterránea S.L.
- Aguirre Formación S.L. · 2025/0652
- Model's answer
- —
- Correct value
- Aguirre Formación S.L.
- Coloma Servicios Informáticos S.L.U. · PRO-2025-0066
- Model's answer
- —
- Correct value
- Coloma Servicios Informáticos S.L.U.
- Centro de Formación Centro S.L.U. · A-2025-1405
- Model's answer
- —
- Correct value
- Centro de Formación Centro S.L.U.
- Talleres Ibérica, S.A. · PF25/1188
Error types (number of fields)
- invalid JSON182
- missing field17
- extra value15
- wrong value2
- lost diacritics1
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
- lost diacritics
- The value is right but missing diacritics (accent marks).
Invoice extraction · French
Weakest fields
Payment reference86.67%
Errors in this field: 6 of 40 example documents. For example:
- Établissements Blanc SAS · F-11122
- Model's answer
- —
- Correct value
- F-11122
- Nettoyage Mercier SASU · AC-10176
- Model's answer
- —
- Correct value
- AC-10176
- Établissements Bertrand SASU · F2025-0066
- Model's answer
- BC-2025-7576
- Correct value
- — (not in the document, the field should stay empty)
- Guérin & Fils EURL · 2026-05-688
- Model's answer
- —
- Correct value
- CL10688
- SA Établissements Gauthier · F2025-1451
- Model's answer
- F2025-1451
- Correct value
- — (not in the document, the field should stay empty)
- Établissements Blanc SAS · F-11122
Due date89.33%
Errors in this field: 4 of 40 example documents. For example:
- Établissements Blanc SAS · F-11122
- Model's answer
- —
- Correct value
- 2025-04-22
- Nettoyage Mercier SASU · AC-10176
- Model's answer
- —
- Correct value
- 2025-04-30
- Guérin & Fils EURL · 2026-05-688
- Model's answer
- —
- Correct value
- 2026-07-12
- Lefèvre & Fils SARL · FAC-2026-00198
- Model's answer
- —
- Correct value
- 2026-08-09
- Établissements Blanc SAS · F-11122
Bank account89.33%
Errors in this field: 3 of 40 example documents. For example:
- Établissements Blanc SAS · F-11122
- Model's answer
- —
- Correct value
- FR7630003035500003114132087
- Nettoyage Mercier SASU · AC-10176
- Model's answer
- —
- Correct value
- FR7618206000210006616417072
- Guérin & Fils EURL · 2026-05-688
- Model's answer
- —
- Correct value
- FR7610278013953991976943070
- Établissements Blanc SAS · F-11122
Supplier90.00%
Errors in this field: 3 of 40 example documents. For example:
- Établissements Blanc SAS · F-11122
- Model's answer
- —
- Correct value
- Établissements Blanc SAS
- Nettoyage Mercier SASU · AC-10176
- Model's answer
- —
- Correct value
- Nettoyage Mercier SASU
- Guérin & Fils EURL · 2026-05-688
- Model's answer
- —
- Correct value
- Guérin & Fils EURL
- Établissements Blanc SAS · F-11122
Error types (number of fields)
- invalid JSON104
- missing field71
- extra value6
- wrong value5
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
Personal data detection · English (US)
Weakest fields
Addresses98.67%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Names99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Email addresses99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Phone numbers99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- extra value5
- wrong value2
- missing field1
What the error types mean
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · Czech
Weakest fields
Names90.67%
Errors in this field: 1 of 40 example documents. For example:
- Naděžda Žáková
- Model's answer
- Naděža Žáková; Libuše Bártová; Viktorie Vávrová
- Correct value
- Naděžda Žáková; Libuše Bártová; Viktorie Vávrová
- Naděžda Žáková
Addresses96.67%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Phone numbers97.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Email addresses98.00%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value13
- invalid JSON7
- extra value6
- missing field6
What the error types mean
- wrong value
- The model gave a value other than the correct one.
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · German
Weakest fields
Names99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Email addresses99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Phone numbers99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Addresses99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- missing field4
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · Spanish
Weakest fields
Addresses98.67%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- missing field2
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · French
Weakest fields
Names98.67%
Errors in this field: 1 of 40 example documents. For example:
- Édouard Leroy
- Model's answer
- Jean; Édouard Leroy
- Correct value
- Édouard Leroy
- Édouard Leroy
Addresses99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- extra value2
- missing field1
What the error types mean
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- missing field
- The value is in the document, but the model left the field empty.
Questions about a document · English (US)
Weakest fields
Question 499.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Question 599.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- number format2
What the error types mean
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Questions about a document · Czech
Weakest fields
Question 292.67%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- helpdesk@maresova-zak.example
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: S jakým předstihem musí poskytovatel oznámit …
- Model's answer
- 0.15
- Correct value
- 15
- SLA: Jaká je reakční doba u požadavků …
Question 692.67%
Errors in this field: 6 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 0.998
- Correct value
- 99.8
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- —
- Correct value
- 2025-09-13
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 0.05
- Correct value
- 5
- SLA: Na jakou slevu z měsíčního paušálu …
- Model's answer
- 0.999
- Correct value
- 99.9
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: Jaká je reakční doba u požadavků …
Question 594.00%
Errors in this field: 2 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 0.20
- Correct value
- 20
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- —
- Correct value
- 1
- SLA: Jaká je reakční doba u požadavků …
Question 494.67%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 6
- Correct value
- 2025-02-25
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- —
- Correct value
- 99.9
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: Jaká je reakční doba u požadavků …
Error types (number of fields)
- number format27
- invalid JSON12
- wrong value8
- extra value4
- missing field1
What the error types mean
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- missing field
- The value is in the document, but the model left the field empty.
Questions about a document · German
Weakest fields
Question 597.33%
Errors in this field: 1 of 40 example documents. For example:
- SLA: Wie lang ist die Reaktionszeit bei …
- Model's answer
- 0.20
- Correct value
- 20
- SLA: Wie lang ist die Reaktionszeit bei …
Question 498.00%
Errors in this field: 1 of 40 example documents. For example:
- SLA: Wie hoch ist die monatliche Pauschale …
- Model's answer
- 0.9995
- Correct value
- 99.95
- SLA: Wie hoch ist die monatliche Pauschale …
Question 698.67%
Errors in this field: 1 of 40 example documents. For example:
- SLA: Wie lang ist die Reaktionszeit bei …
- Model's answer
- 0.998
- Correct value
- 99.8
- SLA: Wie lang ist die Reaktionszeit bei …
Question 399.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- number format6
- missing field4
What the error types mean
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
- missing field
- The value is in the document, but the model left the field empty.
Questions about a document · Spanish
Weakest fields
Question 399.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Question 499.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Question 599.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- number format2
- wrong value1
What the error types mean
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
- wrong value
- The model gave a value other than the correct one.
Questions about a document · French
Weakest fields
Question 597.33%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- 8
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- 2
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- —
- Correct value
- 8
- SLA: Quel avoir sur la redevance mensuelle …
Question 698.00%
Errors in this field: 2 of 40 example documents. For example:
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- 30
- Correct value
- 8
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- 5
- SLA: Quel avoir sur la redevance mensuelle …
Question 199.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Question 299.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- invalid JSON6
- missing field3
- extra value1
- wrong value1
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- missing field
- The value is in the document, but the model left the field empty.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
Document summaries · English (US)
Weakest fields
Length (max words)99.33%
What we ask for: The summary must not exceed the word limit given in the prompt.
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value1
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Document summaries · Czech
Weakest fields
Rejected proposals (must not appear)93.33%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 5 of 40 example documents. For example:
- Rostislav Novák, Miloš Jelínek
- Appeared in the output
- 1810000
- Must not appear
- 1810000
- Miluše Vaňková, Lada Lišková
- Appeared in the output
- 540000
- Must not appear
- 540000
- Blanka Krausová
- Appeared in the output
- —
- Must not appear
- 250000
- Magdalena Sedláčková
- Appeared in the output
- —
- Must not appear
- 210000
- Milan Matějka, Jana Dostálová, Zora Fialová
- Appeared in the output
- —
- Must not appear
- 1430000
- Rostislav Novák, Miloš Jelínek
Superseded deadlines (must not appear)95.33%
What we ask for: Deadlines the document later reschedules. The summary should give only the new deadline.
Errors in this field: 3 of 40 example documents. For example:
- Blanka Krausová
- Appeared in the output
- —
- Must not appear
- 2025-05-26
- Magdalena Sedláčková
- Appeared in the output
- —
- Must not appear
- 2025-10-13
- Milan Matějka, Jana Dostálová, Zora Fialová
- Appeared in the output
- —
- Must not appear
- 2026-09-11
- Blanka Krausová
Amounts96.00%
Errors in this field: 3 of 40 example documents. For example:
- Blanka Krausová
- Model's answer
- —
- Correct value
- 200000; 85000; 1660000
- Magdalena Sedláčková
- Model's answer
- —
- Correct value
- 130000; 1410000; 65000
- Milan Matějka, Jana Dostálová, Zora Fialová
- Model's answer
- —
- Correct value
- 1070000; 480000; 70000
- Blanka Krausová
Deadlines96.00%
Errors in this field: 3 of 40 example documents. For example:
- Blanka Krausová
- Model's answer
- —
- Correct value
- 2025-06-11
- Magdalena Sedláčková
- Model's answer
- —
- Correct value
- 2025-11-12; 2025-09-27
- Milan Matějka, Jana Dostálová, Zora Fialová
- Model's answer
- —
- Correct value
- 2026-10-10; 2026-09-23
- Blanka Krausová
Error types (number of fields)
- invalid JSON36
- extra value5
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Document summaries · German
Weakest fields
Amounts95.33%
Errors in this field: 1 of 40 example documents. For example:
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
- Model's answer
- —
- Correct value
- 50000; 12000; 81000
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
Deadlines95.33%
Errors in this field: 1 of 40 example documents. For example:
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
- Model's answer
- —
- Correct value
- 2025-09-25; 2025-08-16
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
Responsible people95.33%
Errors in this field: 1 of 40 example documents. For example:
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
- Model's answer
- —
- Correct value
- Helen Noack; Pietro Birnbaum; Mehmet Lorch
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
Rejected proposals (must not appear)95.33%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 1 of 40 example documents. For example:
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
- Appeared in the output
- —
- Must not appear
- 72000
- Helen Noack, Pietro Birnbaum, Mehmet Lorch
Error types (number of fields)
- invalid JSON42
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Document summaries · Spanish
Weakest fields
Length (max words)93.33%
What we ask for: The summary must not exceed the word limit given in the prompt.
Errors in this field: 2 of 40 example documents. For example:
- Santiago Manzano, Crescencia Jover
- Words / limit
- —
- Word limit
- 80
- Julieta Acuña, Paloma Chaparro, Ana Belén Parejo
- Words / limit
- —
- Word limit
- 80
- Santiago Manzano, Crescencia Jover
Amounts94.67%
Errors in this field: 2 of 40 example documents. For example:
- Santiago Manzano, Crescencia Jover
- Model's answer
- —
- Correct value
- 16000; 85000; 31000
- Julieta Acuña, Paloma Chaparro, Ana Belén Parejo
- Model's answer
- —
- Correct value
- 53000; 24000; 3500
- Santiago Manzano, Crescencia Jover
Deadlines94.67%
Errors in this field: 2 of 40 example documents. For example:
- Santiago Manzano, Crescencia Jover
- Model's answer
- —
- Correct value
- 2026-03-01
- Julieta Acuña, Paloma Chaparro, Ana Belén Parejo
- Model's answer
- —
- Correct value
- 2026-10-10; 2026-09-23
- Santiago Manzano, Crescencia Jover
Responsible people94.67%
Errors in this field: 2 of 40 example documents. For example:
- Santiago Manzano, Crescencia Jover
- Model's answer
- —
- Correct value
- Santiago Manzano; Crescencia Jover
- Julieta Acuña, Paloma Chaparro, Ana Belén Parejo
- Model's answer
- —
- Correct value
- Julieta Acuña; Paloma Chaparro; Ana Belén Parejo
- Santiago Manzano, Crescencia Jover
Error types (number of fields)
- invalid JSON48
- wrong value2
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
Document summaries · French
Weakest fields
Length (max words)95.33%
What we ask for: The summary must not exceed the word limit given in the prompt.
Errors in this field: 2 of 40 example documents. For example:
- Capucine Maréchal, Victor Perez
- Words / limit
- —
- Word limit
- 80
- Suzanne Marchal
- Words / limit
- 84 / 80
- Word limit
- 80
- Capucine Maréchal, Victor Perez
Superseded deadlines (must not appear)96.00%
What we ask for: Deadlines the document later reschedules. The summary should give only the new deadline.
Errors in this field: 1 of 40 example documents. For example:
- Capucine Maréchal, Victor Perez
- Appeared in the output
- —
- Must not appear
- 2026-02-28
- Capucine Maréchal, Victor Perez
Amounts96.67%
Errors in this field: 1 of 40 example documents. For example:
- Capucine Maréchal, Victor Perez
- Model's answer
- —
- Correct value
- 6500; 70000; 83000
- Capucine Maréchal, Victor Perez
Deadlines96.67%
Errors in this field: 1 of 40 example documents. For example:
- Capucine Maréchal, Victor Perez
- Model's answer
- —
- Correct value
- 2026-03-24
- Capucine Maréchal, Victor Perez
Error types (number of fields)
- invalid JSON30
- wrong value2
- extra value1
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Free with login
Exactly where the model goes wrong: its weakest fields and the actual wrong answers.
Just your email, no password.
Language tax
How many more tokens this model needs for the same text compared with English.
- Czech 1.66×
- German 1.28×
- French 1.37×
- Spanish 1.32×
- Polish 1.59×
Compliance
- In the EU no
- Zero retention (ZDR) no
- DPA yes
- Open weights no
Only provider on OpenRouter: Alibaba (no EU or ZDR endpoint). DPA: general Alibaba Cloud International DPA.
Use it now
The exact request the benchmark sends (same prompt, JSON schema, temperature 0), ready to call Qwen3.8 Flash via OpenRouter.
Contract clauses · English (US)
import json
import os
import httpx
PROMPT = "Extract the key terms of the contract below and return only JSON with these keys:\n\nprovider_name the party that delivers (provider, seller, landlord, licensor, contractor),\n exactly as written, including the legal form\nprovider_id that party's company ID (EIN), if stated\nclient_name the party that pays (client, buyer, tenant, licensee, customer)\nclient_id that party's company ID (EIN), if stated\ncontract_type one of: services, supply, lease, license, maintenance\nprice the agreed price excluding VAT / sales tax (number)\ncurrency ISO 4217 code (USD, EUR, CZK, …)\nprice_period one of: one_time, monthly, yearly\neffective_date the date the contract takes effect (YYYY-MM-DD)\nend_date the date a fixed term ends (YYYY-MM-DD); null if the term is indefinite\nnotice_period_months the termination notice period in months (number); null if there is none\npenalty_pct_per_day the late-payment penalty in percent of the overdue amount per day\n (e.g. 0.05 for 0.05 %); null if there is none\ngoverning_law ISO 3166 country code of the governing law; US states as US-XX (e.g. US-NY)\n\nRules:\n- Resolve relative dates (\"on the date of signature\", \"the first day of the following month\")\n against the signing date.\n- The signing date is not the effective date unless the contract says so.\n- Deposits, advance payments and fixed penalties for other breaches are not the price or the\n late-payment penalty.\n- If a value is not in the contract, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"provider_name\":{\"type\":[\"string\",\"null\"]},\"provider_id\":{\"type\":[\"string\",\"null\"]},\"client_name\":{\"type\":[\"string\",\"null\"]},\"client_id\":{\"type\":[\"string\",\"null\"]},\"contract_type\":{\"type\":[\"string\",\"null\"],\"enum\":[\"services\",\"supply\",\"lease\",\"license\",\"maintenance\",null]},\"price\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"price_period\":{\"type\":[\"string\",\"null\"],\"enum\":[\"one_time\",\"monthly\",\"yearly\",null]},\"effective_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"end_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"notice_period_months\":{\"type\":[\"integer\",\"null\"]},\"penalty_pct_per_day\":{\"type\":[\"number\",\"null\"]},\"governing_law\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"provider_name\",\"provider_id\",\"client_name\",\"client_id\",\"contract_type\",\"price\",\"currency\",\"price_period\",\"effective_date\",\"end_date\",\"notice_period_months\",\"penalty_pct_per_day\",\"governing_law\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "contracts", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Email triage · English (US)
import json
import os
import httpx
PROMPT = "You triage the support inbox. Read the email below and return only JSON with these keys:\n\ncategory one of: billing (invoices, payments, charges), delivery (where is the order, shipping,\n address change), complaint (damaged, wrong or defective goods, warranty claims),\n technical (login, app or system errors, integrations), cancellation (cancel an order,\n contract or subscription), sales (quotes, prices, interest before buying)\npriority one of: urgent, high, normal, low (rules below)\nsentiment one of: positive, neutral, negative (the customer's tone)\ncustomer_name full name of the customer who wrote the request\norder_number the customer's order number, if mentioned\ninvoice_number invoice number, if mentioned\namount the amount of money the request is about (number)\ncurrency ISO 4217 code of that amount (USD, EUR, CZK, …)\ndeadline the date by which the customer wants a resolution or answer (YYYY-MM-DD)\n\nPriority rules:\n- urgent: the customer cannot work right now, or the deadline is at most 1 day after the email date\n- high: the deadline is at most 7 days after the email date, or the customer threatens to cancel\n or escalate (lawyer, ending the business)\n- low: the customer says explicitly that it is not urgent\n- normal: everything else\n\nRules:\n- A thread: judge the newest message; order numbers, invoice numbers and amounts may come from the\n quoted earlier messages. A forwarded email: the customer is the original sender, not the\n colleague who forwarded it.\n- Resolve relative deadlines (\"by Friday\", \"by tomorrow\", \"by the end of the month\") against the\n date of the message that states them.\n- Support ticket numbers, tax IDs and phone numbers are not order or invoice numbers.\n- If a value is not in the email, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"category\":{\"type\":[\"string\",\"null\"],\"enum\":[\"billing\",\"delivery\",\"complaint\",\"technical\",\"cancellation\",\"sales\",null]},\"priority\":{\"type\":[\"string\",\"null\"],\"enum\":[\"urgent\",\"high\",\"normal\",\"low\",null]},\"sentiment\":{\"type\":[\"string\",\"null\"],\"enum\":[\"positive\",\"neutral\",\"negative\",null]},\"customer_name\":{\"type\":[\"string\",\"null\"]},\"order_number\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"amount\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"deadline\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"}},\"required\":[\"category\",\"priority\",\"sentiment\",\"customer_name\",\"order_number\",\"invoice_number\",\"amount\",\"currency\",\"deadline\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "emails", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Invoice extraction · English (US)
import json
import os
import httpx
PROMPT = "Extract the following data from the attached document and return only JSON with these keys:\n\nsupplier_name supplier's name (exactly as printed, including the legal form)\nsupplier_id supplier's tax ID (EIN), if printed\ninvoice_number document number\npayment_reference payment reference to quote with the payment\nissue_date invoice date (YYYY-MM-DD)\ndue_date due date (YYYY-MM-DD)\nnet_by_rate amount per sales tax rate, e.g. {\"8.25\": 1845.00}; non-taxable amounts under \"0\"\nvat_by_rate sales tax per rate, e.g. {\"8.25\": 152.21}\ntotal total due (number, negative for credit notes)\ncurrency ISO 4217 currency code (USD, GBP, EUR, …)\nbank_account supplier's bank details as \"routing-number account-number\", or IBAN\n\nRules:\n- Return the supplier's data only, never the customer's.\n- If a value is not on the document, return null. Do not guess.\n- If the document has no tax breakdown, return net_by_rate and vat_by_rate as null.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"supplier_name\":{\"type\":[\"string\",\"null\"]},\"supplier_id\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"payment_reference\":{\"type\":[\"string\",\"null\"]},\"issue_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"due_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"net_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"vat_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"total\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"bank_account\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"supplier_name\",\"supplier_id\",\"invoice_number\",\"payment_reference\",\"issue_date\",\"due_date\",\"net_by_rate\",\"vat_by_rate\",\"total\",\"currency\",\"bank_account\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "invoices", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Personal data detection · English (US)
import json
import os
import httpx
PROMPT = "Find all personal data of natural persons in the text below (for anonymisation) and return only\nJSON with these keys, each a list of strings (an empty list when there is none):\n\nperson_names full names of people, each person once, as written in the text\nemail_addresses email addresses of people\nphone_numbers phone numbers of people, as written\npostal_addresses home or delivery addresses of people, as written (street, city, ZIP)\nbirth_dates dates of birth (YYYY-MM-DD)\nnational_ids national identification numbers (e.g. SSN, birth numbers)\nbank_accounts bank account numbers of people, as written\n\nRules:\n- Only data of natural persons. Company names, company registration or tax numbers (EIN, IČO,\n VAT), generic company addresses such as info@ or support@, order and ticket numbers are not\n personal data: leave them out.\n- Dates that are not dates of birth (meetings, deliveries, start dates) are not personal data.\n- Do not invent or complete values; copy them from the text.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"person_names\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"email_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"phone_numbers\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"postal_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"birth_dates\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"national_ids\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"bank_accounts\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}}},\"required\":[\"person_names\",\"email_addresses\",\"phone_numbers\",\"postal_addresses\",\"birth_dates\",\"national_ids\",\"bank_accounts\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "pii", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Questions about a document · English (US)
import json
import os
import httpx
PROMPT = "You answer questions about a business document. Below the document there are six numbered\nquestions. Answer each of them using only the text of the document and return only JSON with the\nkeys q1 to q6 (q1 = the answer to question 1, and so on).\n\nRules:\n- Each question states in parentheses the unit or format of the answer. Answer in that unit.\n- Counts (of days, months, hours) and percentages as a number without the unit and without the %\n sign, e.g. \"14\" or \"0.05\".\n- Amounts as a number with a decimal point, no thousands separators and no currency symbol,\n e.g. \"1500.00\".\n- Dates as YYYY-MM-DD.\n- Time ranges in 24-hour format HH:MM-HH:MM, e.g. \"08:00-17:00\".\n- Email addresses exactly as they appear in the document.\n- If the document does not contain the answer to a question, return null. Do not guess and do not\n add information that is not in the document.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"q1\":{\"type\":[\"string\",\"null\"]},\"q2\":{\"type\":[\"string\",\"null\"]},\"q3\":{\"type\":[\"string\",\"null\"]},\"q4\":{\"type\":[\"string\",\"null\"]},\"q5\":{\"type\":[\"string\",\"null\"]},\"q6\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"q1\",\"q2\",\"q3\",\"q4\",\"q5\",\"q6\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "qa", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Document summaries · English (US)
import json
import os
import httpx
PROMPT = "Summarize the document below (meeting minutes, a project status report or an email thread) in at\nmost 80 words.\n\nRules:\n- Include every final decision: the approved amounts, the deadlines that apply and the people\n responsible for the action items (full names).\n- Do not mention rejected proposals or their amounts, and do not mention deadlines that were\n changed (the original dates); state only the deadlines that apply.\n- Copy amounts and dates exactly as in the document; do not round them.\n- Write in the language of the document, factually and in a formal register, as running text\n without headings.\n- Return only JSON in the form {\"summary\": \"…\"}.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\"}},\"required\":[\"summary\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "qwen/qwen3.8-flash",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "summaries", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Free with login
A ready-to-use prompt and JSON schema for this model.
Just your email, no password.
Badge
Embed the current result on your site; the badge updates with every run.
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/qa-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/qa-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/qa-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/qa-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/qa-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/summaries-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/summaries-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/summaries-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/summaries-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/summaries-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>