Mistral
Mistral Large 4New
How Mistral Large 4 by Mistral handles realistic business documents: accuracy, real cost, speed, and where it goes wrong.
List price: $0.68 per 1M input tokens, $2.09 per 1M output tokens · mistralai/mistral-large-4-0
Results
| Task · documents | Rank | Correct fieldsThe bar: the share of correct fields below which manual fixes cost more than a cheaper model saves. The black mark on the meter shows it. | Per 1,000 documents | To fix / 1,000How many of 1,000 documents have at least one wrong field and need someone to check and fix them. | Speed (median) |
|---|---|---|---|---|---|
| Contract clauses · English (US) | 12/14 | 96.10% | $5.24 | 253 | 41.4 sec |
| Contract clauses · Czech | 7/14 | 100.00% | $4.58 | 0 | 43.9 sec |
| Contract clauses · German | 7/14 | 99.44% | $5.12 | 47 | 36.2 sec |
| Contract clauses · Spanish | 9/14 | 100.00% | $4.83 | 0 | 35.0 sec |
| Contract clauses · French | 12/14 | 99.28% | $4.60 | 93 | 34.7 sec |
| Email triage · English (US) | 5/14 | 98.74% | $3.52 | 113 | 19.8 sec |
| Email triage · Czech | 6/14 | 98.15% | $4.41 | 107 | 28.4 sec |
| Email triage · German | 5/14 | 97.85% | $3.78 | 180 | 21.7 sec |
| Email triage · Spanish | 4/14 | 97.93% | $4.03 | 180 | 25.4 sec |
| Email triage · French | 5/14 | 98.30% | $3.70 | 153 | 24.5 sec |
| Invoice extraction · English (US) | 13/14 | 95.39% | $6.65 | 227 | 29.0 sec |
| Invoice extraction · Czech | 10/14 | 97.08% | $5.69 | 113 | 26.3 sec |
| Invoice extraction · German | 9/14 | 97.49% | $6.72 | 87 | 35.3 sec |
| Invoice extraction · Spanish | 11/14 | 96.51% | $7.49 | 120 | 41.3 sec |
| Invoice extraction · French | 14/14 | 88.97% | $8.71 | 347 | 43.4 sec |
| Personal data detection · English (US) | 11/14 | 99.62% | $2.89 | 27 | 28.2 sec |
| Personal data detection · Czech | 10/14 | 99.33% | $3.15 | 47 | 18.5 sec |
| Personal data detection · German | 13/14 | 99.52% | $2.61 | 33 | 18.6 sec |
| Personal data detection · Spanish | 9/14 | 99.81% | $3.16 | 13 | 21.9 sec |
| Personal data detection · French | 13/14 | 98.67% | $2.96 | 53 | 23.6 sec |
| Questions about a document · English (US) | 14/14 | 98.89% | $4.16 | 27 | 17.7 sec |
| Questions about a document · Czech | 14/14 | 89.89% | $7.08 | 220 | 33.6 sec |
| Questions about a document · German | 14/14 | 93.56% | $5.77 | 107 | 31.5 sec |
| Questions about a document · Spanish | 14/14 | 94.22% | $5.74 | 93 | 22.8 sec |
| Questions about a document · French | 14/14 | 92.33% | $5.74 | 127 | 19.8 sec |
| Document summaries · English (US) | 8/14 | 100.00% | $5.62 | 0 | 37.2 sec |
| Document summaries · Czech | 14/14 | 83.33% | $10.72 | 167 | 55.4 sec |
| Document summaries · German | 11/14 | 99.33% | $7.19 | 7 | 48.0 sec |
| Document summaries · Spanish | 14/14 | 91.33% | $7.99 | 87 | 38.3 sec |
| Document summaries · French | 14/14 | 72.67% | $11.26 | 273 | 46.4 sec |
Compare with another model
- Mistral Large 4 vs Claude Haiku 5.5
- Mistral Large 4 vs Claude Sonnet 5
- Mistral Large 4 vs Claude Sonnet 5.5
- Mistral Large 4 vs DeepSeek V4.1 Flash
- Mistral Large 4 vs Gemini 3.5 Flash Lite
- Mistral Large 4 vs Gemini 3.8 Flash
- Mistral Large 4 vs Gemma 4 31B
- Mistral Large 4 vs GLM-5.3 Flash
- Mistral Large 4 vs GPT-6 Luna
- Mistral Large 4 vs GPT-6 Sol
- Mistral Large 4 vs Mistral Medium 3.5
- Mistral Large 4 vs Mistral Small 4
- Mistral Large 4 vs Qwen3.8 Flash
Where it goes wrong
The fields it gets wrong most often and the error types, across all documents in the benchmark.
Contract clauses · English (US)
Weakest fields
Provider74.67%
Errors in this field: 13 of 40 example documents. For example:
- Santiago-Harris Group Co. · Columbus Printing Inc.
- Model's answer
- Santiago-Harris Group Co., a limited liability company
- Correct value
- Santiago-Harris Group Co.
- Pittsburgh Construction Co. · Evans Electric LLC
- Model's answer
- Pittsburgh Construction Co., a limited liability company
- Correct value
- Pittsburgh Construction Co.
- Frye & Sons, Co. · Nunez-Smith Group Inc.
- Model's answer
- Frye & Sons, Co., a limited liability company
- Correct value
- Frye & Sons, Co.
- Kim Electric Co. · Ramos & Sons, Co.
- Model's answer
- Kim Electric Co., a limited liability company
- Correct value
- Kim Electric Co.
- Seattle Printing Inc. · Gutierrez Landscaping LLC
- Model's answer
- Seattle Printing Inc., a Delaware corporation
- Correct value
- Seattle Printing Inc.
- Santiago-Harris Group Co. · Columbus Printing Inc.
Client74.67%
Errors in this field: 13 of 40 example documents. For example:
- Santiago-Harris Group Co. · Columbus Printing Inc.
- Model's answer
- Columbus Printing Inc., a Delaware corporation
- Correct value
- Columbus Printing Inc.
- Pittsburgh Construction Co. · Evans Electric LLC
- Model's answer
- Evans Electric LLC, a limited liability company
- Correct value
- Evans Electric LLC
- Frye & Sons, Co. · Nunez-Smith Group Inc.
- Model's answer
- Nunez-Smith Group Inc., a Delaware corporation
- Correct value
- Nunez-Smith Group Inc.
- Kim Electric Co. · Ramos & Sons, Co.
- Model's answer
- Ramos & Sons, Co., a limited liability company
- Correct value
- Ramos & Sons, Co.
- Seattle Printing Inc. · Gutierrez Landscaping LLC
- Model's answer
- Gutierrez Landscaping LLC, a limited liability company
- Correct value
- Gutierrez Landscaping LLC
- Santiago-Harris Group Co. · Columbus Printing Inc.
Error types (number of fields)
- wrong value76
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Contract clauses · Czech
No errors on this task.
Contract clauses · German
Weakest fields
Client ID96.00%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 3 of 40 example documents. For example:
- Schulz Systemtechnik GmbH · Elbdata GmbH
- Model's answer
- DE577677032
- Correct value
- — (not in the document, the field should stay empty)
- Meyer AG · Hartmann GmbH
- Model's answer
- DE459601009
- Correct value
- — (not in the document, the field should stay empty)
- Fuchs GmbH · Hartmann & Brandt UG (haftungsbeschränkt)
- Model's answer
- DE158006698
- Correct value
- — (not in the document, the field should stay empty)
- Schulz Systemtechnik GmbH · Elbdata GmbH
Provider ID96.67%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 2 of 40 example documents. For example:
- Schulz Systemtechnik GmbH · Elbdata GmbH
- Model's answer
- DE203080964
- Correct value
- — (not in the document, the field should stay empty)
- Meyer AG · Hartmann GmbH
- Model's answer
- DE926360058
- Correct value
- — (not in the document, the field should stay empty)
- Schulz Systemtechnik GmbH · Elbdata GmbH
Error types (number of fields)
- extra value11
What the error types mean
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Contract clauses · Spanish
No errors on this task.
Contract clauses · French
Weakest fields
Billing period90.67%
Errors in this field: 5 of 40 example documents. For example:
- SAS Alpitec · Muller Médias SAS
- Model's answer
- —
- Correct value
- one_time
- Denis et Associés EURL · SAS Alpitrans
- Model's answer
- —
- Correct value
- one_time
- Établissements Martin SAS · Lambert et Associés SAS
- Model's answer
- —
- Correct value
- one_time
- Établissements Joly SARL · Occidata SASU
- Model's answer
- —
- Correct value
- one_time
- Transports Perrin SARL · SAS Fontaine Médias
- Model's answer
- —
- Correct value
- one_time
- SAS Alpitec · Muller Médias SAS
Error types (number of fields)
- missing field13
- wrong value1
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
- wrong value
- The model gave a value other than the correct one.
Email triage · English (US)
Weakest fields
Sentiment90.67%
Errors in this field: 5 of 40 example documents. For example:
- Cody Baldwin
- Model's answer
- neutral
- Correct value
- positive
- Tyler Brown
- Model's answer
- negative
- Correct value
- neutral
- Daniel Stewart
- Model's answer
- negative
- Correct value
- neutral
- Allison Floyd
- Model's answer
- negative
- Correct value
- neutral
- Sandra Hogan
- Model's answer
- negative
- Correct value
- neutral
- Cody Baldwin
Priority98.00%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value17
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · Czech
Weakest fields
Sentiment90.00%
Errors in this field: 2 of 40 example documents. For example:
- Simona Kolářová
- Model's answer
- neutral
- Correct value
- positive
- Ivo Dvořák
- Model's answer
- negative
- Correct value
- neutral
- Simona Kolářová
Priority98.00%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Category99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Customer name99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- wrong value16
- invalid JSON9
What the error types mean
- wrong value
- The model gave a value other than the correct one.
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Email triage · German
Weakest fields
Sentiment83.33%
Errors in this field: 5 of 40 example documents. For example:
- Rolf-Peter Metz
- Model's answer
- negative
- Correct value
- neutral
- Alexej Sauer
- Model's answer
- neutral
- Correct value
- positive
- Dogan Wernecke
- Model's answer
- negative
- Correct value
- neutral
- Nadja Wiek
- Model's answer
- negative
- Correct value
- neutral
- Ibrahim Walter
- Model's answer
- negative
- Correct value
- neutral
- Rolf-Peter Metz
Priority97.33%
Errors in this field: 1 of 40 example documents. For example:
- Friedbert Hoffmann
- Model's answer
- urgent
- Correct value
- normal
- Friedbert Hoffmann
Error types (number of fields)
- wrong value29
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · Spanish
Weakest fields
Sentiment86.67%
Errors in this field: 2 of 40 example documents. For example:
- Rosalva Talavera
- Model's answer
- negative
- Correct value
- neutral
- Consuelo Diego
- Model's answer
- positive
- Correct value
- neutral
- Rosalva Talavera
Priority94.67%
Errors in this field: 2 of 40 example documents. For example:
- Águeda Echeverría
- Model's answer
- urgent
- Correct value
- normal
- Julieta Acuña
- Model's answer
- urgent
- Correct value
- normal
- Águeda Echeverría
Error types (number of fields)
- wrong value28
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Email triage · French
Weakest fields
Sentiment88.00%
Errors in this field: 4 of 40 example documents. For example:
- Tristan Bonneau
- Model's answer
- neutral
- Correct value
- positive
- Zoé Da Costa
- Model's answer
- negative
- Correct value
- neutral
- Capucine Maréchal
- Model's answer
- negative
- Correct value
- neutral
- Raymond Bonnin
- Model's answer
- negative
- Correct value
- neutral
- Tristan Bonneau
Priority96.67%
Errors in this field: 2 of 40 example documents. For example:
- Noël Masson
- Model's answer
- urgent
- Correct value
- normal
- Denis Lopes
- Model's answer
- urgent
- Correct value
- normal
- Noël Masson
Error types (number of fields)
- wrong value23
What the error types mean
- wrong value
- The model gave a value other than the correct one.
Invoice extraction · English (US)
Weakest fields
Bank account82.00%
Errors in this field: 5 of 40 example documents. For example:
- Cochran-Graham Group LLC · A-007505
- Model's answer
- 026009593 675370916691
- Correct value
- 026009593 67537091691
- Mason-Nelson Group LLC · 2025-3952
- Model's answer
- 021000089 780494449864
- Correct value
- 021000089 78049449864
- Sarah Villanueva · 2025-7876
- Model's answer
- 0430000096 2911246075
- Correct value
- 043000096 2911246075
- Archer-Smith Group LLC · A-004846
- Model's answer
- 043000096 184808030382
- Correct value
- 043000096 18480803082
- Gomez & Sons, LLC · 7855
- Model's answer
- 026009593 684199240457
- Correct value
- 026009593 68419924057
- Cochran-Graham Group LLC · A-007505
Payment reference91.33%
Errors in this field: 1 of 40 example documents. For example:
- Cochran-Graham Group LLC · A-007505
- Model's answer
- A-007505
- Correct value
- — (not in the document, the field should stay empty)
- Cochran-Graham Group LLC · A-007505
Supplier97.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Supplier ID (EIN, IČO…)97.33%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- invalid JSON44
- wrong value23
- extra value9
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Invoice extraction · Czech
Weakest fields
Bank account90.67%
Errors in this field: 2 of 40 example documents. For example:
- Sedlák s.r.o. · 20250158
- Model's answer
- 519646-6074030776/0800
- Correct value
- 519646-607403076/0800
- Stavebniny Sýkora s.r.o. · FV-2025-00537
- Model's answer
- 439372192/0600
- Correct value
- 4393721922/0600
- Sedlák s.r.o. · 20250158
Supplier ID (EIN, IČO…)96.00%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 1 of 40 example documents. For example:
- Holub s.r.o. · FV2500655
- Model's answer
- 151555536
- Correct value
- 15155536
- Holub s.r.o. · FV2500655
VAT ID96.00%
What we ask for: The supplier's VAT ID (DIČ, USt-IdNr., VAT number). Not the registration number.
Errors in this field: 1 of 40 example documents. For example:
- Holub s.r.o. · FV2500655
- Model's answer
- CZ151555536
- Correct value
- CZ15155536
- Holub s.r.o. · FV2500655
Invoice number97.33%
Errors in this field: 1 of 40 example documents. For example:
- Východočeská obchodní s.r.o. · FV2501176
- Model's answer
- FV25011176
- Correct value
- FV2501176
- Východočeská obchodní s.r.o. · FV2501176
Error types (number of fields)
- invalid JSON39
- wrong value18
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
Invoice extraction · German
Weakest fields
Bank account94.00%
Errors in this field: 8 of 40 example documents. For example:
- Druckerei Baumann GmbH · 2025-0324
- Model's answer
- DE17100500000827542860
- Correct value
- DE17100500008275428260
- Ostmetall GmbH · AN-2025-00580
- Model's answer
- —
- Correct value
- DE87100100108007224826
- Dachdeckerei Fischer GmbH · 2025/0014
- Model's answer
- —
- Correct value
- DE49500700106461751976
- Malerbetrieb Otfried Haas · RE-2025-00168
- Model's answer
- DE80370400440473174339
- Correct value
- DE80370400447318743389
- Christoph Haas – Hausmeisterservice · 2025-1432
- Model's answer
- DE75500700100391179549
- Correct value
- DE75500700103911791549
- Druckerei Baumann GmbH · 2025-0324
Payment reference96.67%
Errors in this field: 3 of 40 example documents. For example:
- Ostmetall GmbH · AN-2025-00580
- Model's answer
- —
- Correct value
- KD-46430
- Hoffmann UG (haftungsbeschränkt) · 2025/1479
- Model's answer
- KD-73584
- Correct value
- — (not in the document, the field should stay empty)
- Dachdeckerei Fischer GmbH · 2025/0014
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Ostmetall GmbH · AN-2025-00580
Supplier ID (EIN, IČO…)97.33%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 2 of 40 example documents. For example:
- Ostmetall GmbH · AN-2025-00580
- Model's answer
- —
- Correct value
- 220/287/61796
- Dachdeckerei Fischer GmbH · 2025/0014
- Model's answer
- —
- Correct value
- 029/185/04707
- Ostmetall GmbH · AN-2025-00580
VAT ID97.33%
What we ask for: The supplier's VAT ID (DIČ, USt-IdNr., VAT number). Not the registration number.
Errors in this field: 2 of 40 example documents. For example:
- Ostmetall GmbH · AN-2025-00580
- Model's answer
- —
- Correct value
- DE104855219
- Dachdeckerei Fischer GmbH · 2025/0014
- Model's answer
- —
- Correct value
- DE757826106
- Ostmetall GmbH · AN-2025-00580
Error types (number of fields)
- invalid JSON39
- wrong value7
- extra value3
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Invoice extraction · Spanish
Weakest fields
Payment reference94.00%
Errors in this field: 2 of 40 example documents. For example:
- Leyre Lladó Vall · F2025-1476
- Model's answer
- 4548118187
- Correct value
- — (not in the document, the field should stay empty)
- Ortuño e Hijos, S.L. · FV26-01184
- Model's answer
- —
- Correct value
- FV26-01184
- Leyre Lladó Vall · F2025-1476
Bank account94.67%
Errors in this field: 3 of 40 example documents. For example:
- Leyre Lladó Vall · F2025-1476
- Model's answer
- ES5500732302517326069716
- Correct value
- ES5000732302517326069716
- Escobar Comunicación S.L. · RECT-2025-0924
- Model's answer
- ES82303509913806856155505
- Correct value
- ES8230350991380685615505
- Ortuño e Hijos, S.L. · FV26-01184
- Model's answer
- —
- Correct value
- ES3314650805741517390630
- Leyre Lladó Vall · F2025-1476
Supplier95.33%
Errors in this field: 2 of 40 example documents. For example:
- Climatización Levante S.L. · F2025-1305
- Model's answer
- Climaticación Levante S.L.
- Correct value
- Climatización Levante S.L.
- Ortuño e Hijos, S.L. · FV26-01184
- Model's answer
- —
- Correct value
- Ortuño e Hijos, S.L.
- Climatización Levante S.L. · F2025-1305
Net amount by tax rate95.33%
Errors in this field: 1 of 40 example documents. For example:
- Ortuño e Hijos, S.L. · FV26-01184
- Model's answer
- —
- Correct value
- 10 %: 3152.70, 21 %: 1515.52
- Ortuño e Hijos, S.L. · FV26-01184
Error types (number of fields)
- invalid JSON52
- wrong value8
- extra value5
- missing field3
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- missing field
- The value is in the document, but the model left the field empty.
Invoice extraction · French
Weakest fields
VAT ID76.67%
What we ask for: The supplier's VAT ID (DIČ, USt-IdNr., VAT number). Not the registration number.
Errors in this field: 8 of 40 example documents. For example:
- Établissements Blanc SAS · F-11122
- Model's answer
- FR7386307085
- Correct value
- FR73866307085
- Établissements Denis SASU · FAC-2025-00031
- Model's answer
- FR0436060047
- Correct value
- FR04360600472
- SARL Garcia Informatique · F2025-0111
- Model's answer
- FR6895757217
- Correct value
- FR68957572217
- Nicolas & Fils SARL · 2025-12-350
- Model's answer
- FR6230149159
- Correct value
- FR62301149159
- Carrosserie Alain Meunier EI · FAC-2026-00869
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- Établissements Blanc SAS · F-11122
Payment reference86.00%
Errors in this field: 8 of 40 example documents. For example:
- Établissements Michel SA · FAC25100622
- Model's answer
- FAC25100622
- Correct value
- — (not in the document, the field should stay empty)
- Établissements Bertrand SASU · F2025-0066
- Model's answer
- BC-2025-7576
- Correct value
- — (not in the document, the field should stay empty)
- Constance Martin – Rédactrice (EI) · FAC25080663
- Model's answer
- FAC250806663
- Correct value
- FAC25080663
- SA Établissements Gauthier · F2025-1451
- Model's answer
- F2025-1451
- Correct value
- — (not in the document, the field should stay empty)
- Carrosserie Alain Meunier EI · FAC-2026-00869
- Model's answer
- —
- Correct value
- FAC-2026-00869
- Établissements Michel SA · FAC25100622
Bank account86.00%
Errors in this field: 6 of 40 example documents. For example:
- Établissements Michel SA · FAC25100622
- Model's answer
- FR76300030355400070183552555
- Correct value
- FR7630003035540007018355255
- Établissements Denis SASU · FAC-2025-00031
- Model's answer
- FR76300020218405986275550235
- Correct value
- FR7630002021840598627550235
- Carrosserie Alain Meunier EI · FAC-2026-00869
- Model's answer
- —
- Correct value
- FR7630066028955970224752532
- Novabat SASU · 2026-04-646
- Model's answer
- —
- Correct value
- FR7610278016980005898269212
- Bernard et Associés SASU · FAC-2025-01115
- Model's answer
- —
- Correct value
- FR7617515003460006296482336
- Établissements Michel SA · FAC25100622
Supplier ID (EIN, IČO…)88.67%
What we ask for: The company registration number (Czech IČO, German Handelsregister, US EIN…). Not the VAT ID (DIČ, USt-IdNr.). If the document has none, the field should stay empty.
Errors in this field: 6 of 40 example documents. For example:
- Nettoyage Mercier SASU · AC-10176
- Model's answer
- 596223297000018
- Correct value
- 59622329700018
- Établissements Bonnet EURL · AV-10599
- Model's answer
- 619996333000164
- Correct value
- 61999633300164
- Carrosserie Alain Meunier EI · FAC-2026-00869
- Model's answer
- —
- Correct value
- 86714689600032
- SARL Lemaire et Associés · ACP-2025-00515
- Model's answer
- 795592005000016
- Correct value
- 79559200500016
- Novabat SASU · 2026-04-646
- Model's answer
- —
- Correct value
- 61910333600019
- Nettoyage Mercier SASU · AC-10176
Error types (number of fields)
- invalid JSON169
- wrong value41
- extra value5
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- wrong value
- The model gave a value other than the correct one.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
Personal data detection · English (US)
Weakest fields
Addresses98.00%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Phone numbers99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- missing field4
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · Czech
Weakest fields
Addresses96.67%
Errors in this field: 1 of 40 example documents. For example:
- Jonáš Toman
- Model's answer
- —
- Correct value
- Libichovská 91, 170 34 Šenov
- Jonáš Toman
Names98.67%
Errors in this field: 1 of 40 example documents. For example:
- Soňa Vítková
- Model's answer
- Soňa Vítková; Roman Pokorný; Kristýn Hrubá
- Correct value
- Soňa Vítková; Roman Pokorný; Kristýna Hrubá
- Soňa Vítková
Error types (number of fields)
- missing field5
- extra value1
- wrong value1
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- wrong value
- The model gave a value other than the correct one.
Personal data detection · German
Weakest fields
Addresses96.67%
Errors in this field: 1 of 40 example documents. For example:
- Gunnar Zorbach
- Model's answer
- —
- Correct value
- Erol-Pechel-Platz 2-8, 64136 Düren
- Gunnar Zorbach
Error types (number of fields)
- missing field5
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · Spanish
Weakest fields
Addresses98.67%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- missing field2
What the error types mean
- missing field
- The value is in the document, but the model left the field empty.
Personal data detection · French
Weakest fields
Names96.67%
Errors in this field: 1 of 40 example documents. For example:
- Susanne Couturier
- Model's answer
- Susanne Couturier; Marc Paris; Julien
- Correct value
- Susanne Couturier; Marc Paris
- Susanne Couturier
Addresses97.33%
Errors in this field: 1 of 40 example documents. For example:
- Michel Martins
- Model's answer
- —
- Correct value
- 48, boulevard Reynaud, 43794 Paris
- Michel Martins
Email addresses99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Phone numbers99.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- invalid JSON7
- extra value4
- missing field3
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- extra value
- The model filled in something that does not belong there: the field should have stayed empty (the document does not contain it), or it added an extra value, e.g., a rejected proposal in a summary.
- missing field
- The value is in the document, but the model left the field empty.
Questions about a document · English (US)
Weakest fields
Question 698.00%
Errors in this field: 2 of 40 example documents. For example:
- SLA: What service credit, as a share …
- Model's answer
- 0.999
- Correct value
- 99.9
- SLA: To what email address does the …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: What service credit, as a share …
Question 198.67%
Errors in this field: 1 of 40 example documents. For example:
- SLA: What service credit, as a share …
- Model's answer
- 0.05
- Correct value
- 5
- SLA: What service credit, as a share …
Question 298.67%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Question 399.33%
No errors in the 40 example documents; all errors are in the non-public part of the dataset.
Error types (number of fields)
- invalid JSON6
- number format4
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Questions about a document · Czech
Weakest fields
Question 288.67%
Errors in this field: 4 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Jak dlouho poskytovatel uchovává zálohy dat? …
- Model's answer
- —
- Correct value
- 8
- SLA: S jakým předstihem musí poskytovatel oznámit …
- Model's answer
- 0.15
- Correct value
- 15
- SLA: Jaká je reakční doba u požadavků …
Question 688.67%
Errors in this field: 5 of 40 example documents. For example:
- SLA: Od kterého dne je dokument účinný? …
- Model's answer
- 0.998
- Correct value
- 99.8
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- —
- Correct value
- 5
- SLA: Na jakou slevu z měsíčního paušálu …
- Model's answer
- 0.999
- Correct value
- 99.9
- SLA: Jak dlouho poskytovatel uchovává zálohy dat? …
- Model's answer
- —
- Correct value
- 1
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: Od kterého dne je dokument účinný? …
Question 390.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- —
- Correct value
- 2026-04-25
- SLA: Jaká smluvní pokuta hrozí poskytovateli za …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: Jak dlouho poskytovatel uchovává zálohy dat? …
- Model's answer
- —
- Correct value
- 99.5
- SLA: Jaká je reakční doba u požadavků …
Question 490.00%
Errors in this field: 4 of 40 example documents. For example:
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
- Model's answer
- 0.999
- Correct value
- 99.9
- SLA: Jaká je reakční doba u požadavků …
- Model's answer
- —
- Correct value
- 99.7
- SLA: Jaká smluvní pokuta hrozí poskytovateli za …
- Model's answer
- 0.2
- Correct value
- 20
- SLA: Jak dlouho poskytovatel uchovává zálohy dat? …
- Model's answer
- —
- Correct value
- 32000.00
- SLA: Na jakou e-mailovou adresu hlásí objednatel …
Error types (number of fields)
- invalid JSON60
- number format31
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Questions about a document · German
Weakest fields
Question 392.67%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
- Model's answer
- —
- Correct value
- 99.7
- SLA: Wie lange bewahrt der Anbieter Datensicherungen …
- Model's answer
- 0.995
- Correct value
- 99.5
- SLA: An welche E-Mail-Adresse meldet der Auftraggeber …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
Question 293.33%
Errors in this field: 2 of 40 example documents. For example:
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
- Model's answer
- —
- Correct value
- 4
- SLA: An welche E-Mail-Adresse meldet der Auftraggeber …
- Model's answer
- —
- Correct value
- 2025-12-24
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
Question 593.33%
Errors in this field: 2 of 40 example documents. For example:
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
- Model's answer
- —
- Correct value
- 12
- SLA: An welche E-Mail-Adresse meldet der Auftraggeber …
- Model's answer
- —
- Correct value
- 14
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
Question 194.00%
Errors in this field: 2 of 40 example documents. For example:
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: An welche E-Mail-Adresse meldet der Auftraggeber …
- Model's answer
- —
- Correct value
- kundenservice@davids-dippel.example
- SLA: Welche Vertragsstrafe schuldet der Anbieter bei …
Error types (number of fields)
- invalid JSON48
- number format10
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Questions about a document · Spanish
Weakest fields
Question 294.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- —
- Correct value
- 99.7
- SLA: ¿Qué abono sobre la cuota mensual …
- Model's answer
- —
- Correct value
- 4
- SLA: ¿A qué dirección de correo electrónico …
- Model's answer
- —
- Correct value
- 2025-12-24
- SLA: ¿Cuál es el tiempo de respuesta …
Question 394.00%
Errors in this field: 4 of 40 example documents. For example:
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: ¿Qué abono sobre la cuota mensual …
- Model's answer
- —
- Correct value
- 2026-01-24
- SLA: ¿Durante cuánto tiempo conserva el Proveedor …
- Model's answer
- 0.995
- Correct value
- 99.5
- SLA: ¿A qué dirección de correo electrónico …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: ¿Cuál es el tiempo de respuesta …
Question 494.00%
Errors in this field: 4 of 40 example documents. For example:
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- —
- Correct value
- 1
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- 0.997
- Correct value
- 99.7
- SLA: ¿Qué abono sobre la cuota mensual …
- Model's answer
- —
- Correct value
- 1
- SLA: ¿A qué dirección de correo electrónico …
- Model's answer
- —
- Correct value
- 12
- SLA: ¿Cuál es el tiempo de respuesta …
Question 694.00%
Errors in this field: 4 of 40 example documents. For example:
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- —
- Correct value
- 2025-12-21
- SLA: ¿Cuál es el tiempo de respuesta …
- Model's answer
- 0.05
- Correct value
- 5
- SLA: ¿Qué abono sobre la cuota mensual …
- Model's answer
- —
- Correct value
- 99.9
- SLA: ¿A qué dirección de correo electrónico …
- Model's answer
- —
- Correct value
- 99.7
- SLA: ¿Cuál es el tiempo de respuesta …
Error types (number of fields)
- invalid JSON42
- number format10
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Questions about a document · French
Weakest fields
Question 192.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- 6
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- —
- Correct value
- 5
- SLA: À quelle adresse e-mail le Client …
- Model's answer
- —
- Correct value
- service-client@mallet.example
- SLA: Quel est le délai d'intervention pour …
Question 292.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- —
- Correct value
- 4
- SLA: À quelle adresse e-mail le Client …
- Model's answer
- —
- Correct value
- 2025-12-24
- SLA: Quel est le délai d'intervention pour …
Question 392.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- contact@herve.example
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- —
- Correct value
- 2026-01-24
- SLA: À quelle adresse e-mail le Client …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: Quel est le délai d'intervention pour …
Question 592.00%
Errors in this field: 3 of 40 example documents. For example:
- SLA: Quel est le délai d'intervention pour …
- Model's answer
- —
- Correct value
- 20
- SLA: Quel avoir sur la redevance mensuelle …
- Model's answer
- —
- Correct value
- — (not in the document, the field should stay empty)
- SLA: À quelle adresse e-mail le Client …
- Model's answer
- —
- Correct value
- 14
- SLA: Quel est le délai d'intervention pour …
Error types (number of fields)
- invalid JSON54
- number format15
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
- number format
- A number in the wrong form, e.g., 0.0005 instead of 0.05%.
Document summaries · English (US)
No errors on this task.
Document summaries · Czech
Weakest fields
Amounts83.33%
Errors in this field: 6 of 40 example documents. For example:
- Alois Sedlák, Peter Tesař, Rudolf Vítek
- Model's answer
- —
- Correct value
- 85000; 350000; 430000
- Andrea Dostálová, Mojmír Strnad, Luděk Hruška
- Model's answer
- —
- Correct value
- 950000; 1310000; 240000
- Václav Vítek, Svatoslav Němec
- Model's answer
- —
- Correct value
- 115000; 1340000; 145000
- Teodor Hruška
- Model's answer
- —
- Correct value
- 250000; 90000; 180000
- Vanda Vacková
- Model's answer
- —
- Correct value
- 590000; 410000; 210000
- Alois Sedlák, Peter Tesař, Rudolf Vítek
Deadlines83.33%
Errors in this field: 6 of 40 example documents. For example:
- Alois Sedlák, Peter Tesař, Rudolf Vítek
- Model's answer
- —
- Correct value
- 2025-08-23
- Andrea Dostálová, Mojmír Strnad, Luděk Hruška
- Model's answer
- —
- Correct value
- 2026-08-20; 2026-07-07
- Václav Vítek, Svatoslav Němec
- Model's answer
- —
- Correct value
- 2026-10-02; 2026-08-17
- Teodor Hruška
- Model's answer
- —
- Correct value
- 2026-03-25; 2026-05-05
- Vanda Vacková
- Model's answer
- —
- Correct value
- 2026-07-08; 2026-06-18
- Alois Sedlák, Peter Tesař, Rudolf Vítek
Responsible people83.33%
Errors in this field: 6 of 40 example documents. For example:
- Alois Sedlák, Peter Tesař, Rudolf Vítek
- Model's answer
- —
- Correct value
- Alois Sedlák; Peter Tesař; Rudolf Vítek
- Andrea Dostálová, Mojmír Strnad, Luděk Hruška
- Model's answer
- —
- Correct value
- Andrea Dostálová; Mojmír Strnad; Luděk Hruška
- Václav Vítek, Svatoslav Němec
- Model's answer
- —
- Correct value
- Václav Vítek; Svatoslav Němec
- Teodor Hruška
- Model's answer
- —
- Correct value
- Teodor Hruška
- Vanda Vacková
- Model's answer
- —
- Correct value
- Vanda Vacková
- Alois Sedlák, Peter Tesař, Rudolf Vítek
Rejected proposals (must not appear)83.33%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 6 of 40 example documents. For example:
- Alois Sedlák, Peter Tesař, Rudolf Vítek
- Appeared in the output
- —
- Must not appear
- 115000
- Andrea Dostálová, Mojmír Strnad, Luděk Hruška
- Appeared in the output
- —
- Must not appear
- 1500000
- Václav Vítek, Svatoslav Němec
- Appeared in the output
- —
- Must not appear
- 165000
- Teodor Hruška
- Appeared in the output
- —
- Must not appear
- 370000
- Vanda Vacková
- Appeared in the output
- —
- Must not appear
- 900000
- Alois Sedlák, Peter Tesař, Rudolf Vítek
Error types (number of fields)
- invalid JSON150
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Document summaries · German
Weakest fields
Amounts99.33%
Errors in this field: 1 of 40 example documents. For example:
- Dogan Wernecke
- Model's answer
- —
- Correct value
- 15000; 69000; 4500
- Dogan Wernecke
Deadlines99.33%
Errors in this field: 1 of 40 example documents. For example:
- Dogan Wernecke
- Model's answer
- —
- Correct value
- 2025-11-18
- Dogan Wernecke
Responsible people99.33%
Errors in this field: 1 of 40 example documents. For example:
- Dogan Wernecke
- Model's answer
- —
- Correct value
- Dogan Wernecke
- Dogan Wernecke
Rejected proposals (must not appear)99.33%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 1 of 40 example documents. For example:
- Dogan Wernecke
- Appeared in the output
- —
- Must not appear
- 21000
- Dogan Wernecke
Error types (number of fields)
- invalid JSON6
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Document summaries · Spanish
Weakest fields
Amounts91.33%
Errors in this field: 3 of 40 example documents. For example:
- Lucas Santiago
- Model's answer
- —
- Correct value
- 8500; 88000; 3500
- Mamen Sabater, María Del Carmen Blasco, Victoria Calvet
- Model's answer
- —
- Correct value
- 21000; 3000
- Amador Maestre, Marciano Goicoechea
- Model's answer
- —
- Correct value
- 57000; 10000; 11500
- Lucas Santiago
Deadlines91.33%
Errors in this field: 3 of 40 example documents. For example:
- Lucas Santiago
- Model's answer
- —
- Correct value
- 2026-06-01; 2026-07-20
- Mamen Sabater, María Del Carmen Blasco, Victoria Calvet
- Model's answer
- —
- Correct value
- 2026-04-05; 2026-05-19
- Amador Maestre, Marciano Goicoechea
- Model's answer
- —
- Correct value
- 2025-06-30
- Lucas Santiago
Responsible people91.33%
Errors in this field: 3 of 40 example documents. For example:
- Lucas Santiago
- Model's answer
- —
- Correct value
- Lucas Santiago
- Mamen Sabater, María Del Carmen Blasco, Victoria Calvet
- Model's answer
- —
- Correct value
- Mamen Sabater; María Del Carmen Blasco; Victoria Calvet
- Amador Maestre, Marciano Goicoechea
- Model's answer
- —
- Correct value
- Amador Maestre; Marciano Goicoechea
- Lucas Santiago
Rejected proposals (must not appear)91.33%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 3 of 40 example documents. For example:
- Lucas Santiago
- Appeared in the output
- —
- Must not appear
- 13500
- Mamen Sabater, María Del Carmen Blasco, Victoria Calvet
- Appeared in the output
- —
- Must not appear
- 29000
- Amador Maestre, Marciano Goicoechea
- Appeared in the output
- —
- Must not appear
- 81000
- Lucas Santiago
Error types (number of fields)
- invalid JSON78
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Document summaries · French
Weakest fields
Amounts72.67%
Errors in this field: 7 of 40 example documents. For example:
- Odette Bourdon
- Model's answer
- —
- Correct value
- 11500; 24000
- Sabine Breton, David Coulon
- Model's answer
- —
- Correct value
- 3000; 4500; 11500
- Augustin Bruneau, Thomas Millet, Aurélie Muller
- Model's answer
- —
- Correct value
- 22000; 14000; 5500
- Aimée Hamel
- Model's answer
- —
- Correct value
- 6500; 70000; 3000
- Raymond Bonnin, Hugues Tessier, Grégoire Aubry
- Model's answer
- —
- Correct value
- 23000; 80000
- Odette Bourdon
Deadlines72.67%
Errors in this field: 7 of 40 example documents. For example:
- Odette Bourdon
- Model's answer
- —
- Correct value
- 2026-09-15; 2026-07-26
- Sabine Breton, David Coulon
- Model's answer
- —
- Correct value
- 2026-06-09
- Augustin Bruneau, Thomas Millet, Aurélie Muller
- Model's answer
- —
- Correct value
- 2026-05-10; 2026-05-25
- Aimée Hamel
- Model's answer
- —
- Correct value
- 2025-11-12; 2025-09-27
- Raymond Bonnin, Hugues Tessier, Grégoire Aubry
- Model's answer
- —
- Correct value
- 2025-04-06
- Odette Bourdon
Responsible people72.67%
Errors in this field: 7 of 40 example documents. For example:
- Odette Bourdon
- Model's answer
- —
- Correct value
- Odette Bourdon
- Sabine Breton, David Coulon
- Model's answer
- —
- Correct value
- Sabine Breton; David Coulon
- Augustin Bruneau, Thomas Millet, Aurélie Muller
- Model's answer
- —
- Correct value
- Augustin Bruneau; Thomas Millet; Aurélie Muller
- Aimée Hamel
- Model's answer
- —
- Correct value
- Aimée Hamel
- Raymond Bonnin, Hugues Tessier, Grégoire Aubry
- Model's answer
- —
- Correct value
- Raymond Bonnin; Hugues Tessier; Grégoire Aubry
- Odette Bourdon
Rejected proposals (must not appear)72.67%
What we ask for: Amounts from proposals the document rejects. They must not appear in the summary; they would mislead the reader.
Errors in this field: 7 of 40 example documents. For example:
- Odette Bourdon
- Appeared in the output
- —
- Must not appear
- 16000
- Sabine Breton, David Coulon
- Appeared in the output
- —
- Must not appear
- 4000
- Augustin Bruneau, Thomas Millet, Aurélie Muller
- Appeared in the output
- —
- Must not appear
- 32000
- Aimée Hamel
- Appeared in the output
- —
- Must not appear
- 10500
- Raymond Bonnin, Hugues Tessier, Grégoire Aubry
- Appeared in the output
- —
- Must not appear
- 35000
- Odette Bourdon
Error types (number of fields)
- invalid JSON246
What the error types mean
- invalid JSON
- The answer could not be read as JSON, so none of its fields count.
Free with login
Exactly where the model goes wrong: its weakest fields and the actual wrong answers.
Just your email, no password.
Language tax
How many more tokens this model needs for the same text compared with English.
- Czech 1.89×
- German 1.23×
- French 1.28×
- Spanish 1.26×
- Polish 2.01×
Compliance
- In the EU yes
- Zero retention (ZDR) yes
- DPA yes
- Open weights no
EU and ZDR: Mistral. Mistral hosts data in the EU by default.
Use it now
The exact request the benchmark sends (same prompt, JSON schema, temperature 0), ready to call Mistral Large 4 via OpenRouter.
Contract clauses · English (US)
import json
import os
import httpx
PROMPT = "Extract the key terms of the contract below and return only JSON with these keys:\n\nprovider_name the party that delivers (provider, seller, landlord, licensor, contractor),\n exactly as written, including the legal form\nprovider_id that party's company ID (EIN), if stated\nclient_name the party that pays (client, buyer, tenant, licensee, customer)\nclient_id that party's company ID (EIN), if stated\ncontract_type one of: services, supply, lease, license, maintenance\nprice the agreed price excluding VAT / sales tax (number)\ncurrency ISO 4217 code (USD, EUR, CZK, …)\nprice_period one of: one_time, monthly, yearly\neffective_date the date the contract takes effect (YYYY-MM-DD)\nend_date the date a fixed term ends (YYYY-MM-DD); null if the term is indefinite\nnotice_period_months the termination notice period in months (number); null if there is none\npenalty_pct_per_day the late-payment penalty in percent of the overdue amount per day\n (e.g. 0.05 for 0.05 %); null if there is none\ngoverning_law ISO 3166 country code of the governing law; US states as US-XX (e.g. US-NY)\n\nRules:\n- Resolve relative dates (\"on the date of signature\", \"the first day of the following month\")\n against the signing date.\n- The signing date is not the effective date unless the contract says so.\n- Deposits, advance payments and fixed penalties for other breaches are not the price or the\n late-payment penalty.\n- If a value is not in the contract, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"provider_name\":{\"type\":[\"string\",\"null\"]},\"provider_id\":{\"type\":[\"string\",\"null\"]},\"client_name\":{\"type\":[\"string\",\"null\"]},\"client_id\":{\"type\":[\"string\",\"null\"]},\"contract_type\":{\"type\":[\"string\",\"null\"],\"enum\":[\"services\",\"supply\",\"lease\",\"license\",\"maintenance\",null]},\"price\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"price_period\":{\"type\":[\"string\",\"null\"],\"enum\":[\"one_time\",\"monthly\",\"yearly\",null]},\"effective_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"end_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"notice_period_months\":{\"type\":[\"integer\",\"null\"]},\"penalty_pct_per_day\":{\"type\":[\"number\",\"null\"]},\"governing_law\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"provider_name\",\"provider_id\",\"client_name\",\"client_id\",\"contract_type\",\"price\",\"currency\",\"price_period\",\"effective_date\",\"end_date\",\"notice_period_months\",\"penalty_pct_per_day\",\"governing_law\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "contracts", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Email triage · English (US)
import json
import os
import httpx
PROMPT = "You triage the support inbox. Read the email below and return only JSON with these keys:\n\ncategory one of: billing (invoices, payments, charges), delivery (where is the order, shipping,\n address change), complaint (damaged, wrong or defective goods, warranty claims),\n technical (login, app or system errors, integrations), cancellation (cancel an order,\n contract or subscription), sales (quotes, prices, interest before buying)\npriority one of: urgent, high, normal, low (rules below)\nsentiment one of: positive, neutral, negative (the customer's tone)\ncustomer_name full name of the customer who wrote the request\norder_number the customer's order number, if mentioned\ninvoice_number invoice number, if mentioned\namount the amount of money the request is about (number)\ncurrency ISO 4217 code of that amount (USD, EUR, CZK, …)\ndeadline the date by which the customer wants a resolution or answer (YYYY-MM-DD)\n\nPriority rules:\n- urgent: the customer cannot work right now, or the deadline is at most 1 day after the email date\n- high: the deadline is at most 7 days after the email date, or the customer threatens to cancel\n or escalate (lawyer, ending the business)\n- low: the customer says explicitly that it is not urgent\n- normal: everything else\n\nRules:\n- A thread: judge the newest message; order numbers, invoice numbers and amounts may come from the\n quoted earlier messages. A forwarded email: the customer is the original sender, not the\n colleague who forwarded it.\n- Resolve relative deadlines (\"by Friday\", \"by tomorrow\", \"by the end of the month\") against the\n date of the message that states them.\n- Support ticket numbers, tax IDs and phone numbers are not order or invoice numbers.\n- If a value is not in the email, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"category\":{\"type\":[\"string\",\"null\"],\"enum\":[\"billing\",\"delivery\",\"complaint\",\"technical\",\"cancellation\",\"sales\",null]},\"priority\":{\"type\":[\"string\",\"null\"],\"enum\":[\"urgent\",\"high\",\"normal\",\"low\",null]},\"sentiment\":{\"type\":[\"string\",\"null\"],\"enum\":[\"positive\",\"neutral\",\"negative\",null]},\"customer_name\":{\"type\":[\"string\",\"null\"]},\"order_number\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"amount\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"deadline\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"}},\"required\":[\"category\",\"priority\",\"sentiment\",\"customer_name\",\"order_number\",\"invoice_number\",\"amount\",\"currency\",\"deadline\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "emails", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Invoice extraction · English (US)
import json
import os
import httpx
PROMPT = "Extract the following data from the attached document and return only JSON with these keys:\n\nsupplier_name supplier's name (exactly as printed, including the legal form)\nsupplier_id supplier's tax ID (EIN), if printed\ninvoice_number document number\npayment_reference payment reference to quote with the payment\nissue_date invoice date (YYYY-MM-DD)\ndue_date due date (YYYY-MM-DD)\nnet_by_rate amount per sales tax rate, e.g. {\"8.25\": 1845.00}; non-taxable amounts under \"0\"\nvat_by_rate sales tax per rate, e.g. {\"8.25\": 152.21}\ntotal total due (number, negative for credit notes)\ncurrency ISO 4217 currency code (USD, GBP, EUR, …)\nbank_account supplier's bank details as \"routing-number account-number\", or IBAN\n\nRules:\n- Return the supplier's data only, never the customer's.\n- If a value is not on the document, return null. Do not guess.\n- If the document has no tax breakdown, return net_by_rate and vat_by_rate as null.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"supplier_name\":{\"type\":[\"string\",\"null\"]},\"supplier_id\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"payment_reference\":{\"type\":[\"string\",\"null\"]},\"issue_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"due_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"net_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"vat_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"total\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"bank_account\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"supplier_name\",\"supplier_id\",\"invoice_number\",\"payment_reference\",\"issue_date\",\"due_date\",\"net_by_rate\",\"vat_by_rate\",\"total\",\"currency\",\"bank_account\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "invoices", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Personal data detection · English (US)
import json
import os
import httpx
PROMPT = "Find all personal data of natural persons in the text below (for anonymisation) and return only\nJSON with these keys, each a list of strings (an empty list when there is none):\n\nperson_names full names of people, each person once, as written in the text\nemail_addresses email addresses of people\nphone_numbers phone numbers of people, as written\npostal_addresses home or delivery addresses of people, as written (street, city, ZIP)\nbirth_dates dates of birth (YYYY-MM-DD)\nnational_ids national identification numbers (e.g. SSN, birth numbers)\nbank_accounts bank account numbers of people, as written\n\nRules:\n- Only data of natural persons. Company names, company registration or tax numbers (EIN, IČO,\n VAT), generic company addresses such as info@ or support@, order and ticket numbers are not\n personal data: leave them out.\n- Dates that are not dates of birth (meetings, deliveries, start dates) are not personal data.\n- Do not invent or complete values; copy them from the text.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"person_names\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"email_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"phone_numbers\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"postal_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"birth_dates\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"national_ids\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"bank_accounts\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}}},\"required\":[\"person_names\",\"email_addresses\",\"phone_numbers\",\"postal_addresses\",\"birth_dates\",\"national_ids\",\"bank_accounts\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "pii", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Questions about a document · English (US)
import json
import os
import httpx
PROMPT = "You answer questions about a business document. Below the document there are six numbered\nquestions. Answer each of them using only the text of the document and return only JSON with the\nkeys q1 to q6 (q1 = the answer to question 1, and so on).\n\nRules:\n- Each question states in parentheses the unit or format of the answer. Answer in that unit.\n- Counts (of days, months, hours) and percentages as a number without the unit and without the %\n sign, e.g. \"14\" or \"0.05\".\n- Amounts as a number with a decimal point, no thousands separators and no currency symbol,\n e.g. \"1500.00\".\n- Dates as YYYY-MM-DD.\n- Time ranges in 24-hour format HH:MM-HH:MM, e.g. \"08:00-17:00\".\n- Email addresses exactly as they appear in the document.\n- If the document does not contain the answer to a question, return null. Do not guess and do not\n add information that is not in the document.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"q1\":{\"type\":[\"string\",\"null\"]},\"q2\":{\"type\":[\"string\",\"null\"]},\"q3\":{\"type\":[\"string\",\"null\"]},\"q4\":{\"type\":[\"string\",\"null\"]},\"q5\":{\"type\":[\"string\",\"null\"]},\"q6\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"q1\",\"q2\",\"q3\",\"q4\",\"q5\",\"q6\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "qa", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Document summaries · English (US)
import json
import os
import httpx
PROMPT = "Summarize the document below (meeting minutes, a project status report or an email thread) in at\nmost 80 words.\n\nRules:\n- Include every final decision: the approved amounts, the deadlines that apply and the people\n responsible for the action items (full names).\n- Do not mention rejected proposals or their amounts, and do not mention deadlines that were\n changed (the original dates); state only the deadlines that apply.\n- Copy amounts and dates exactly as in the document; do not round them.\n- Write in the language of the document, factually and in a formal register, as running text\n without headings.\n- Return only JSON in the form {\"summary\": \"…\"}.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\"}},\"required\":[\"summary\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-large-4-0",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "summaries", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Free with login
A ready-to-use prompt and JSON schema for this model.
Just your email, no password.
Badge
Embed the current result on your site; the badge updates with every run.
<a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/contracts-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/contracts-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/contracts-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/contracts-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/contracts-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/emails-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/emails-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/emails-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/emails-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/emails-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/invoices-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/invoices-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/invoices-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/invoices-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/invoices-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/pii-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/pii-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/pii-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/pii-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/pii-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/qa-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/qa-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/qa-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/qa-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/qa-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/summaries-en.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/summaries-cs.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/summaries-de.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/summaries-es.svg" alt="Mistral Large 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-large-4/"><img src="https://modelverdict.ai/badge/mistral-large-4/summaries-fr.svg" alt="Mistral Large 4 on ModelVerdict"></a>