Mistral · open weights
Mistral Small 4
How Mistral Small 4 by Mistral handles real business documents: accuracy, true cost, speed and where it goes wrong.
List price: $0.15 per 1M input tokens, $0.60 per 1M output tokens · mistralai/mistral-small-2603
Results
| Task · documents | Rank | Correct fields | Error-free documents | Per 1,000 documents | To fix / 1,000 | Speed (median) | Bar |
|---|---|---|---|---|---|---|---|
| Contract clauses · English (US) | 10/10 | 91.5% | 43% | $0.17 | 567 | 1.1 sec | below the bar |
| Contract clauses · Czech | 10/10 | 99.0% | 92% | $0.23 | 80 | 1.2 sec | meets the bar |
| Contract clauses · German | 10/10 | 94.0% | 45% | $0.21 | 553 | 1.2 sec | below the bar |
| Email triage · English (US) | 10/10 | 89.7% | 37% | $0.09 | 627 | 0.8 sec | below the bar |
| Email triage · Czech | 10/10 | 87.7% | 32% | $0.12 | 685 | 0.8 sec | below the bar |
| Email triage · German | 10/10 | 87.1% | 29% | $0.11 | 706 | 0.8 sec | below the bar |
| Invoice extraction · English (US) | 9/10 | 95.6% | 58% | $0.41 | 420 | 1.8 sec | meets the bar |
| Invoice extraction · Czech | 8/10 | 93.6% | 56% | $0.44 | 440 | 2.0 sec | below the bar |
| Invoice extraction · German | 9/10 | 94.6% | 50% | $0.44 | 500 | 2.0 sec | below the bar |
| Invoice extraction · Spanish | 9/10 | 91.9% | 31% | $0.44 | 691 | 2.2 sec | below the bar |
| Invoice extraction · French | 8/10 | 94.2% | 50% | $0.45 | 500 | 2.1 sec | below the bar |
| Personal data detection · English (US) | 9/10 | 99.4% | 96% | $0.08 | 40 | 0.8 sec | meets the bar |
| Personal data detection · Czech | 8/10 | 98.8% | 92% | $0.09 | 81 | 0.9 sec | meets the bar |
| Personal data detection · German | 10/10 | 99.1% | 94% | $0.09 | 65 | 0.9 sec | meets the bar |
Where it goes wrong
The fields it gets wrong most often and the kinds of errors, from all documents of the benchmark.
Contract clauses · English (US)
Weakest fields
- Provider43.3%
- Client46.7%
Kinds of errors (number of fields)
Contract clauses · Czech
Weakest fields
- Provider95.3%
- Client95.3%
- Effective date96.7%
Kinds of errors (number of fields)
Contract clauses · German
Weakest fields
- Provider ID62.4%
- Client ID63.1%
- Provider98.6%
- Client98.6%
Kinds of errors (number of fields)
Email triage · English (US)
Weakest fields
- Priority61.3%
- Sentiment71.3%
- Deadline76.0%
- Amount99.3%
Kinds of errors (number of fields)
Email triage · Czech
Weakest fields
- Priority52.4%
- Deadline69.8%
- Sentiment72.5%
- Amount98.0%
Kinds of errors (number of fields)
Email triage · German
Weakest fields
- Priority55.2%
- Sentiment65.0%
- Deadline70.6%
- Amount96.5%
Kinds of errors (number of fields)
Invoice extraction · English (US)
Weakest fields
- Net by tax rate72.0%
- Payment reference87.3%
- Invoice date97.3%
- Bank account98.0%
Kinds of errors (number of fields)
Invoice extraction · Czech
Weakest fields
- Tax point76.0%
- Net by tax rate90.0%
- Bank account90.0%
- Supplier91.3%
Kinds of errors (number of fields)
Invoice extraction · German
Weakest fields
- Supplier ID (EIN, IČO…)80.0%
- Payment reference87.1%
- Tax point87.9%
- Bank account91.4%
Kinds of errors (number of fields)
Invoice extraction · Spanish
Weakest fields
- VAT ID53.6%
- Tax point69.1%
- Payment reference88.1%
- Bank account95.2%
Kinds of errors (number of fields)
Invoice extraction · French
Weakest fields
- Bank account75.0%
- Payment reference85.5%
- Tax point92.1%
- Supplier ID (EIN, IČO…)93.4%
Kinds of errors (number of fields)
Personal data detection · English (US)
Weakest fields
- Names97.3%
- Addresses98.7%
Kinds of errors (number of fields)
Personal data detection · Czech
Weakest fields
- Names94.0%
- Addresses98.7%
- Dates of birth98.7%
Kinds of errors (number of fields)
Personal data detection · German
Weakest fields
- Names95.0%
- Addresses99.3%
- Bank accounts99.3%
Kinds of errors (number of fields)
Compliance
- In the EU yes
- No retention (ZDR) yes
- DPA yes
- Open weights yes
EU and ZDR: Mistral. Mistral hosts data in the EU by default.
Use it now
The exact request of the benchmark (same prompt, JSON schema, temperature 0), ready to call Mistral Small 4 via OpenRouter.
Contract clauses · English (US)
import json
import os
import httpx
PROMPT = "Extract the key terms of the contract below and return only JSON with these keys:\n\nprovider_name the party that delivers (provider, seller, landlord, licensor, contractor),\n exactly as written, including the legal form\nprovider_id that party's company ID (EIN), if stated\nclient_name the party that pays (client, buyer, tenant, licensee, customer)\nclient_id that party's company ID (EIN), if stated\ncontract_type one of: services, supply, lease, license, maintenance\nprice the agreed price excluding VAT / sales tax (number)\ncurrency ISO 4217 code (USD, EUR, CZK, …)\nprice_period one of: one_time, monthly, yearly\neffective_date the date the contract takes effect (YYYY-MM-DD)\nend_date the date a fixed term ends (YYYY-MM-DD); null if the term is indefinite\nnotice_period_months the termination notice period in months (number); null if there is none\npenalty_pct_per_day the late-payment penalty in percent of the overdue amount per day\n (e.g. 0.05 for 0.05 %); null if there is none\ngoverning_law ISO 3166 country code of the governing law; US states as US-XX (e.g. US-NY)\n\nRules:\n- Resolve relative dates (\"on the date of signature\", \"the first day of the following month\")\n against the signing date.\n- The signing date is not the effective date unless the contract says so.\n- Deposits, advance payments and fixed penalties for other breaches are not the price or the\n late-payment penalty.\n- If a value is not in the contract, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"provider_name\":{\"type\":[\"string\",\"null\"]},\"provider_id\":{\"type\":[\"string\",\"null\"]},\"client_name\":{\"type\":[\"string\",\"null\"]},\"client_id\":{\"type\":[\"string\",\"null\"]},\"contract_type\":{\"type\":[\"string\",\"null\"],\"enum\":[\"services\",\"supply\",\"lease\",\"license\",\"maintenance\",null]},\"price\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"price_period\":{\"type\":[\"string\",\"null\"],\"enum\":[\"one_time\",\"monthly\",\"yearly\",null]},\"effective_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"end_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"notice_period_months\":{\"type\":[\"integer\",\"null\"]},\"penalty_pct_per_day\":{\"type\":[\"number\",\"null\"]},\"governing_law\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"provider_name\",\"provider_id\",\"client_name\",\"client_id\",\"contract_type\",\"price\",\"currency\",\"price_period\",\"effective_date\",\"end_date\",\"notice_period_months\",\"penalty_pct_per_day\",\"governing_law\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-small-2603",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "contracts", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Email triage · English (US)
import json
import os
import httpx
PROMPT = "You triage the support inbox. Read the email below and return only JSON with these keys:\n\ncategory one of: billing (invoices, payments, charges), delivery (where is the order, shipping,\n address change), complaint (damaged, wrong or defective goods, warranty claims),\n technical (login, app or system errors, integrations), cancellation (cancel an order,\n contract or subscription), sales (quotes, prices, interest before buying)\npriority one of: urgent, high, normal, low (rules below)\nsentiment one of: positive, neutral, negative (the customer's tone)\ncustomer_name full name of the customer who wrote the request\norder_number the customer's order number, if mentioned\ninvoice_number invoice number, if mentioned\namount the amount of money the request is about (number)\ncurrency ISO 4217 code of that amount (USD, EUR, CZK, …)\ndeadline the date by which the customer wants a resolution or answer (YYYY-MM-DD)\n\nPriority rules:\n- urgent: the customer cannot work right now, or the deadline is at most 1 day after the email date\n- high: the deadline is at most 7 days after the email date, or the customer threatens to cancel\n or escalate (lawyer, ending the business)\n- low: the customer says explicitly that it is not urgent\n- normal: everything else\n\nRules:\n- A thread: judge the newest message; order numbers, invoice numbers and amounts may come from the\n quoted earlier messages. A forwarded email: the customer is the original sender, not the\n colleague who forwarded it.\n- Resolve relative deadlines (\"by Friday\", \"by tomorrow\", \"by the end of the month\") against the\n date of the message that states them.\n- Support ticket numbers, tax IDs and phone numbers are not order or invoice numbers.\n- If a value is not in the email, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"category\":{\"type\":[\"string\",\"null\"],\"enum\":[\"billing\",\"delivery\",\"complaint\",\"technical\",\"cancellation\",\"sales\",null]},\"priority\":{\"type\":[\"string\",\"null\"],\"enum\":[\"urgent\",\"high\",\"normal\",\"low\",null]},\"sentiment\":{\"type\":[\"string\",\"null\"],\"enum\":[\"positive\",\"neutral\",\"negative\",null]},\"customer_name\":{\"type\":[\"string\",\"null\"]},\"order_number\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"amount\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"deadline\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"}},\"required\":[\"category\",\"priority\",\"sentiment\",\"customer_name\",\"order_number\",\"invoice_number\",\"amount\",\"currency\",\"deadline\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-small-2603",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "emails", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Invoice extraction · English (US)
import json
import os
import httpx
PROMPT = "Extract the following data from the attached document and return only JSON with these keys:\n\nsupplier_name supplier's name (exactly as printed, including the legal form)\nsupplier_id supplier's tax ID (EIN), if printed\ninvoice_number document number\npayment_reference payment reference to quote with the payment\nissue_date invoice date (YYYY-MM-DD)\ndue_date due date (YYYY-MM-DD)\nnet_by_rate amount per sales tax rate, e.g. {\"8.25\": 1845.00}; non-taxable amounts under \"0\"\nvat_by_rate sales tax per rate, e.g. {\"8.25\": 152.21}\ntotal total due (number, negative for credit notes)\ncurrency ISO 4217 currency code (USD, GBP, EUR, …)\nbank_account supplier's bank details as \"routing-number account-number\", or IBAN\n\nRules:\n- Return the supplier's data only, never the customer's.\n- If a value is not on the document, return null. Do not guess.\n- If the document has no tax breakdown, return net_by_rate and vat_by_rate as null.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"supplier_name\":{\"type\":[\"string\",\"null\"]},\"supplier_id\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"payment_reference\":{\"type\":[\"string\",\"null\"]},\"issue_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"due_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"net_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"vat_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"total\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"bank_account\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"supplier_name\",\"supplier_id\",\"invoice_number\",\"payment_reference\",\"issue_date\",\"due_date\",\"net_by_rate\",\"vat_by_rate\",\"total\",\"currency\",\"bank_account\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-small-2603",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "invoices", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Personal data detection · English (US)
import json
import os
import httpx
PROMPT = "Find all personal data of natural persons in the text below (for anonymisation) and return only\nJSON with these keys, each a list of strings (an empty list when there is none):\n\nperson_names full names of people, each person once, as written in the text\nemail_addresses email addresses of people\nphone_numbers phone numbers of people, as written\npostal_addresses home or delivery addresses of people, as written (street, city, ZIP)\nbirth_dates dates of birth (YYYY-MM-DD)\nnational_ids national identification numbers (e.g. SSN, birth numbers)\nbank_accounts bank account numbers of people, as written\n\nRules:\n- Only data of natural persons. Company names, company registration or tax numbers (EIN, IČO,\n VAT), generic company addresses such as info@ or support@, order and ticket numbers are not\n personal data: leave them out.\n- Dates that are not dates of birth (meetings, deliveries, start dates) are not personal data.\n- Do not invent or complete values; copy them from the text.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"person_names\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"email_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"phone_numbers\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"postal_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"birth_dates\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"national_ids\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"bank_accounts\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}}},\"required\":[\"person_names\",\"email_addresses\",\"phone_numbers\",\"postal_addresses\",\"birth_dates\",\"national_ids\",\"bank_accounts\"],\"additionalProperties\":false}")
def run(document_text: str) -> dict:
"""Text of the document in, the extracted fields as a dict out."""
response = httpx.post(
"https://openrouter.ai/api/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
json={
"model": "mistralai/mistral-small-2603",
"temperature": 0,
"messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
# For a scan or photo, send [{"type": "text", "text": PROMPT},
# {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
"response_format": {
"type": "json_schema",
"json_schema": {"name": "pii", "strict": False, "schema": SCHEMA},
},
},
timeout=120,
)
response.raise_for_status()
return json.loads(response.json()["choices"][0]["message"]["content"])
Badge
Embed the current result on your site; the badge updates with every run.
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-en.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-de.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-en.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-de.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-en.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-de.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-es.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-fr.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-en.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a><a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-de.svg" alt="Mistral Small 4 on ModelVerdict"></a>Compare with another model
- Mistral Small 4 vs Claude Sonnet 5
- Mistral Small 4 vs DeepSeek V4.1 Flash
- Mistral Small 4 vs Gemini 3.5 Flash Lite
- Mistral Small 4 vs Gemini 3.8 Flash
- Mistral Small 4 vs Gemma 4 31B
- Mistral Small 4 vs GPT-6 Luna
- Mistral Small 4 vs GPT-6 Sol
- Mistral Small 4 vs Mistral Medium 3.5
- Mistral Small 4 vs Qwen3.8 Flash