Gemma 4 31B vs Mistral Medium 3.5
Same documents, same prompt: where each model gets it right, what it really costs, and how fast it responds.
Verdict
Equally accurate; Gemma 4 31B is cheaper in 30 of 30 combinations.
More accurate
Tie
about the same
Cheaper
Gemma 4 31B
cheaper in 30 of 30
Faster
Mistral Medium 3.5
1.0 sec vs 2.9 sec
Fewer fixes
Mistral Medium 3.5
20 vs 27 documents per 1,000
Across 30 task and document-language combinations (speed and fixes: median).
At a glance
| Accuracy (correct fields) | Per 1,000 documents | |||||
|---|---|---|---|---|---|---|
| Task · documents | Gemma 4 31B | Mistral Medium 3.5 | Difference | Gemma 4 31B | Mistral Medium 3.5 | Difference |
| Invoice extraction | 97.64% | 97.76% | ▲ 0.12 | $0.17 | $4.37 | +2510 % |
| Email triage | 96.52% | 94.89% | ▼ 1.63 | $0.12 | $1.47 | +1121 % |
| Personal data detection | 99.90% | 99.71% | ▼ 0.19 | $0.08 | $1.15 | +1257 % |
| Contract clauses | 99.64% | 98.10% | ▼ 1.54 | $0.18 | $2.35 | +1205 % |
| Document summaries | 100.00% | 100.00% | ±0 | $0.13 | $1.57 | +1128 % |
| Questions about a document | 100.00% | 100.00% | ±0 | $0.15 | $1.81 | +1069 % |
| Accuracy (correct fields) | Per 1,000 documents | |||||
|---|---|---|---|---|---|---|
| Task · documents | Gemma 4 31B | Mistral Medium 3.5 | Difference | Gemma 4 31B | Mistral Medium 3.5 | Difference |
| Invoice extraction | 96.10% | 93.49% | ▼ 2.61 | $0.21 | $4.69 | +2113 % |
| Email triage | 97.11% | 94.52% | ▼ 2.59 | $0.16 | $1.95 | +1126 % |
| Personal data detection | 99.81% | 99.81% | ±0 | $0.12 | $1.40 | +1076 % |
| Contract clauses | 99.95% | 100.00% | ▲ 0.05 | $0.25 | $3.06 | +1103 % |
| Document summaries | 100.00% | 100.00% | ±0 | $0.19 | $2.31 | +1120 % |
| Questions about a document | 100.00% | 100.00% | ±0 | $0.21 | $2.51 | +1086 % |
| Accuracy (correct fields) | Per 1,000 documents | |||||
|---|---|---|---|---|---|---|
| Task · documents | Gemma 4 31B | Mistral Medium 3.5 | Difference | Gemma 4 31B | Mistral Medium 3.5 | Difference |
| Invoice extraction | 94.87% | 97.38% | ▲ 2.51 | $0.20 | $4.72 | +2272 % |
| Email triage | 95.11% | 92.74% | ▼ 2.37 | $0.15 | $1.77 | +1094 % |
| Personal data detection | 99.90% | 100.00% | ▲ 0.1 | $0.10 | $1.38 | +1258 % |
| Contract clauses | 94.41% | 97.13% | ▲ 2.72 | $0.22 | $2.76 | +1154 % |
| Document summaries | 100.00% | 100.00% | ±0 | $0.16 | $1.98 | +1115 % |
| Questions about a document | 100.00% | 100.00% | ±0 | $0.18 | $2.13 | +1074 % |
| Accuracy (correct fields) | Per 1,000 documents | |||||
|---|---|---|---|---|---|---|
| Task · documents | Gemma 4 31B | Mistral Medium 3.5 | Difference | Gemma 4 31B | Mistral Medium 3.5 | Difference |
| Invoice extraction | 94.62% | 96.87% | ▲ 2.25 | $0.20 | $4.75 | +2249 % |
| Email triage | 97.11% | 94.15% | ▼ 2.96 | $0.15 | $1.79 | +1127 % |
| Personal data detection | 99.90% | 99.62% | ▼ 0.28 | $0.10 | $1.41 | +1260 % |
| Contract clauses | 94.41% | 100.00% | ▲ 5.59 | $0.24 | $3.02 | +1172 % |
| Document summaries | 99.67% | 100.00% | ▲ 0.33 | $0.16 | $1.86 | +1079 % |
| Questions about a document | 100.00% | 100.00% | ±0 | $0.18 | $2.18 | +1102 % |
| Accuracy (correct fields) | Per 1,000 documents | |||||
|---|---|---|---|---|---|---|
| Task · documents | Gemma 4 31B | Mistral Medium 3.5 | Difference | Gemma 4 31B | Mistral Medium 3.5 | Difference |
| Invoice extraction | 93.54% | 98.56% | ▲ 5.02 | $0.22 | $4.82 | +2084 % |
| Email triage | 96.22% | 94.15% | ▼ 2.07 | $0.16 | $1.78 | +1016 % |
| Personal data detection | 99.81% | 99.62% | ▼ 0.19 | $0.12 | $1.43 | +1095 % |
| Contract clauses | 94.62% | 100.00% | ▲ 5.38 | $0.26 | $3.01 | +1077 % |
| Document summaries | 99.11% | 100.00% | ▲ 0.89 | $0.16 | $1.87 | +1037 % |
| Questions about a document | 100.00% | 100.00% | ±0 | $0.19 | $2.19 | +1039 % |
Difference: Mistral Medium 3.5 vs. Gemma 4 31B; accuracy in percentage points, price in percent. The better value is in bold. Click a task to see the details.
Invoice extraction · English (US) (150 documents)
Mistral Medium 3.5 is more accurate by 0.1 pp. Gemma 4 31B is 26.1× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 97.64% | 97.76% |
| Error-free documents | 81.33% | 79.33% |
| Per 1,000 documents | $0.17 | $4.37 |
| To fix / 1,000 | 187 | 207 |
| Speed (median) | 8.9 sec | 2.3 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Invoice extraction · Czech (150 documents)
Gemma 4 31B is more accurate by 2.6 pp. Gemma 4 31B is 22.1× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 96.10% | 93.49% |
| Error-free documents | 66.67% | 63.33% |
| Per 1,000 documents | $0.21 | $4.69 |
| To fix / 1,000 | 333 | 367 |
| Speed (median) | 10.4 sec | 2.6 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Invoice extraction · German (150 documents)
Mistral Medium 3.5 is more accurate by 2.5 pp. Gemma 4 31B is 23.7× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 94.87% | 97.38% |
| Error-free documents | 64.00% | 74.67% |
| Per 1,000 documents | $0.20 | $4.72 |
| To fix / 1,000 | 360 | 253 |
| Speed (median) | 7.7 sec | 2.5 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Invoice extraction · Spanish (150 documents)
Mistral Medium 3.5 is more accurate by 2.2 pp. Gemma 4 31B is 23.5× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 94.62% | 96.87% |
| Error-free documents | 50.67% | 66.67% |
| Per 1,000 documents | $0.20 | $4.75 |
| To fix / 1,000 | 493 | 333 |
| Speed (median) | 7.4 sec | 2.5 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Invoice extraction · French (150 documents)
Mistral Medium 3.5 is more accurate by 5 pp. Gemma 4 31B is 21.8× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 93.54% | 98.56% |
| Error-free documents | 50.67% | 84.67% |
| Per 1,000 documents | $0.22 | $4.82 |
| To fix / 1,000 | 493 | 153 |
| Speed (median) | 6.5 sec | 2.6 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Email triage · English (US) (150 documents)
Gemma 4 31B is more accurate by 1.6 pp. Gemma 4 31B is 12.2× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 96.52% | 94.89% |
| Error-free documents | 72.00% | 57.33% |
| Per 1,000 documents | $0.12 | $1.47 |
| To fix / 1,000 | 280 | 427 |
| Speed (median) | 2.2 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Email triage · Czech (150 documents)
Gemma 4 31B is more accurate by 2.6 pp. Gemma 4 31B is 12.3× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 97.11% | 94.52% |
| Error-free documents | 74.67% | 55.33% |
| Per 1,000 documents | $0.16 | $1.95 |
| To fix / 1,000 | 253 | 447 |
| Speed (median) | 2.5 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Email triage · German (150 documents)
Gemma 4 31B is more accurate by 2.4 pp. Gemma 4 31B is 11.9× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 95.11% | 92.74% |
| Error-free documents | 62.00% | 48.00% |
| Per 1,000 documents | $0.15 | $1.77 |
| To fix / 1,000 | 380 | 520 |
| Speed (median) | 3.6 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Email triage · Spanish (150 documents)
Gemma 4 31B is more accurate by 3 pp. Gemma 4 31B is 12.3× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 97.11% | 94.15% |
| Error-free documents | 74.00% | 54.00% |
| Per 1,000 documents | $0.15 | $1.79 |
| To fix / 1,000 | 260 | 460 |
| Speed (median) | 2.2 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Email triage · French (150 documents)
Gemma 4 31B is more accurate by 2.1 pp. Gemma 4 31B is 11.2× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 96.22% | 94.15% |
| Error-free documents | 68.67% | 54.00% |
| Per 1,000 documents | $0.16 | $1.78 |
| To fix / 1,000 | 313 | 460 |
| Speed (median) | 2.4 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Personal data detection · English (US) (150 documents)
Gemma 4 31B is more accurate by 0.2 pp. Gemma 4 31B is 13.6× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.90% | 99.71% |
| Error-free documents | 99.33% | 98.00% |
| Per 1,000 documents | $0.08 | $1.15 |
| To fix / 1,000 | 7 | 20 |
| Speed (median) | 2.4 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Personal data detection · Czech (150 documents)
Both are equally accurate. Gemma 4 31B is 11.8× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.81% | 99.81% |
| Error-free documents | 98.67% | 98.67% |
| Per 1,000 documents | $0.12 | $1.40 |
| To fix / 1,000 | 13 | 13 |
| Speed (median) | 2.9 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Personal data detection · German (150 documents)
Mistral Medium 3.5 is more accurate by 0.1 pp. Gemma 4 31B is 13.6× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.90% | 100.00% |
| Error-free documents | 99.33% | 100.00% |
| Per 1,000 documents | $0.10 | $1.38 |
| To fix / 1,000 | 7 | 0 |
| Speed (median) | 1.9 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Personal data detection · Spanish (150 documents)
Gemma 4 31B is more accurate by 0.3 pp. Gemma 4 31B is 13.6× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.90% | 99.62% |
| Error-free documents | 99.33% | 97.33% |
| Per 1,000 documents | $0.10 | $1.41 |
| To fix / 1,000 | 7 | 27 |
| Speed (median) | 3.6 sec | 1.0 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Personal data detection · French (150 documents)
Gemma 4 31B is more accurate by 0.2 pp. Gemma 4 31B is 12× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.81% | 99.62% |
| Error-free documents | 98.67% | 97.33% |
| Per 1,000 documents | $0.12 | $1.43 |
| To fix / 1,000 | 13 | 27 |
| Speed (median) | 4.3 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Contract clauses · English (US) (150 documents)
Gemma 4 31B is more accurate by 1.5 pp. Gemma 4 31B is 13× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.64% | 98.10% |
| Error-free documents | 97.33% | 87.33% |
| Per 1,000 documents | $0.18 | $2.35 |
| To fix / 1,000 | 27 | 127 |
| Speed (median) | 2.5 sec | 1.1 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Contract clauses · Czech (150 documents)
Both are equally accurate. Gemma 4 31B is 12× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.95% | 100.00% |
| Error-free documents | 99.33% | 100.00% |
| Per 1,000 documents | $0.25 | $3.06 |
| To fix / 1,000 | 7 | 0 |
| Speed (median) | 6.0 sec | 1.2 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Contract clauses · German (150 documents)
Mistral Medium 3.5 is more accurate by 2.7 pp. Gemma 4 31B is 12.5× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 94.41% | 97.13% |
| Error-free documents | 27.33% | 75.33% |
| Per 1,000 documents | $0.22 | $2.76 |
| To fix / 1,000 | 727 | 247 |
| Speed (median) | 2.6 sec | 1.2 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Contract clauses · Spanish (150 documents)
Mistral Medium 3.5 is more accurate by 5.6 pp. Gemma 4 31B is 12.7× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 94.41% | 100.00% |
| Error-free documents | 27.33% | 100.00% |
| Per 1,000 documents | $0.24 | $3.02 |
| To fix / 1,000 | 727 | 0 |
| Speed (median) | 5.0 sec | 1.3 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Contract clauses · French (150 documents)
Mistral Medium 3.5 is more accurate by 5.4 pp. Gemma 4 31B is 11.8× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 94.62% | 100.00% |
| Error-free documents | 30.67% | 100.00% |
| Per 1,000 documents | $0.26 | $3.01 |
| To fix / 1,000 | 693 | 0 |
| Speed (median) | 5.3 sec | 1.3 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Document summaries · English (US) (150 documents)
Both are equally accurate. Gemma 4 31B is 12.3× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.13 | $1.57 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 2.2 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Document summaries · Czech (150 documents)
Both are equally accurate. Gemma 4 31B is 12.2× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.19 | $2.31 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 3.4 sec | 1.4 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Document summaries · German (150 documents)
Both are equally accurate. Gemma 4 31B is 12.1× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.16 | $1.98 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 2.9 sec | 1.1 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Document summaries · Spanish (150 documents)
Mistral Medium 3.5 is more accurate by 0.3 pp. Gemma 4 31B is 11.8× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.67% | 100.00% |
| Error-free documents | 98.00% | 100.00% |
| Per 1,000 documents | $0.16 | $1.86 |
| To fix / 1,000 | 20 | 0 |
| Speed (median) | 2.6 sec | 1.0 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Document summaries · French (150 documents)
Mistral Medium 3.5 is more accurate by 0.9 pp. Gemma 4 31B is 11.4× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 99.11% | 100.00% |
| Error-free documents | 94.67% | 100.00% |
| Per 1,000 documents | $0.16 | $1.87 |
| To fix / 1,000 | 53 | 0 |
| Speed (median) | 3.2 sec | 1.0 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Questions about a document · English (US) (150 documents)
Both are equally accurate. Gemma 4 31B is 11.7× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.15 | $1.81 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 2.1 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Questions about a document · Czech (150 documents)
Both are equally accurate. Gemma 4 31B is 11.9× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.21 | $2.51 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 1.9 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Questions about a document · German (150 documents)
Both are equally accurate. Gemma 4 31B is 11.7× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.18 | $2.13 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 2.1 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Questions about a document · Spanish (150 documents)
Both are equally accurate. Gemma 4 31B is 12× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.18 | $2.18 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 1.9 sec | 0.8 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.
Questions about a document · French (150 documents)
Both are equally accurate. Gemma 4 31B is 11.4× cheaper.
| Metric | Gemma 4 31B | Mistral Medium 3.5 |
|---|---|---|
| Correct fields | 100.00% | 100.00% |
| Error-free documents | 100.00% | 100.00% |
| Per 1,000 documents | $0.19 | $2.19 |
| To fix / 1,000 | 0 | 0 |
| Speed (median) | 2.1 sec | 0.9 sec |
| Valid answers | 100.00% | 100.00% |
Free with login
Accuracy field by field and the documents where the models differ.
Just your email, no password.