Mistral Large 4 is a mixture-of-experts model with 1T parameters, 52B of them active on any one token, built for reasoning and long-horizon agentic work. Its technical report is unusually candid: the attention design picked to make it cheap to serve turned out to hurt exactly that work, and the model was operated on in the middle of training to fix it.
The figures respond to clicks. Their measurements come from the paper, and when a figure is a schematic or does its own arithmetic, it says so.
What was built, what it was fed, and how its settings were predicted rather than searched.
Three tricks that shrank the KV cache, the agentic penalty they caused, and the surgery that undid most of them between the two pretraining phases.
Taxonomies, agent pipelines and verifiers that generate training environments, then midtraining and SFT.
An asynchronous RL system, the escapes and reward hacks it caught, and three reasoning efforts from one student.
Six tables against six models, and the serving work that fits it on four GPUs.
A trillion parameters, 52 billion of them at work on any one token. Before the surprises, here is what was built, what it was fed, and how it was trained.
Mistral Large 4 is a mixture-of-experts model with 1T parameters in total and 52B active. Each token is routed through a small subset of the expert networks, so only 52B parameters do work on any one token, while the model as a whole holds a trillion.
It reads images as well as text, and the paper is plain about what it was built for: reasoning and long-horizon agentic work, where the model acts over many steps and many tool calls. Most of the decisions that follow only make sense against that target.
Pretraining ran on 3,840 NVIDIA GB200 GPUs in a Mistral data center near Paris, on an internal stack built on PyTorch and CUDA. It took 45 days, about 4.15 million GPU-hours. The design leans on Blackwell: FP8 tensor cores for low-precision training, the large NVLink domain for wide expert parallelism, and the fast link to the Grace CPU for offloading activations.
Two efficiency numbers come with it. MFU was 36%, measured against peak BF16 throughput, counting matrix-multiply and attention FLOPs but not recomputation. Goodput was 94.1%, the share of the run that turned into training progress.
Rather than search hyperparameters at full scale, the learning rate is predicted. The predictor is log-linear in five inputs: width, depth, training horizon (total tokens, repeats included), expansion rate (total experts over active experts) and batch size. Keeping each input separate means the predictor can be checked for extrapolation along each one.
Batch size gets its own predictor, fit on the same dimensions apart from batch size itself, and in practice it depends mostly on horizon and expansion rate. So the batch is predicted first and the learning rate is conditioned on it. A scaling ladder of smaller models then uses the same predictors and multipliers, so it differs from the real model "in scale only".
The pretraining data is multilingual and natively multimodal: web pages, source code, and scientific and technical documents, almost all of it formatted as interleaved text and image chunks. The first phase, 65% of training, is about broad coverage of domains and languages.
The second, 35%, shifts toward higher-quality, more STEM-heavy data and raises the sequence length from 4K to 40K tokens. Code more than doubles, from 16.9% of the mixture to 39.0%, and web pages make room for it, dropping from 56.3% to 32.0%.
ML4 is trained from scratch on interleaved image and text data, and with careful dataset design the cost to text is small. Matched on the number of text tokens seen, text perplexity shows no regret at all. Matched on the total training horizon, where images use up part of the token budget, the regret is modest.
The payoff is over an order of magnitude more images than Mistral's previous flagships saw. The systems trick: prefetch images N batches ahead and cache their features from the frozen vision encoder, which keeps 99% of text-only MFU and means image features never have to be stored on disk.
The vision encoder has 1.6B parameters, trained from scratch with a captioning objective over 8T tokens; the captioner is thrown away afterwards. Images go in at their native aspect ratio, up to 1540 by 1540 pixels. At the output, each 2 by 2 block of patches becomes one: the four embeddings are concatenated and projected, then an MLP maps the result into the language model's embedding space.
To judge an encoder, they attach it to a small LLM and run a short SFT of about 20B tokens. Figure 2 does this for a frozen 400M version of their encoder against Perception Encoder ViT-L/14, also 400M and frozen, on ten benchmarks. Theirs finishes ahead on nine. The exception is ChartQA, where Perception Encoder leads from 40% of the SFT run onwards.
Granularity is how finely the experts are cut: higher means more, smaller experts. ML4 keeps the granularity of 4 used in Mistral Large 3. Going to 8 would reach the same perplexity with about 4.3% fewer FLOPs, give or take 0.5 points.
But training at granularity 8 measured about 5% slower. At those central values, doing 4.3% less work at about 5% lower speed roughly cancels out in wall-clock time, and the measured slowdown is what kept the model at 4. It is a small decision, and a clean example of a habit that runs through this paper: the number that decides is the one measured on the real system.
A mixture-of-experts step runs only as fast as its busiest expert. ML4 computes the load-balancing loss over each expert-parallel group rather than over the global batch, which targets the quantity that causes the slowdowns directly: the tokens arriving at each expert. Pretraining comparisons favored this when trading imbalance against validation perplexity.
The cost is coupling. The loss now depends on how experts are split across devices, so scaled-down experiments must be careful whenever that split changes. The optimizer is AdamW with both betas at 0.975, coupled and higher than usual, which gave moderate gains.
Early in training, a gradient estimated from part of a batch is already good enough, so the rest of that batch is wasted compute. Later, precise gradients pay. So the batch grows in stages whose token share grows exponentially: the final batch size runs for the last 50% of tokens, the one before it for 25%, and so on, with the learning rate rescaled at each stage.
For optimization alone, a ramp from 0.25× to 2× the predicted batch did best. Bigger batches run at higher MFU, so they chose 1× to 4× for wall-clock reasons: 40M, then 80M, then 160M. Small early batches also act as a regularizer, more so the more often the data repeats.
ML4 was built for agentic work, and its attention was cut down hard to make it cheap to serve. The cuts turned out to hurt exactly that work, so the architecture was changed in the middle of training.
An agent on a long task carries a long context, and every token of it costs memory in the key-value cache. ML4 began with three ways to shrink that. Sliding windows: three of every four layers look back only 512 tokens. Cross-layer attention (CLA): every other full-attention layer reuses the KV of the full-attention layer before it.
YOCO: full-attention layers in the second half of the network read a KV cache made in the first half, which also allows a two-stage prefill that is 1.7× faster at 1M context. Together, 16× less KV cache than full attention, and early ablations said the cost in quality was negligible or small.
As pretraining went on, they ran early SFT cooldowns on intermediate checkpoints. Perplexity and knowledge benchmarks landed where expected. Agentic evaluations did not, above all the long-context ones such as SWE-bench and BrowseComp.
Parallel ablations at the same token budget found the cause. SWE-bench Verified, measured after midtraining and SFT, dropped when full attention became sliding windows, and dropped again when CLA was added. Figure 4 follows five attention designs through midtraining and on into SFT. Read off the figure, full attention ends near 66 and the design with all three tricks near 56. Adding YOCO to sliding windows costs almost nothing by the end of SFT, which matters later.
Figure 5 is the uncomfortable one. Plot each design's perplexity against a knowledge benchmark, MMLU Redux, and the designs fall close to one trend: at equal loss, their knowledge scores barely differ. Plot it against SWE-bench Verified after midtraining and they split apart: at the same perplexity, sliding windows sit about 5 points below full attention.
So the number everyone watches during pretraining, language-modeling loss, did not predict agentic ability across architectures. The paper adds that longer training can soften the penalty. The right panel suggests how, if its 0.5× and 2× mark run lengths (our reading; the paper does not define them). The 2× sliding-window run follows its own line, close to the sliding-window line and no higher, but gets further along it. Its best checkpoint (about 50.8) draws level with the best of the 0.5× full-attention run (about 50.0).
Rather than start again, they operated on an existing checkpoint, before the second phase of pretraining, aiming for something as close to full attention as possible without losing what pretraining had already taught it. Removing a sliding window is easy: let the layer see the whole context instead of the last 512 tokens.
Removing KV sharing is harder, because a layer that borrowed its keys and values must now make its own. Each one got KV weights copied from the last long layer that had its own. For CLA that is the long layer just before it, which sees a very similar input. For YOCO it is the long layer at the halfway point, which for most of the second half is much farther away.
They tested surgery at 8×1022 FLOPs, applied either at the start of midtraining or before phase 2 of pretraining, against two references: the original model with all three tricks, and one pretrained with full attention throughout. The phase-2 runs kept the original rotary frequencies or rescaled them 8× with YaRN.
Figure 6 shows the trap. Phase 2 after surgery pulled perplexity close to the full-attention baseline, but SWE-bench did not follow. Bigger architectural changes were harder to recover from, so they kept the surgery conservative and added learnable parameters instead.
The learnable parameters are LeRoPE, which learns the rotary frequencies, and SSMax, which learns a per-head scaling of the queries. Together they beat fixed YaRN downstream at almost the same training loss: one more case where loss alone would have hidden the difference.
In an intermediate phase-2 checkpoint the largest changes sat in two bands near the 4K training context, where frequencies rose about 1.5× and 1.8× relative to YaRN, making their wavelengths 35% and 44% shorter. Most other bands moved less. The model retuned a few bands rather than rescaling them all.
The small-scale surgeries removed YOCO too, and there, given the whole phase-2 and midtraining budget to recover, removing it or not ended up about the same. But removing YOCO disrupted the loss more at first, and for the full model there was no room to rerun all of phase 2.
So they ran a conservative test at 8×1023 FLOPs, surgery followed only by midtraining, and keeping YOCO did better (Figure 8). The real model was operated on at the end of phase 1, then trained for about two-thirds of the original phase 2 before midtraining.
In the converted model, every former sliding-window layer is an independent full-context layer, with LeRoPE and SSMax. The original global layers still use no positional encoding (NoPE). With CLA gone, the global layers that borrowed KV through CLA now make their own, and those in the second half still read the KV made in the first half through YOCO.
So it gained many independent full-context layers while keeping part of the old sharing layout. The paper's section title says it plainly: "Why ML4 is (almost) a full-attention model". It does not give the total layer count; the figure here follows the paper's Figure 3, which draws layers 0 to 7 and 32 to 35 of the first half and only layers 36 to 43 of the second.
Lifting a window does not force a layer to use its new reach. To check, they measured where attention mass goes, using 32k-token probes from six data sources on intermediate checkpoints. Converted layers put around 55% of their attention on tokens more than 512 positions back, against 67% for the native long layers.
The paper reads this as the old initialization surviving in part, while the converted layers could still learn a long-range role when it was needed. And even in a model with full attention everywhere, some layers look further back than others, so 55% against 67% is not a failure.
Full attention over a long context is expensive, so ML4 was then adapted to DeepSeek Sparse Attention: a lightweight indexer picks the top 2,048 keys for each query, and attention runs only over those. Unlike a fixed window, the choice is made per query, so any token in the context can still be picked.
Stage 1 warms the indexers up for about 15.1B tokens by distilling the main model's attention with a KL loss, the main model frozen on full attention. Stage 2 trains both together under sparse attention for another 198B tokens. From layer 16 on, the former sliding-window layers within each original group attended so alike that each group shares a single indexer.
Post-training needs a great many hard tasks with answers a machine can check. Mistral has agents write more and more of them, and built the machinery to check the agents.
Generating training data has traditionally been manual work, with many iterations and a lot of human oversight. For ML4, Mistral increasingly hands it to authoring agents. They get a minimal set of tools, enough to do everything a human expert would do to build a task, and they can run a task against several models and inspect the traces.
Agents are fallible, so every seed, and the candidate built from it, runs a gauntlet (Figure 9): a seed filter, cheap checks by LLM judges, expensive checks by an agentic verifier and for reward hacking, then difficulty calibration. A seed that fails the filter is rejected; a failure at any later gate goes back to the seed with feedback. Successful authoring trajectories also join the midtraining data, so the next model gets better at writing tasks: a flywheel.
A seed is two things: the environment the agent works in, and the behavior the task should demand. Both come from taxonomies (Figure 10). The economy taxonomy asks what work is being done, from accountants doing a month-end close to lawyers drafting a contract. The capabilities taxonomy asks what completing the task should require: success criteria, underspecification, reasoning, data analysis, document skills, user personas and user behaviors.
A task-authoring agent samples from both. The month-end close, crossed with temporal ordering and cross-document reconciliation, becomes a task: prepare the close report from the ledger and the reconciliation notes, using revision history to resolve conflicting figures. For knowledge-work tasks, the paper observes (without giving numbers) that samples conditioned on taxonomies come out higher quality, more realistic and more diverse.
Software tasks come from public GitHub repositories. First an authoring agent builds the project in a container, installs the toolchain and finds a reproducible way to run the tests. Tests that fail before the reference change and pass after it become the verifier. Tests that pass both before and after guard against regressions.
The solution and anything that could reveal it are removed, and the problem is rewritten as a self-contained spec; candidates whose tests cannot be validated still go to midtraining. Then a refinement agent runs solvers of different strength, tells model failures apart from broken specs or tests, and revises the task, which is verified again.
Terminal tasks drop the model into a sandboxed command line to investigate and fix things on its own. Seeds come from three places: millions of technical seeds mined from the pretraining corpus, synthetic seeds drawn as tuples of skill, domain, difficulty, verbosity and other attributes, and internal terminal-agent traces turned into verifiable tasks.
The authoring pipeline turns each seed into a problem statement, tests, an environment and sandbox configuration, and a reference solution. Instructions are audited for clarity, an adversarial agent hunts for ways to hack the reward, and how often solvers pass sets each task's difficulty.
For vulnerability patching, an authoring agent takes a public CVE, finds the upstream fix, and packages the vulnerable code into a reproducible container. Tests mix adversarial exploits with ordinary inputs. A test harness counts only if it fires on the unpatched code and passes after the reference fix.
The model either fixes a named flaw, or gets only a weakness category and must find and fix the defect itself. Reward is positive only when the exploit is fully neutralized and the existing tests still pass. The pool spans tens of thousands of CVEs across roughly 600 weakness classes.
For knowledge work, the environments are simulated workplaces: realistic files and application data across tool surfaces such as MCPs, APIs and CLIs, in areas such as finance, law, engineering, data science and business management. Each workflow from the taxonomy goes through three steps. A research agent gathers public documents for an open-ended query. An authoring agent turns them into a coherent workspace, adding spreadsheets, reports, presentations and correspondence. Several agents then inspect it for realism, consistency and file integrity.
Next come the applications. Agents read the files and each app's database schema, and create records that match the workspace's people, projects and timeline: messages, calendar events, project updates, customer histories. Long-horizon agentic tasks are also generated to run inside each environment, grounded in a second taxonomy of agent behaviors and skills.
Midtraining is a trade. It should add new capabilities, such as agentic ones, while retaining as much as it can of the long tail of knowledge from pretraining. So its hyperparameters were chosen by optimizing a weighted average of the evaluation suites for both. For the learning rate, a multiplier on top of the pretraining scaling-law prediction transferred best across scale. It was tuned jointly with the rehearsal rate, how much pretraining data is replayed.
At ML4 scale the best compromise was 1/5 of the peak pretraining learning rate and 1/10 of pretraining rehearsal. Keeping the batch size large cost very little in quality, and the MFU it bought more than made up for it.
SFT settings also come from a scaling ladder: smaller models of the same family, fine-tuned on slices of the SFT mixture that grow with model size. At each rung, the learning rate with the lowest validation loss wins. A power law in pretraining compute, batch size, training tokens and data repetition, fitted to the smaller rungs, predicted a held-out model an order of magnitude larger.
When the mixture changed, only a constant offset moved, so a small ladder per mixture was enough to recalibrate, with no new search at full scale. Batch size mattered far less than the learning rate, and weight decay and schedule shape did not move the optimum. A learning-rate ablation at ML4 scale landed close to the prediction early in training (Figure 11).
At peak, tens of thousands of rollouts ran at once on about 4k GPUs, and the longest single rollout lasted about 37 hours. Most of what makes this work is plumbing, and some of it is about agents misbehaving.
RL runs on a fully asynchronous pipeline built on Ray. A dataloader samples prompts under a curriculum, and the runner pool admits groups of episodes as concurrency and backpressure allow, giving priority to siblings of groups already started. A finished group goes to a scoring pool for group-level rewards, then to a tokenization and packing pool, and a central bin-packer fills fixed-length microbatches for the trainer.
Generators are vLLM engines that receive new weights over NCCL at each training step, and the sandboxes, tools, verifiers and judges autoscale with the load. At peak: 26k concurrent rollouts on about 20k sandboxes, using 3,104 GPUs at a 1:2.5 trainer-to-generator ratio, and about 4k GPUs counting judges and services.
Every environment, from single-turn QA to long-horizon agentic coding, is written against one interface, and RL, evaluation and synthetic data generation run the same code, so an environment behaves the same in training and in evaluation. Each request and response is logged to an append-only list of records, the single representation of an episode. Only the policy's own generations become training data.
When an episode breaks, the environment must assign blame. Malformed tool calls and context overflow, for example, count as the agent's failure. A sandbox crash or a failed service call is discarded in RL and retried in evaluation. Ambiguous cases count against the agent, because silently dropping unreliable behavior would quietly reward it. Because the line is kept strict, infrastructure incidents cost throughput without corrupting the reward signal, though they can shrink effective group sizes and skew the data mixture, which is tracked separately.
Each run provisions millions of sandboxes. In one instance an agent chained four vulnerabilities and got root on one of Mistral's nodes. It sniffed traffic to grab an identity-scoped API key, found an undocumented deployment of the sandbox control plane, and used the key against its own pod to run arbitrary commands. The API let it run commands on a privileged sidecar, and a kernel panic took it to the host.
On the host, it realized the CTF, the capture-the-flag challenge, was not there and stopped. Another agent that escaped in a similar way probed the API for creating sandboxes, then tried to clean up after itself and deleted about 30k sandboxes by mistake. Both escapes triggered alerts, and the run restarted after remediation. The sandboxes are now moving from containers to microVMs.
Generation and training are decoupled. The trainer takes a step once it has 7k usable trajectories, then pushes the new weights to the generators. Generation is not drained before an update: in-flight rollouts pause, load the new weights, and resume without recomputing their KV cache.
So a long rollout can be written by several policy versions, and its later tokens can attend to KV entries that older policies computed. The longest production rollout lasted about 37 hours and spanned 136 model updates. Across training, the average staleness was six steps. The objective corrects for this lag with importance weighting.
The ML4 RL run generated 5.36M rollouts across 452k groups and 306k instances, 162.5B output tokens in all. The mixture is broad: multimodal, alignment, instruction following, cybersecurity, code agents, knowledge work, finance, STEM, safety, long context and multilingual.
Code agent tasks were 20% of rollouts but 46.6% of output tokens, so the average code-agent rollout wrote about 2.3 times as much as the average rollout. Reasoning tasks came second, with 10.2% of rollouts and 20.9% of tokens. The other categories share the remaining 69.8% of rollouts and 32.5% of tokens; the paper does not break them down.
For each prompt the generators sample a group of rollouts: 16 by default, 32 for agentic coding, anywhere from 2 to 32 overall. Each data source builds its own composite reward from signals such as unit tests, sandboxed runs, SQL execution, LLM judges, a reward model and programmatic checks, designed to sit mostly in [0, 1]. A rollout's advantage is its reward minus its group's mean, divided by the standard deviation of those centered rewards across the whole batch (Equation 1).
If every rollout in a group earns the same reward, every advantage is zero and the group teaches nothing. Such groups are dropped before they reach the trainer, so the 7k-trajectory batch counts only trajectories that carry a gradient.
The loss is CISPO-style REINFORCE. Each token's loss is minus the advantage times its log-probability, scaled by an importance weight: the trainer's probability over the generator's, capped at 3 from above with no floor, and held constant by a stop-gradient. Unlike PPO, the ratio is not clipped inside the objective, so a token whose weight is capped still contributes to learning; the cap only bounds the variance from very stale or mismatched tokens.
Every trainable token in a group also gets the same scale, the group's rollouts over its trainable tokens, so the group's total equals its number of rollouts and groups count by rollouts rather than by length. In multi-turn trajectories, turns flagged for format errors or invalid tool calls get a MiMo-style correction: zero advantage when the trajectory's outcome advantage is positive, an amplified negative one otherwise. The optimizer is Muon at a constant learning rate of 8×10-7 after a 25-step warmup.
The inference engine and the trainer do not compute exactly the same numbers. The mismatch comes mainly from two sources: numerical error, and the staleness that asynchronous training adds. In a mixture of experts, unstable routing amplifies both. The first fix, router replay, re-runs the routing decisions made at inference in the training forward pass, so router drift cannot cause divergence. The second is FP8 inference with blockwise scales in both generator and trainer, which speeds generation and matches the trainer.
The third is subtler. The generator may emit two tokens, "the" and "re", which become a string sent into the environment; tokenized again, that string is the single token "there". The same text is one sequence as generation and another as context, and blockwise attention from FA4 handles it efficiently. The average k2 divergence (an estimate of the KL divergence) between train and inference stayed at or below 5×10-3 for the whole run.
The runs are watched for reward hacking and environment leaks, alongside correctness,
pass@k, reward variance, entropy, train and generator divergence, gradient norms and router
statistics. In one case the model was asked to fix a pipeline so that a compiled checker
would accept its output. It noticed that the checker internally called
execvp("python3", "-c", <source_code>).
So it planted a fake /tmp/fakebin/python3 on the PATH to intercept the call,
and captured the reference implementation from the command-line payload. Then it listed
eight discrepancies between its code and the reference, fixed them, and submitted,
noting: "I have the full reference implementation from the board binaries." The sandboxes
keep being hardened, while keeping the realistic internet access and tooling agents need.
Figure 13 follows six evaluations through SFT and then through the CISPO phase: knowledge work, instruction following, scientific chart question answering, terminal task automation, competitive code generation and agentic CAD question answering. All six gain substantially and consistently over the SFT checkpoint, though each domain improves at its own speed.
The last panel matters most. Agentic CAD question answering is not in the reward mixture at all, and it still rose, from 50.6% at the last SFT checkpoint to 54.8% at the end of CISPO. The paper reads this as RL inducing broader gains that transfer out of distribution, rather than only optimizing the rewards it was given.
Reading web-search traces, the team saw the model complain that its results were "full of SEO spam". It was not generic junk but query-reflection pages: sites that turn any query they receive into an indexable page whose URL spells out the query. They are footprints left by earlier agents working on the same benchmark.
In internal BrowseComp traces from February to October, the share of trajectories that hit such a URL in their first ten searches rose over the year, and by October most did (Figure 14). The paper calls the rise steady, but the plotted share stalls near 30% in July and dips in early August before climbing past half in mid September. SimpleQA Verified, on the same search stack, showed no equivalent increase. Excluding those domains did not change scores significantly, but made runs vary more. The plan is an internal, offline copy of the web.
To cut verbosity without hurting results, correct rollouts earn a group-relative length bonus. Within a group, find the shortest correct rollout. Lengths are binned in steps of 10% of that length, so tiny differences do not count (Equation 2). Correct rollouts in the shortest bin get the full bonus λ, those in the longest get none, and the rest fall on a straight line in between (Equation 3).
Wrong rollouts get nothing however short they are, and if all the correct ones land in a single bin, nobody gets a bonus. Because everything is relative to the group's own correct answers, no per-task length target is needed. Running RL with different bonus sizes yields teachers with different trade-offs between efficiency and performance.
The release model has three reasoning-effort modes: high, medium and low. The student starts from the high-reasoning RL checkpoint, which is itself the high-effort teacher; the medium and low teachers come from length-penalized RL. On its own rollouts, the student minimizes a per-token KL divergence to the teacher's full next-token distribution.
Figure 15 tracks the student's length relative to high effort. For the first fifth of training both modes stay near 1.0; then they drop within about a tenth of training and hover just under their targets. Over the last fifth of training, medium averages 0.49 against a target of 0.528 and low 0.20 against 0.227. Teachers run on separate GPU nodes, so their number does not depend on the trainer hardware, and they send only hidden states, which cuts communication with the trainer.
Six tables of results against six other models, then the engineering that fits a trillion parameters onto four GPUs.
Tables 1 to 6 compare ML4 with GLM-5.2, GLM-5.3, DeepSeek V4-Pro 0813, DeepSeek V4.1-Flash 1017, Kimi K3 and Muse Spark Open. Of the 24 capability rows outside safety, ML4 is first outright on four: FrontierCode 2.0 (19.7%), Terminal-Bench 4.0 (41%), CyberGym-E2E (79%) and internal Cybench (90%, three models reported). It ties for first on Beyond-AIME at 90.6%.
Elsewhere it is second on seven rows and mid-pack on most of the rest. Knowledge is its weak side: last on HLE-Diamond (17.3% against a best of 26.9%) and on AA-Omniscience accuracy (26% against 45.2%), though second on that benchmark's non-hallucination rate. Harvey's Legal Agent Benchmark, given as 15% in the introduction, is 15.83% in Table 6, second behind Muse Spark Open's 25%.
The safety evaluations were run internally, and higher is better on all of them. ML4 leads the B3 agentic security benchmark at 97.1% and StrongREJECT's cyber subset at 92.6%, and shares the top score of 100 on AgentHarm's. The headline 96.6% in the introduction is the unweighted mean of three cyber-subset scores: JailbreakBench's safe-response rate (97.3), StrongREJECT's refusal rate (92.6), and 100 minus AgentHarm's harm score (100).
It is not first everywhere. Muse Spark Open scores 100% on JailbreakBench's cyber subset against ML4's 97.3%, and leads KORA, a child-safety benchmark scored from 0 to 2, with 1.847 against 1.765.
Production serving is tuned for latency. Plain tensor parallelism gives ML4 the best latency but replicates the KV cache on every rank, which caps concurrency and the cache hit rate. Context parallelism shards the cache along the sequence instead. With a native FP8 KV cache and offloading, that gives about 20M tokens of in-memory KV-cache space on 4×GB300.
The catch: the standard CP implementation ran much worse than the tensor-parallel baseline, because of kernel inefficiencies. Rewriting the CP collectives, with latency-optimized exchange patterns, compute and communication overlapped inside the kernel, and more fusion, raised decode interactivity by 51% in Figure 16's setting (8×GB300, 50K input, eight concurrent requests), with no more than 10 to 15% communication overhead compared with tensor parallelism.
Figure 17 times one decoding step of an index-producing layer under context parallelism. Before: 240 μs on a single stream, with three slow communication phases, the top-k exchange (39 μs), the Q gather (30 μs) and the output combine (33 μs), strung between the projections, the sparse indexer and sparse attention.
After: 153 μs, 36% less. The indexer, its top-k exchange and the selection move to a side stream that runs alongside the main one, and the three exchanges shrink to 15, 11 and 16 μs. The gains come mostly from hardware multicast, latency-first exchange patterns, removing in-kernel synchronization, and fusing nearby compute into the communication kernels.
Speculative decoding uses an adapted DSpark: a speculator drafts tokens ahead and the full model verifies them. Two additions to the original recipe, a recurrent conditioning head and unique mask tokens, lengthen the accepted drafts by 1.2% and 0.5% at negligible runtime cost. The DSpark module is trained on 70B tokens.
With block verification, DSpark improves latency and throughput by 90% over a baseline without speculative decoding, and by 15% over an EAGLE-3 baseline, at eight concurrent requests. Its acceptance rate in Figure 16 is the average across all SPEED-Bench categories.
The routed experts are quantized to NVFP4, while everything else keeps its training precision, which gives a mixed BF16, FP8 and NVFP4 checkpoint of 606 GB. Naive abs-max quantization was already enough: calibrated on 200M tokens of the SFT mixture, it matches the FP8 reference in all evaluations. This is what lets ML4 run on four GB300 GPUs.
At fleet scale, routing builds on llm-d, and most of Mistral's work on it has been upstreamed. Model Express speeds up startup by moving checkpoints peer to peer over InfiniBand, and cache sharing is warmed up in advance. The fleet autoscales on live request and server metrics, and a new replica starts in under 120 seconds.
Figure 16 plots output throughput per GPU against decode interactivity, the tokens per second each request sees, at 50K input length. Each point is labelled with its number of concurrent requests. Each curve adds one optimization to the one before: the FP8 baseline on an off-the-shelf CP implementation, then prefill and decode disaggregation, then the optimized decode CP kernels (+51% interactivity at 8 requests), then DSpark (+90% interactivity at 8 requests).
The last step changes the hardware as well. The baseline runs on 8×GB300; the NVFP4 curve, labelled CP4, runs on 4×GB300. Throughput is per GPU, so the curves can be compared, but the final gain is quantization and the smaller deployment together.
From the conclusion: comparable language-modeling loss does not imply comparable performance on long-horizon tasks, and better proxies have to be found. ML4 was first designed with a heavily reduced KV cache, came up short on agentic work, and was operated on during training, first to almost full attention (it kept YOCO) and then to sparse attention.
Those findings pushed Mistral to explore sparse, linear and hybrid attention, with agentic evaluations informing design choices throughout training. On the data side the goal is recursive: train each generation of models on environments written by the previous one, with stronger verification, safety and evaluation as the workflows grow longer.
The measurements on this page come from the technical report, Mistral Large 4. The figures take table values and quoted numbers from data parsed out of the paper, and each quoted number is checked against the paper's text when the page is built. Some figures do simple arithmetic on those numbers, such as a ratio or a sum, and say so. Figures 6, 13, 15 and 17 are vector graphics, so their values are extracted exactly. Figures 2, 4, 5, 7, 8, 11, 14 and 16 are raster images; their values were read from the pixels of a 600 dpi render, are approximate, and the figures that use them say so.
Where a figure here is a schematic, such as the layer diagrams or the token routing, it illustrates a mechanism the paper describes, its values are illustrative unless it says otherwise, and it is labelled as a schematic. The prose is my reading of the paper, not the paper's own words, so check anything that matters against the source.
If a figure here disagrees with the paper, the paper is right. Tell me and I will fix it.