Avinash Sooriyarachchi · Research Scientist, Mistral AI

§01Current work

Robotics foundation models, and the simulation they train and get tested in

Most of my time goes to vision-language-action models, and to the System-2 planner sitting above them that has to break a long instruction into steps a policy can execute. Neither has the high-quality training data it needs in the form it's needed, so I build a lot of that too.

A lot of it comes down to real-to-sim. I build pipelines that take a single real camera frame and rebuild it as a physically valid simulated scene, and those scenes get used two ways. As a post-training environment, a policy can collect experience that would be slow or expensive to gather on real hardware. As an eval harness, you can run a policy a hundred times and score it, instead of filming one good demo and calling it a result.

Robotics isn't the only thing I work on here. I was a lead contributor on Shieldstral, a multimodal safety classifier, and on Mistral Small Creative. Before any of it I worked on NLP, recommender systems and multimodal models, going back to 2017.

§02Publications

2026 Shieldstral arXiv preprint · Core author A 3B multimodal safety classifier that matches or beats models around seven times its size on text safety, and sets state of the art on multimodal. Instead of scoring against a fixed taxonomy, it asks whether content violates the policy you hand it. Paper
2026 Robostral Navigate arXiv preprint · Core author An 8B vision-language navigation model that works from a single RGB camera. It picks waypoints by pointing at the next target in the current view, so everything happens in image space instead of a robot-specific action space. Paper
2026 Voxtral TTS arXiv preprint · Mistral AI Expressive multilingual text to speech. It clones a voice from as little as three seconds of reference audio, using autoregressive semantic tokens plus flow matching for the acoustics. Paper
2026 Voxtral Realtime arXiv preprint · Mistral AI Streaming speech recognition that holds offline transcription quality at sub-second latency. Trained end to end for streaming, with explicit alignment between the audio and text streams. Paper
2026 Ministral 3 arXiv preprint · Mistral AI Dense models at 3B, 8B and 14B. Each one ships as a base, an instruction-tuned and a reasoning variant, all of them with image understanding, all Apache 2.0. Paper
2016 Fiber Based Approaches as Medicine Delivery Systems ACS Biomaterials Science & Engineering 2(9) · Co-first author A review of micro- and nanofibers as drug carriers, covering electrospinning, microfluidic, wet-spinning, rotary and self-assembly fabrication. This came out of my mechanical engineering research at Iowa State, and it's still my most cited paper. Paper

Mistral ships models as collectively authored technical reports. I've marked the ones where I'm in the core author block.

§03Open source

makeMoE ★ 812

A sparse mixture-of-experts language model I wrote from scratch in pure PyTorch, in the spirit of Karpathy's makemore. The expert modules are the easy part. Most of the work went into the noisy top-k gating and the initialization, which are what decide whether an MoE trains at all. Meant as a readable reference for the architecture behind Mixtral, DBRX and Grok.

The NeurIPS 2024 tutorial on dynamic sparsity builds its mixture-of-experts notebook on this, says so in the notebook, and trains on data pulled straight from the repo. The M2L Summer School 2024 lists it as assigned reading.

Repository Write-up NeurIPS 2024 tutorial M2L Summer School Hacker News

seemore ★ 260

Same idea as makeMoE, pointed at vision. A ViT image encoder feeds a multimodal projection module and a character-level autoregressive decoder, wired end to end so you can step through the whole thing. A follow-up, seeMoE, swaps the decoder for a mixture-of-experts version.

Repository seemore write-up seeMoE write-up

Tuning the Finetuning ★ 46

An empirical look at what matters when fine-tuning with QLoRA. The short version is that you should target every linear layer, not just the attention blocks, and that a modest rank is enough.

Repository LoRA guide

multimodal_LLMs ★ 12

Fine-tuning BLIP-2 and Flan-T5-XL for caption generation with PEFT/LoRA. An early look at parameter-efficient adaptation of multimodal models.

BrickPilot ★ 10

Build your own AI coding assistant, wiring a self-hosted code model into VS Code alongside GitHub Copilot.

Repository Write-up

Finetuning-Guide ★ 10

Fine-tuning OpenLLaMA models on instruction-following datasets with QLoRA, end to end.

text-to-insights ★ 4

Going from natural language to SQL to an actual answer, using language models, vector search and resource-efficient fine-tuning.

Explainable-ML-Application ★ 3

Interactive explainable machine learning, as a containerized Streamlit app for poking at model behavior.

Repository Write-up

§04Writing

2026 Mistral Large 4, taken apart A visual breakdown of the Mistral Large 4 paper. Forty-eight scenes you scroll through: the pretraining recipe, the surgery that rebuilt its attention in the middle of training, agents that write their own training environments, RL across millions of sandboxes, serving, and the results, including where it trails. avisoori1x
2026 The Dreaming Robot A twenty minute film on what a world model actually is, meaning a program that answers what happens next if I do this, without doing it. Built on a real arm in simulation with a really trained model. YouTube
2026 Robostral Navigate, taken apart A visual breakdown of the Robostral Navigate paper, a navigation model I worked on. It drives from a single RGB camera by pointing at a pixel. Thirty scenes you scroll through, several of them live 3D, so you can move the camera and watch the image coordinate move with it. avisoori1x
2026 Shieldstral, taken apart A visual breakdown of the Shieldstral paper, a moderation model I worked on. Thirty-four scenes you scroll through: moderation reframed as one yes/no question, the data pipeline behind it, the training and the merge, and the results including the ones that are less flattering. avisoori1x
2024 Common Sense Product Recommendations using Large Language Models with Sam Sawyer, Colton Peltier, Bryan SmithUses a language model's world knowledge to suggest complementary products, then grounds those suggestions against a retailer's real catalog with embeddings and vector search. A collaborative filtering system can't get there on its own. Databricks
2023 Harnessing Machine Learning for Anomaly Detection in the Building Products Industry with Joy Garnett, Sathish GangichettyDimensionality reduction plus anomaly detection over manufacturing sensor data, wrapped in enough MLOps to retrain and redeploy continuously. Databricks
2023 Getting Up to Speed on Real-Time Machine Learning with Spark and SBERT Sentence-transformer inference inside a streaming pipeline, and where the latency really comes from. ODSC
2022 Near Real-Time Anomaly Detection with Delta Live Tables and Databricks Machine Learning with Sathish GangichettyEmbedding a trained isolation forest inside a streaming ETL pipeline to catch fraudulent card transactions in near real time. Databricks
2022 Building a Similarity-based Image Recommendation System for e-Commerce A contrastive-learning image similarity model for product recommendation, trained distributed across GPU clusters and served end to end. Databricks

6 more

2023 Even Santa Claus has AI fever with Alex Miller, Kunal Marwah, Mikaila GarfinkelSynthetic data generation and named-entity recognition, pointed at extracting gift requests from letters. A Christmas post. Databricks
2022 There's No AI in Pumpkin Pie, But There Should Be: Delivering Novel Application Experiences Using ML & AI with Nicole Jingting Lu, Bryan Smith, Sam Steiny, Mikaila GarfinkelA recipe-finder application built on image embeddings and similarity search over a recipe dataset. Databricks
2021 Use of convolutions for time series prediction and natural language processing Disclaimer: I assume familiarity with machine learning and a general sense of how feed forward (i.e., the most basic type of) neural networks work for the… This site
2021 Building a Natural Language Processing API with SAS Viya, FastAPI, and Docker Interactively testing the service we will build in this blog The notebook for training the topic model and obtaining sentiments is at: notebook The code… This site
2021 Effective Hyperparameter Optimization of LightGBM Classifiers with Optuna LightGBM is a really convenient to use, fast to train and usually accurate implementation of boosted trees. Here I use optuna for hyperparameter search… This site
2021 Bayesian Optimization for Neural Architecture Search In this blog I will be using bayes_opt, a general purpose bayesian optimization framework to optimize hyperparameters of a fastai tabular learner with… This site

§05Talks

2024 Innovating with Multimodality and Reasoning with Mistral AI ODSC West · also at AWS GenAI Developer Day, San Francisco Where multimodal input and explicit reasoning change what you can build on a foundation model. Given in the "What's Next in AI" track, and again for a developer audience at the AWS SF Loft with AICamp. Talk
2024 Empowering Generative AI with Databricks and AMD Data + AI Summit Running generative AI workloads on AMD accelerators inside Databricks, and what it takes to make a non-NVIDIA stack practical. Talk
2023 Text to Insights: Building Real-Time Analytics Systems with Generative AI ODSC Generative AI Summit Wiring language models, vector databases and efficient fine-tuning into something that turns a plain-English question into a live analytical answer. Talk
2023 Getting Up to Speed on Real-Time Machine Learning ODSC Europe What people actually mean by "real time" across streaming inference, feature freshness and online learning, and which parts are worth paying for. Talk
2023 Monetizing Data Assets: Sharing Data, Models and Features Data + AI Summit Treating models and feature tables as shareable, governed products instead of artifacts locked inside one team's workspace. Talk
2022 Streaming Machine Learning with Databricks and GitHub Actions GitHub Universe Continuous delivery for streaming ML. Model training and deployment driven from CI on a structured streaming pipeline. Talk

§06Podcasts & press

2025 Inside Applied AI at Mistral: Benchmarks, Guardrails & Career Advice Chris's Corner · with Chris Richardson Thirty-five minutes on building on top of foundation models. Fine-tuning, DPO and quantization, why evaluation and guardrails end up being the hard part of any enterprise deployment, and how to get into applied AI research. Podcast
2025 Mistral AI on AWS: 3 Features Developers Need to Know AWS Developers Short walkthrough of running Mistral models on AWS. Video
2024 Building a Mixture of Experts Model from Scratch with makeMoE Fahd Mirza Not mine. Fahd Mirza reads through the makeMoE code, and does a good job of it. Coverage

§07Background

2024 to now Research Scientist Mistral AI Robotics foundation models, mostly manipulation and navigation. Contributor on the Robostral Navigate release, and a lead contributor on Shieldstral and Mistral Small Creative. I joined in 2024 on the applied side and led the US applied science team before moving to research. Robostral Navigate Shieldstral Mistral Small Creative
2021 to 2024 Mosaic AI - Solutions Architect Lead, Digital Native and Emerging Databricks Field engineering for digital native and emerging accounts in the Bay Area. LLM fine-tuning, recommender systems, vector search and streaming ML. Most of my writing and conference talks come from this period.
2019 to 2021 Senior Solutions Architect, AI/ML SAS Applied machine learning on SAS Viya in Cary, North Carolina. Explainability, NLP services and model deployment. I led the AutoML prototype team.
2018 to 2019 Founder & Lead Data Scientist RandomWalk Technologies I built Fora, a conversational analytics engine that answered plain-English questions about social media data. Ames, Iowa, out of the ISU Startup Factory.
2016 to 2017 M.S.E., Mechanical Engineering and Applied Mechanics University of Pennsylvania Started as a PhD student in Jordan Raney's Architected Materials Lab, researching mechanical metamaterials on a Dean's Research Fellowship, with coursework in numerical methods and computational mechanics. I left with a master's after getting into machine learning. Wharton's Modern Data Mining course was the turning point.
2016 B.S., Mechanical Engineering Iowa State University Undergraduate research in bio-microfluidics with Nastaran Hashemi. That work became a co-first author review in ACS Biomaterials Science & Engineering.

§08Elsewhere

LinkedIn is the most reliable way to reach me. Most of what I build ends up on GitHub sooner or later, and the papers are on Scholar.

gh github.com/AviSoori1x GitHub
sch Google Scholar Google Scholar
in linkedin.com/in/avi-data-ml LinkedIn
x @aviTwit3 X
hf huggingface.co/AviSoori1x Hugging Face