trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
NeurIPS 2026 Workshop “Who Verifies the Agents?” · accepted
Hadi Mohammadi
I build production LLM and ranking systems, and research explainable NLP and LLM evaluation.
Senior AI & Data Science Expert - R&D Specialist at AcademicTransfer and PhD in Explainable NLP from Utrecht University.
About
At a glance
At AcademicTransfer I build the LLM matching and ranking behind the recruitment platform of 22 Dutch research universities and university medical centres. It reads 10,000+ applications a month. It has cut time-to-hire by 20% and improved candidate–job matching by 17%.
My PhD at Utrecht University asked why a model gives the answer it does and whether that explanation can be trusted. Its papers appeared at *SEM 2026, in the CLIN Journal and Applied Sciences, and at an ACL 2025 workshop. This autumn adds a long paper at INLG 2026 and my first solo paper at a NeurIPS 2026 workshop.
Hiring
Industry ML & LLM systems
Each role and its results, applied ML projects, and open-source libraries for auditing LLM judges.
Research
Explainable & culturally fair NLP
Explainability, moral alignment and LLM evaluation: the research line behind my PhD thesis.
Learning
Explainable AI, hands-on
A course and a demo that run in the browser, field notes on production LLM systems, and open guides for people studying AI.
Experience
Experience & education
Professional experience
-
Dec 2025 – now
Senior AI & Data Science Expert - R&D Specialist
AcademicTransfer, Utrecht
- Leads the company’s 2026 WBSO R&D programme as R&D Specialist since Mar 2026: an MCP-based academic query engine and an explainable CV matching engine with recruiters in the loop.
- Technical lead of the AI layer and applicant-tracking system.
-
Jan 2024 – Dec 2025
AI & Data Science Expert
AcademicTransfer, Utrecht CV Priority Sorter and generative CV pipelines: manual effort halved
- Deployed transformer models on Azure ML (MLflow, Docker, CI/CD); evaluated ranking with A/B tests, uplift analysis, and bandit simulations.
-
Feb 2024 – Mar 2025
Researcher & NLP Developer
Utrecht University School of Economics, FIRMBACKBONE
Built the fbb-sustainability-analysis-cli tool: keyword extraction and sustainability scoring of company websites.
-
Dec 2021 – Jan 2023
Senior Data Scientist, Customer Behaviour
SnowaTec (Innovation Center), Entekhab Electronic, Tehran
ETL warehouse (+16% data accessibility), KPI & satisfaction index (+2%), ML customer segmentation (+7.5% marketing impact).
-
Apr – Oct 2020
Head of Data Science & BI Team
Bdood.bikes (shared bicycle system), Tehran
Dynamic pricing and usage prediction (+5% revenue), fleet intelligence (−8% maintenance costs), retention analyses (+4.5%).
-
Jul 2019 – Apr 2020
Data Scientist
Bdood.bikes, Tehran
Predictive maintenance and battery-health algorithms (−3.8% downtime); GIS models for high-risk riders (+3% safety, +5% UX).
Education
-
Feb 2023 – Oct 2025
defended Sep 2026PhD, Explainable NLP
Utrecht University · thesis Let Me Explain!
-
Sep 2018 – Apr 2021
MSc Industrial Engineering: Macro Systems
University of Tehran · thesis on dynamic pricing with reinforcement learning
-
Sep 2011 – Jul 2016
BSc Mechanical Engineering
Isfahan University of Technology, Golpayegan College
- Jun 2024Generative Modeling Summer School, TU Eindhoven
- 2023Oxford Machine Learning Summer School (OxML 2023), ML x Finance & NLP
- Aug 2022Explainable AI Summer School (XAISS), TU Delft, scholarship recipient
Core stack
- Python · PyTorch
- Hugging Face Transformers
- LLM fine-tuning (LoRA, PEFT)
- Preference optimization (DPO, GRPO)
- LLM-as-judge evaluation
- RAG · agents · MCP
- Ranking & matching models
- Azure ML · MLflow · Docker
- A/B testing · uplift · bandits
- SQL / PostgreSQL
Research
What I work on
Explainable NLP
NLP models whose decisions domain experts can inspect and question, through post-hoc methods and token-level explanations.
Cultural fairness in LLMs
How LLMs represent cultural variation in moral judgments and social norms, and how to evaluate their alignment across cultures.
LLM evaluation
Interpretable chain-of-thought and LLM-as-judge evaluation, to tell when a model’s predictions and explanations can be trusted.
Human-AI annotation reliability
Whether LLM annotations can stand in for human ones: model outputs against human rationalizations, and demographic bias in annotation.
PhD thesis: Let Me Explain! Explainable NLP for Understanding Large Language Models
Selected publications
Recent research
Reasoning–Answer Entailment Under GRPO and DPO Fine-Tuning
INLG 2026, Utrecht · long paper, pp. 718–730
EvalMORAAL: Interpretable Chain-of-Thought and LLM-as-Judge Evaluation for Moral Alignment in Large Language Models
*SEM 2026 (ACL), pp. 497–515
Exploring Cultural Variations in Moral Judgments with Large Language Models
Computational Linguistics in the Netherlands Journal, Vol. 15 (2026), pp. 59–77
News
Recent news & updates
- Oct 2026INLG 2026 paper published in the conference proceedings under its final title, “Reasoning–Answer Entailment Under GRPO and DPO Fine-Tuning”, with Dr. Anastasia Giachanou. About the paper
- Oct 2026Invited talk at the HAI SIG Day of Utrecht University’s Human-centered AI focus area, in the NLP@U session: “From Explainable NLP to Human-Centered AI: Connecting research with practice”.
- Sep 2026First solo-author paper accepted at the NeurIPS 2026 workshop “Who Verifies the Agents?”: “trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories”. Paper
- Sep 2026PhD defended at Utrecht University on 9 September 2026. About the thesis
- Aug 2026Long paper accepted at INLG 2026 in Utrecht (19–21 October): “Chain-of-Thought Faithfulness Under GRPO and DPO Fine-Tuning”, with Dr. Anastasia Giachanou. INLG 2026
Earlier news, 2023–2025 (11)
- Dec 2025Best Oral Presentation Award at the IOPS Winter Conference 2025 for research on cultural moral alignment in LLMs. Event details
- Dec 2025€5,000 Applied Data Science grant for “Optimizing the CV Priority Sorter for Fair and Efficient Prioritization of Academic CVs”, with Dr. Robert A. Bagheri, Dr. Georg Krempl, and Jeroen Sparla.
- Jul 2025€9,000 CUCo Spark Grant for “ReDOSE: Towards a more Circular Pharmaceutical Industry using AI”, an interdisciplinary collaboration across WUR, TU/e, and UMC Utrecht.
- Jul 2025Paper accepted at ECAI 2025 (LUHME Workshop) on cross-cultural morality understanding with large language models.
- Jun 2025€5,000 Applied Data Science grant for “Rule Extraction from Human Evaluation Using LLMs and Reinforcement Learning”, with Dr. Anastasia Giachanou and Dr. Shihan Wang.
- Apr 2025Paper accepted at ACL 2025 (GeBNLP Workshop) on LLM reliability assessment and annotation reliability.
- Jan 2025Selected for the LERU Doctoral Summer School on AI in Copenhagen, representing the Faculty of Social and Behavioural Sciences, Utrecht University.
- Dec 2024€14,400 Enfield grant (European Lighthouse to Manifest Trustworthy and Green AI) for research on “User Perspectives on Explainable AI”.
- Nov 2024Co-founded the NLP@U Young Scholar Network: a community for PhD students, postdocs, and early-career NLP researchers at Utrecht University and UMC Utrecht. YSN website
- Nov 2024Communications Chair at BNAIC/BeNeLearn 2024, the Joint International Scientific Conferences on AI and Machine Learning at Utrecht University.
- Dec 2023€7,500 Applied Data Science grant for “Assessing Reliability of Annotations in the Context of Model Predictions and Explanations”, with Dr. Pablo Mosteiro Romero, Dr. Anastasia Giachanou, and Prof. Dr. Massimo Poesio.
Newsletters
In English and Persian
English · every other Saturday
Weekend Learning
One finding every two weeks on how LLM evaluation breaks: judges, benchmarks, calibration. Every number is recomputed from the data, and each issue ends with an exercise you can run in about an hour.
Issue 9 · 24 Oct 2026:
Swapping the three words a heatmap blames flips 54% or 43% of a sentiment model’s verdicts
Persian · every other Friday
یادگیری آخر هفته
Its Persian sister, for beginners: one basic idea from machine learning and language models, with lessons you can read, play with, run in plain Python or download for offline use.
Issue 3 · 31 Oct 2026:
۱۷۵ میلیارد پارامتر جیپیتی ۳ همان کاری را میکنند که دو پارامتر کرایهی تاکسی
Teaching
Teaching & supervision
Master’s thesis co-supervision
Nine MSc students co-supervised at Utrecht University (2024–2026), with Dr. Robert A. Bagheri and Dr. Anastasia Giachanou.
Students by cohort
- 2024 Mijntje Meijer, Evi Papadopoulou, Lourenço Santos Moitinho de Almeida (graduated)
- 2025 Hein Brouwer, Tamás Kozák, Mohamad Jouhar
- 2026 Aliyah Vos, Geanina Verestiuc, Thomas Rietman
I also built Opening the Black Box, a free, self-paced course on explainable AI with thirteen hands-on labs. The instructor-led five-day version is proposed as a summer school for universities and companies in 2027.
Teaching experience
- Dec 2023Practical Assistant, Transformers Workshop: UMC Utrecht.
- Sep–Oct 2023Computer Lab Assistant: MSc Applied Data Science, Utrecht University.
- Aug–Sep 2021Lead Teaching Assistant, Deep Learning: Neuromatch Academy.
More
Explore further
Organizations
Where I’ve worked & contributed
Contact
Get in touch
I’m based in Utrecht, The Netherlands, and email is the fastest way to reach me for research collaboration, supervision, speaking or industry opportunities.
- Email hadi.mohammadi@outlook.com
- LinkedIn mohammadi-hadi
- ORCID 0000-0003-0860-9200
