Full Papers
6 pages + references · Oral presentation
NLP/CORE Unranked
JUDGe 2026
First Workshop on Reliable Evaluation for Language Models (JUDGe)
JUDGe 2026 is a full-day NeurIPS workshop focused on the reliability and validity of LLM-based evaluators, addressing how evaluation failures cascade within AI systems. It brings together researchers and practitioners to examine systemic issues in LLM judges, including bias, criteria drift, and deployment risks, with the goal of establishing community standards for evaluator transparency.
Paper fit
Contribution paths
A strong submission should clearly identify its contribution and evaluate it appropriately.
Short Papers
4 pages + references · Poster presentation
Research areas in scope
01
Core Evaluation Validity
Construct validityCalibrationHuman–model alignmentCriteria pre-registrationInter-judge consistency
02
Surface vs. Semantic Sensitivity
Robustness of LLM judges to meaning-preserving paraphraseLength and formatting biasSemantic content scoring
03
Bias & Sycophancy
Positional and ordering bias in pairwise evaluationSelf-preferenceFeedback loop risks in judge-guided alignment
04
Safety-Relevant Evaluation
Semantic drift under iterative transformationAdversarial robustness of safety evaluatorsMeaning preservation
05
Benchmark Construction
Adversarial benchmark designCriteria drift in datasetsDomain-specific frameworksNegative-result datasets
06
Judge Design & Architecture
Prompt engineeringFine-tuning vs. promptingMulti-judge ensemblesTool-augmented judgesCapability gaps
07
Practitioner Perspectives
Production evaluation pipelinesPractitioner–researcher gapsDeployment failure case studiesCost–quality trade-offs
08
Agentic & Emerging Topics
Reasoning chain validityMulti-turn agentic evaluationCross-lingual reliabilityEthical dimensions of automated evaluation
Policies worth checking twice
- Papers must be submitted via OpenReview
- All deadlines are 11:59 PM AoE
- Notifications precede the mandatory NeurIPS deadline of September 29, 2026
- Camera-ready versions due October 15, 2026
- Works in progress, negative results, practitioner case studies, and cross-disciplinary contributions are particularly encouraged
Official sources
Compiled from the official call for papers. The organizers’ pages remain authoritative.
Last verified September 9, 2026