LLM-as-a-Judge Is a Cost Problem. Executable Programs Fix It.

LLM-as-a-judge has become the default for automated evaluation, but the assumption underneath it, that you need a live model call to score a candidate, turns out to be wrong. The decision logic is separable from the model itself. Distill it once, run it forever.

PAJAMA works by extracting the scoring logic from an LLM into a committee of executable programs, each encoding a distinct facet of the original judge's decision process. Think of it as compiling a senior engineer's code review checklist into a battery of automated linters: the expertise is captured once, then applied cheaply and consistently to every new submission. Where LLM judges produce a verdict through an opaque forward pass, programmatic judges expose every scoring step as readable, editable code. When the committee's confidence is low, a fallback mechanism escalates that specific case to an LLM, routing only the genuinely ambiguous examples back to the expensive path.

That routing signal matters beyond just evaluation. Because programs produce structured, deterministic outputs, their verdicts become training labels for reward models at a cost that would be prohibitive with proprietary API calls.

Across five datasets and four model families, programmatic judges match the performance of a 13B-parameter LLM judge. On RewardBench, a reward model trained on program-generated labels outperforms one trained on a proprietary LLM's labels. The cost difference: two orders of magnitude. For ML teams running evaluation at scale or building reward pipelines for RLHF, the takeaway is direct: the per-sample API call is not a requirement, it is a default that can be replaced without sacrificing accuracy.

We're thinking: The deeper shift here is not cost reduction, it is auditability. We have spent years treating LLM judges as acceptable precisely because they correlate with human preference, but correlation is not an explanation. PAJAMA programs are inspectable: a team can read why a score was assigned, edit the logic when it is wrong, and version-control the evaluation criteria alongside the model. That changes the evaluation loop from a black-box oracle into a software artifact. The contrarian read is that this only works if the original LLM's decision logic is actually coherent enough to distill, and for edge-case distributions it may not be. But for any team already trusting LLM judges in production, the inspectability argument alone justifies the switch.

Key takeaways:

• PAJAMA distills LLM judge logic into a committee of executable scoring programs with a confidence-based fallback that escalates low-certainty cases to an LLM, making evaluation both inspectable and editable.
• Programmatic judges match a 13B LLM judge across five datasets and four model families; reward models trained on their labels beat proprietary-LLM-labeled alternatives on RewardBench at 100x lower API cost. Caveat: performance on highly ambiguous or domain-specific edge cases may still require the LLM fallback path more heavily than average results suggest.
• Teams running LLM-as-a-judge pipelines for evaluation or reward modeling should pilot PAJAMA's program distillation approach as a drop-in cost reduction, and treat the resulting program committee as a versioned artifact in their evaluation infrastructure.

Source: Codifying the Judge: Scalable Evaluation via Program Distillation