I Tried Making my Personal (Dangerous) LLM Benchmark to Cheat in Escape Rooms

Not too long ago, DeepSeek introduced their newest mannequin, R1, and article after article got here out praising its efficiency relative to price, and the way the discharge of such open-source fashions may genuinely change the course of LLMs eternally. That’s actually thrilling! And likewise, too massive of a scope to write down about… however when a mannequin like DeepSeek comes out of nowhere with a metal chair, boasting related efficiency ranges to different fashions, what does efficiency actually imply on this context?

In the event you comply with AI releases, you’ve seen this dance earlier than. Each new mannequin drops with its graphs exhibiting the way it’s by some means concurrently higher than GPT-4 on math issues whereas being smaller and extra environment friendly. However what precisely are these benchmarks measuring? How are they created? And extra importantly, how can we minimize via the hype to create our personal benchmarks for particular use circumstances?

I wished to study extra about LLM Benchmarking.

Half 1: What’s a Benchmark? (in 3 seconds)

TL:DR — The SATs (a number of, truly) for LLMs.

Half 1.1: What’s a Benchmark? (in additional than 3 seconds)

Earlier than we dive into the nitty-gritty of particular benchmarks, let’s take a second to unpack what we even imply by “LLM Benchmark.” As a result of calling them the “SATs for AI” feels each proper and in addition barely oversimplified.

LLM benchmarks are, at their core, structured assessments used to measure how properly giant language fashions carry out on sure duties. These duties may be something from figuring out if an announcement is true or false, to summarizing a authorized doc, to producing legitimate Python capabilities. Consider them as curated impediment programs specifically designed by AI researchers to check each related muscle these fashions may need. These frameworks usually present a dataset of inputs with identified appropriate outputs, permitting for constant comparability between fashions.

Trendy benchmarks make use of varied analysis methodologies. Classification metrics like accuracy work for duties with discrete appropriate solutions, whereas overlap-based metrics (BLEU, ROUGE) consider free-form textual content era. Some benchmarks use practical testing for code era, or make use of different LLMs as judges to guage response high quality.

A typical benchmark often comes packaged as:

• A standardized dataset of questions, prompts, or duties (with appropriate or reference solutions).
• An analysis protocol specifying the way to measure success, like accuracy, F1 rating, BLEU/ROUGE for textual content era, or cross/fail charges for coding duties.
• A leaderboard or some type of comparative scoreboard, typically with massive flashy graphs.

Some actually well-known benchmarks embody MMLU for testing multitask language understanding, TruthfulQA for assessing factual accuracy, and HumanEval for measuring coding capabilities. Outcomes are fairly typically printed on public leaderboards, which let’s folks carry out some clear comparability between completely different fashions.

What Makes a Good Benchmark?

1. A Clear Job Definition: We would like duties which can be unambiguous. The extra easy and well-specified the problem, the better it’s to belief the outcomes.
2. Knowledge Integrity: The check set shouldn’t be floating round within the coaching information. As a result of if the mannequin’s seen the very same query 50 occasions earlier than, the analysis is about as helpful as giving a math quiz to somebody who already has the reply key.
3. Quantifiable Metrics: You want a normal for scoring efficiency — like what number of occasions the mannequin’s code passes check circumstances or how shut the generated abstract is to a “ground-truth” abstract.
4. Job Range & Issue: If a benchmark is simply too simple, everybody simply ACES it on day one, and we study… properly, nothing. If it’s too area of interest (like “We check solely the mannequin’s means to rely the digits of Pi for 20 minutes”), that’s additionally not so useful.

Life Ain’t All about The Grades

Benchmarks seize solely a slice of what LLMs can do. In the true world, your chatbot may must juggle area information, maintain observe of dialog context, abide by your organization’s insurance policies, and produce fluent, non-offensive replies. No single standardized check on the market absolutely covers that. As we’ll see within the upcoming case research, the design and execution of a benchmark can closely form the image you get of your mannequin’s efficiency… and typically lead you astray should you’re not cautious with the way you measure success.

Now that we now have a way of what Llm Benchmarks are designed to perform (and the place they could fall quick), let’s discover a few examples to see how folks truly construct and use them in follow — with combined outcomes!

Case Research #1: Leetcode as an LLM Benchmark

As a pupil within the tech house, the phrase “Leetcode” popping up throughout my seek for cool benchmarks raised by blood strain by a statistically vital quantity. In contrast to Leetcode, which sucks, the paper “Efficiency Research of LLM-Generated Code on Leetcode” was very attention-grabbing — it asks a deceptively easy query: can we use Leetcode to benchmark LLM code era? Their findings reveal each the promise and pitfalls of this strategy.

The Benchmark Design

The researchers constructed a three-stage validation system. Native assessments catch fundamental errors, Leetcode’s decide verifies correctness, and a customized benchmarking setup measures efficiency. This setup revealed one thing important: benchmarking code efficiency is tougher than it seems.

After they in contrast native measurements to Leetcode’s metrics, they discovered solely a 0.28 correlation. Leetcode’s measurements confirmed a lot increased variation (0.089 vs 0.035 regionally). Even worse, Leetcode’s rankings proved unstable — an identical options may drop from the 77th to 54th percentile simply primarily based on submission timing.

The Actual Issues

Three main points emerged that problem Leetcode’s viability as a benchmark:

Knowledge Contamination: Utilizing public issues dangers LLMs having seen the options throughout coaching. The researchers had to make use of solely issues from 2023 to mitigate this.

Platform Instability: Leetcode’s metrics drift over time — reminiscence measurements confirmed a -0.24 correlation with check date. This makes reproducible benchmarking practically not possible.

Measurement Reliability: The weak correlation between native and platform measurements raises questions on what we’re truly testing.

What It Means for LLM Benchmarking

This research doesn’t simply critique Leetcode — it highlights what we want in a code era benchmark: reproducible measurements, dependable efficiency metrics, and assured training-test separation. Till we now have platforms constructed particularly for this function, we must be extraordinarily cautious about utilizing competitors platforms as benchmarks.

So! We all know that not all benchmarks are viable benchmarks — what a few extra mainstream one?

Case Research #2: SuperGLUE — Constructing a Higher Language Understanding Benchmark

The SuperGLUE paper tackles an interesting downside in AI benchmarking: what do you do when fashions get too good at your assessments? When GLUE turned inadequate (with fashions surpassing human efficiency), the researchers needed to rethink how we measure language understanding.

The Benchmark Design

SuperGLUE’s core innovation is its job choice methodology. The researchers collected job proposals from the NLP group and filtered them via a rigorous course of: every job wanted clear analysis metrics, public coaching information, and — most significantly — vital headroom between machine and human efficiency.

This resulted in eight duties (I’ve simplified the desk from the doc right here, it’s rather less readable however you must get the sense of what the questions are asking):

What makes these duties particular is their range in format. In contrast to GLUE’s concentrate on sentence classification, SuperGLUE contains coreference decision, studying comprehension, and extra com plex reasoning duties. Every job measures completely different elements of language understanding whereas sustaining clear, quantifiable metrics.

---

Half 2: Let’s Construct a Bodily Reasoning Benchmark: To Cheat at Escape Rooms

After taking a look at some benchmarks like SuperGLUE and Leetcode, I had an thought: what if we examined LLMs on one thing utterly completely different — bodily reasoning… via escape room puzzles?

It’s a fairly legitimate thought — escape rooms poses prospects and penalties for failure — screw up one too many puzzles, and your folks will suppose you’re fairly silly, and relegate you to spectator obligation. Fortunately for us nonetheless, they (or the poor workers) don’t know that you could sneak a cellphone into an escape room — and you already know simply who to ask for the solutions. Right now, LLMs face off in opposition to the puzzles of a bodily escape room.

Be aware: That is NOT a rigorous tutorial benchmark (please don’t cite this in papers, why would you even need to do this?), and even near it, and it’s simply speculated to be a enjoyable method to check LLM benchmarking and analysis. Please don’t destroy my prompts, I’m conscious they’re unhealthy.

Why Bodily Reasoning?

For actual, although… most LLM benchmarks concentrate on linguistic duties (like SuperGLUE) or code era (like Leetcode). And for good cause — these are well-defined domains with clear analysis metrics. However real-world downside fixing typically requires understanding bodily rules and their interactions. The well-known “Can GPT-4 do physics?” debates often focus on mathematical problem-solving, not sensible bodily reasoning.

current benchmarks taught me a number of key rules:

1. Clear analysis metrics are essential (from SuperGLUE’s task-specific scores)
2. Issues ought to have unambiguous options (from HumanEval’s check circumstances)
3. The benchmark ought to check distinct capabilities (from MMLU’s topic classes)

Designing the Issues

I settled on escape room puzzles for 2 causes. First, they naturally mix bodily reasoning with clear objectives. Second, they’ve unambiguous success situations — both you clear up it via the meant method, otherwise you don’t. Third, and most significantly, they let me embody “pink herrings” — irrelevant objects that check if the LLM can establish what issues bodily. Fourth, I simply actually like doing escape rooms (did I point out that already?),

I’m conscious that that is greater than two causes, but when LLMs can’t rely what number of rs’ there are in strawberry, I’m allowed to mess up on occasion too.

Right here’s how I structured the 5 core issues:

Fluid Dynamics (FLUID_001) (Ping pong ball caught in a tube)

• Checks understanding of buoyancy and fluid displacement
• Impressed by traditional physics issues however in sensible context
• Contains deliberately irrelevant objects (like squishy meals fashions)

Gentle Properties (UV_001) (UV gentle on a push numebr lock)

• Checks understanding of UV fluorescence and materials properties
• Combines a number of bodily rules (gentle, materials science)
• Requires understanding of environmental situations

Mechanical Understanding (CIPHER_001) (A cipher ring)

• Checks spatial reasoning and mechanical alignment
• No pink herrings — assessments for correlating a dial to a cypher wheel
• Requires understanding rotational symmetry

Power Utility (VAC_001) (Can caught in gap)

• Checks understanding of vacuum forces and floor adhesion
• A number of doable resolution approaches
• Requires understanding drive multiplication

Collaborative Physics (COLLAB_001) (Can two folks shimmy a key?)

• Checks understanding of bodily constraints in multi-agent eventualities
• Requires combining a number of bodily rules
• Checks understanding of software creation and friction

Sounds actually fancy… however it’s just a few fundamental bodily puzzles. You possibly can entry them on my GitHub.

The Technical Half

The benchmark implementation has three fundamental elements:

Downside Definition Layer

Issues are outlined in a structured JSON format that enforces constant analysis:

{
    "problem_id": "FLUID_001",
    "setup": {
        "state of affairs": "A ping pong ball is on the backside of a slender tube...",
        "available_items": ["bottle of water", "squishy food models"...],
        "constraints": ["tube too narrow for manual retrieval"]
    },
    "physical_principles": ["buoyancy", "fluid displacement"],
    "red_herrings": ["squishy food models", "milk carton"],
    "resolution": {
        "steps": ["pour water into tube", "allow ball to float"],
        "key_insights": ["water displaces air", "ping pong ball less dense"]
    }
}

This construction attracts from SuperGLUE’s design — every part is clearly separated and machine-readable. The physical_principles area explicitly lists what’s being examined, whereas red_herrings helps in scoring the LLM’s means to disregard irrelevant data.

2. Analysis Framework

The analysis system makes use of Python’s asyncio for concurrent testing, with retry logic for just a little bit extra API stability:

@retry(cease=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def evaluate_response(self, standards: JudgingCriteria) -> Dict:
    """Consider a mannequin's response utilizing GPT-4 as decide."""
    async with aiohttp.ClientSession() as session:
        # ... analysis logic

The scoring system seems at three elements:

Bodily Understanding Rating (PUS) ∈ [0,2]

• Measures understanding of related bodily rules
• Calculated as normalized sum of demonstrated rules

Answer Path Rating (SPS) ∈ [0,2]

• Evaluates completeness and correctness of resolution steps
• Considers sensible feasibility of proposed options

Purple Herring Dealing with (RHH) ∈ {0,1}

• A Binary rating for avoiding irrelevant objects
• Checks means to concentrate on bodily related elements

And sure, there are additionally so many different scoring strategies, higher and worse, that could possibly be used! For instance, RHH could possibly be about how many irrelevant objects are used within the resolution, or it could possibly be a measure of how viable the use is… the purpose is that selecting these metrics are sometimes occasions fairly arbitrary, however are very essential to creating your benchmark is credible, which mine may be very a lot not.

Moreover, I didn’t need to rewrite any code after. Sue me.

3. Mannequin Interface Layer

The benchmark helps a number of LLM backends via a typical interface:

class ModelInterface:
    """Interface for various LLM APIs."""
    async def generate_response(self, immediate: str) -> str:
        elevate NotImplementedError

class GPT4Interface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # GPT-4 particular implementation
class ClaudeInterface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # Claude particular implementation

Two fashions… I can’t actually afford any extra, please perceive.

Let’s Check It!

So after some refilling of API balances, I used to be able to go. I ran the benchmark, and after completely not losing credit on like, 5 completely different errors, the outcomes had been in. They usually had been… attention-grabbing! I used GPT4 and Claude 3.5 Haiku (I’m conscious these should not equally spec-ed fashions) as our victims.

The Problem: A can is caught in a gap that nearly precisely matches its quantity. A mix is written on the underside of the can that must be retrieved.

"VAC_001": {
        "problem_id": "VAC_001",
        "setup": {
            "available_items": [
                "ruler",
                "notebook",
                "suction cup",
                "UV light"
            ],
            "constraints": [
                "can fits too tightly for finger grip",
                "hole is same diameter as can",
                "can is clean and dry"
            ]
        },
        "physical_principles": [
            "vacuum suction",
            "surface adhesion",
            "force multiplication"
        ],
        "red_herrings": [
            "ruler",
            "notebook",
            "UV light"
        ],
        "resolution": {
            "steps": [
                "inspect suction cup and can surface",
                "ensure can surface is clean",
                "firmly press suction cup onto can top",
                "create vacuum seal",
                "pull can out using suction cup"
            ],
            "key_insights": [
                "suction cup creates vacuum seal",
                "vacuum force allows extraction",
                "other items don't provide necessary force"
            ]
        }

GPT-4’s Answer

Bodily Ideas: The primary bodily precept concerned on this resolution is the idea of air strain differential, which is identical precept that enables a suction cup to stay to a floor. Air strain is increased on one aspect of the suction cup (outdoors) in comparison with the opposite (inside), and this distinction in strain causes the cup to stay to the floor.

Answer Steps:

1. Connect the suction cup to the highest of the can. The flat floor of the can ought to present a superb contact for the suction cup to stick to.
2. As soon as the suction cup is firmly connected, gently pull upwards. The air strain differential ought to trigger the suction cup (and thus the can) to be lifted out of the opening.
3. As soon as the can is out of the opening, take away the suction cup from the highest of the can.
4. Use the UV gentle to light up the mixture written on the underside of the can. Some inks solely turn into seen underneath UV gentle, which is why this step is critical.
5. Write down the mixture on the pocket book to make sure it’s not forgotten.

Required Objects:

• Suction cup: That is required to stick to the can and pull it out of the opening, because the can suits too tightly for finger grip.
• UV gentle: That is required to disclose the mixture written on the underside of the can, as it could be written in ink that solely turns into seen underneath UV gentle.
• Pocket book: That is required to report the mixture as soon as it’s seen, making certain it’s not forgotten.”

Not unhealthy in any respect! GPT-4 manages to determine the bodily precept behind the answer (even when it is rather wordy), and manages to step-by-step describe the way to clear up the issue. It does find yourself utilizing the UV gentle and pocket book, however that’s extra of an issue on my finish, not punishing using pointless objects within the immediate sufficient. Additionally, the makes use of for this stuff appear decently cheap… so why not!

Haiku-3.5’s Answer

Bodily Ideas: The important thing bodily rules concerned on this state of affairs are the idea of quantity and using suction to extract the can from the opening. For the reason that can suits nearly precisely into the opening, it’s seemingly that the can is caught as a result of tight match and the shortage of any air gaps or house between the can and the opening. This tight match creates a vacuum-like impact, making it tough to grip and pull the can out.

Answer Steps:

1. Measure the diameter of the can utilizing the ruler. This can assist decide the scale of the opening and make sure that the can suits tightly inside it.
2. Examine the underside of the can to find the mixture written on it. Use the UV gentle to light up the underside of the can and make the mixture seen, if it isn’t already clearly seen.
3. Connect the suction cup to the highest of the can. The suction cup will create a brief seal on the can’s floor, permitting you to grip and pull it out of the opening.
4. Gently pull the can upwards utilizing the suction cup. The suction drive needs to be sturdy sufficient to beat the vacuum-like impact and extract the can from the opening.
5. As soon as the can is out, report the mixture written on the underside of the can within the pocket book.

Required Objects:

• Ruler: To measure the diameter of the can and make sure the tight match within the gap.
• UV gentle: To light up the underside of the can and make the mixture seen, if it isn’t already clearly seen.
• Suction cup: To create a brief seal on the can’s floor and supply the mandatory grip to tug the can out of the opening.
• Pocket book: To report the mixture written on the underside of the can.

However, Haiku’s resolution was.. definitely an answer of all time.

First, it desires to measure the diameter of the can utilizing the ruler??? To find out the scale of the opening and make sure that the can suits tightly inside it? Why would we have to do that? And do we want a ruler for that?

Second, it tells us to examine the underside of the can to find the mixture, when all the downside is about not having the ability to pull the can out of the opening conventionally. This may simply be a difficulty of order, however now I actually perceive my associates’ emotions at any time when I might inform them “simply repair it man” to their quite a few issues.

However it will definitely does get the answer. So… not the worst.

Right here’s a elaborate radar graph of the outcomes!

We see that each fashions are fairly related of their capabilities, with GPT-4 being barely higher in bodily understanding and resolution path, and Haiku being barely higher in pink herring dealing with. Total although, each fashions type of suck. Dang.

There are additionally solely… 5 questions.

In the event you’d prefer to see the complete breadth of questions, they’re on my GitHub.

LLM-as-a-Decide

By the best way, the tactic I used to generate the evaluations, LLM-as-a-judge, has gained vital traction within the AI group, notably after the work of Zheng et al. of their 2023 paper “Judging LLM-as-a-Decide.” The method has confirmed remarkably efficient, reaching over 80% settlement with human evaluators in duties starting from code evaluation to dialogue high quality analysis!

Right here’s the place my experiment will get type of cool (arguably, possibly, subjectively) — I used this technique and had GPT-4 decide different LLMs’ bodily reasoning talents. Sure, I’m utilizing an AI to guage different AIs.

Why does this work? Properly, judging a response is definitely a less complicated job than producing one. When GPT-4 generates an answer to a bodily puzzle, it must:

• Perceive the bodily rules concerned
• Plan a sequence of steps
• Take into account all constraints
• Generate a coherent clarification

However when judging, it solely must test if particular standards are met in an current resolution. The analysis immediate may be very targeted:

def _create_evaluation_prompt(self, standards: JudgingCriteria) -> str:
    return f"""You're an skilled decide evaluating an LLM's understanding of bodily reasoning puzzles.

Consider primarily based on three standards:
2. Bodily Understanding Rating (0-2): Does the answer accurately apply related bodily rules?
3. Answer Path Rating (0-2): Are the steps full and possible?
4. Purple Herring Dealing with (0-1): Does it keep away from utilizing irrelevant objects?
State of affairs: {standards.state of affairs}
Bodily Ideas Required: {standards.correct_principles}
Answer Given: {standards.model_response}
"""

To validate this strategy, I adopted the validation framework prompt by Zheng et al., performing spot-checks of GPT-4’s evaluations in opposition to my very own judgments. Surprisingly (or maybe unsurprisingly, given the broader analysis on LLM analysis), it was remarkably constant in figuring out each appropriate bodily understanding and flawed reasoning.

Is that this excellent? Completely not. There’s one thing philosophically bizarre about utilizing one LLM to guage one other. However in follow, it could actually work surprisingly properly — similar to how I moan and groan in regards to the visible presentation of a dish on Masterchef, whereas setting my kitchen aflame attempting to microwave a sizzling canine.

What I Realized

Constructing this benchmark taught me a number of issues about benchmark design:

Clear Metrics Matter: Even for advanced duties like bodily reasoning, you want unambiguous scoring standards.

Purple Herrings Are Highly effective: Together with irrelevant objects reveals rather a lot about an LLM’s reasoning course of.

Context Management is Exhausting: Guaranteeing LLMs don’t “hallucinate” extra bodily context is difficult.

Is that this an ideal benchmark? Not even shut. Please don’t rub it in. Is it scientifically rigorous? Undoubtedly not. However it’s been an interesting exploration into a facet of LLM capabilities, and typically the very best we are able to study can come from simply attempting issues out and seeing what occurs.

Now, should you’ll excuse me, I can be sneaking in a cellphone with an web connection into my subsequent escape room, for causes that I’m legally unmotivated to reveal.

[1] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, “Judging LLM-as-a-Decide with MT-Bench and Chatbot Enviornment,” Proceedings of the thirty seventh Convention on Neural Data Processing Programs (NeurIPS 2023), Datasets and Benchmarks Monitor (2023)

[2] T. Coignion, C. Quinton, R. Rouvoy, “A Efficiency Research of LLM-Generated Code on Leetcode,” In twenty eighth Worldwide Convention on Analysis and Evaluation in Software program Engineering (EASE 2024), Salerno, Italy (2024)

[3] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, “SuperGLUE: A Stickier Benchmark for Normal-Objective Language Understanding Programs,” In thirty third Convention on Neural Data Processing Programs (NeurIPS 2019), Vancouver, Canada (2019)

[5] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Music, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z.F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao et al., “DeepSeek-R1: Incentivizing Reasoning Functionality in LLMs through Reinforcement Studying,” arXiv preprint arXiv:2501.12948 (2025)

[6] Except in any other case said, all pictures are created by the writer.