DEV Community

Join the Kaggle Benchmarking Challenge: $2,500 in Prizes for FIVE Winners!

Jem on September 23, 2026

We're excited to team up with Kaggle for a brand new challenge! Running through October 11, the Kaggle Benchmarking Challenge asks you to build a ...

Read full post

FrancisTRᴅᴇᴠ • Sep 23

My benchmark is gonna be interesting (assuming if this goes to plan). Looking forward to see what others comes up!!

Richard Smith • Oct 2

That part about benchmarks coming from specific itches really resonates. I've tried building general evaluations before and they never feel as grounded as when you're testing something you've actually struggled with.

Oleksandr • Sep 30

Marked the dates. Proper benchmarking is honestly such an underrated discipline in modern ML—everyone wants to train heavy models, but nobody wants to spend time building clean evaluation layers. Looking forward to seeing the constraints and what telemetry metrics the community prioritizes for this challenge.

Sam LABBE • Sep 24

I'm in.

tao tao • Sep 25

It sounds great. I can't help but want to join in now.

Kshitij • Sep 24

New to Kaggle, let's try this one.

Dan • Sep 27

okay I can

Sk Saado • Oct 2

I will be da winners 🏆

meili jiang • Sep 28

Hi~everyone, i'm here

Mahender Pratap • Sep 27

interesting

me • Sep 29

This is the kind of tooling I’d actually use. Less interested in how confidently the agent explains its work, more interested in whether I can check the result without retracing every step.

MR • Sep 24

This is a submission for the Kaggle Benchmarking Challenge

Can LLMs Actually "Think" Code, or Just Fix Syntax? A Logic Bug Benchmark

Many LLMs are great at fixing missing semicolons, but how do they handle silent, destructive logical flaws? I built a targeted benchmark to find out.

🔍 What I Benchmarked

I measured Multi-step Reasoning and Code Generation/Debugging performance. Specifically, I set out to measure a model's ability to detect and fix silent logical bugs (e.g., off-by-one errors, incorrect loop boundaries, and race conditions) versus syntax errors in Python and Go.

This behavior interested me because syntax errors are caught by compilers, but logical bugs make it to production. I wanted to see if models truly understand code execution flow or just rely on pattern matching.

🤖 Models Tested

I benchmarked three models using zero-shot prompting:

Model A (Claude 3.5 Sonnet): Chosen for its industry-leading reputation in software engineering tasks.
Model B (GPT-4o): Chosen as the gold standard for general-purpose instruction following.
Model C (Llama-3-70B): Chosen to evaluate how a top-tier open-source model competes with proprietary giants.

📊 Findings & Real-World Meaning

The results taught me something real and surprising about current LLM limitations:

The Copy-Paste Bias: All models scored above 95% on syntax fixes. However, when facing logical bugs, performance dropped drastically.
The Shocking Insight: Claude 3.5 Sonnet outperformed GPT-4o by 22% on multi-step logical debugging. GPT-4o often hallucinated that the logic was correct if the code "looked" clean.
Llama-3's Failure Pattern: Llama-3 continuously fell into a loop of fixing the syntax but completely missing the race condition, proving it relies heavily on surface-level pattern matching.

What this means practically: These results changed my view. We cannot trust LLMs to review complex backend logic autonomously yet; they are excellent syntax assistants but mediocre logical auditors.

Next Up: I plan to measure how chain-of-thought (CoT) prompting alters these scores.

🔗 My Benchmark

You can view my full dataset, prompts, and evaluation pipeline here:
👉 My Kaggle Benchmark Notebook & Dataset (Replace this with your actual Kaggle link)

howcani howcani • Sep 24

I went and computed the estimator behind one of your cells, because the [0, 0] case has a sharper edge than your paragraph gives it. No refit of your data -- this is a simulation and an exact enumeration of the percentile bootstrap for a difference in proportions and a difference in medians, at cell shapes of the kind you describe.

1. The tie fraction, and its closed form. Call t the probability that one resample reproduces the observed difference exactly. For a cell with one event in one arm and none in the other, t = (1 - 1/n)^n, so it rises to 1/e = 0.3679 and stays there for any n: simulation gives 0.3632 at n=40, 0.3660 at n=100, 0.3674 at n=400, 0.3677 at n=1000. The number of observations stops mattering once the cell has one event in it.

2. [0, 0] is not printed by that. It requires the statistic to be constant over every resample, since the quoted 2.5th and 97.5th percentiles must both land on 0. For a difference in proportions that happens exactly at the boundary: both arms at zero events, or both at all events. I enumerated every cell with n1, n2 up to 12 -- 288 of them have tie probability exactly 1.0, and the largest tie probability over all the non-boundary cells in that range is 0.375. Simulation agrees at n=40 and n=1000 for both 0/n versus 0/n and n/n versus n/n: 1.0000.

So the [0, 0] cells in a zero-heavy table are the empty-side cells, and the phenomenon is a point mass rather than a nearly constant statistic. It is also invariant in n: 0/5 vs 0/5, 0/40 vs 0/40 and 0/1000 vs 0/1000 print the identical string, so the n displayed beside such a row never enters the interval. The label that carries the information is not [0, 0] but no interval: the resampling distribution is a point mass (0 events in both arms) -- which is a statement about the comparison having an empty side, exactly the exclusion rule field of your contract, not a statement about precision.

3. The same zero-heaviness does not collapse the median column, which supports your rule that the interval inherits the metric beneath it. 50 observations per arm with 30 versus 28 zeros: the difference in medians ties at the observed 0 in 69.4% of resamples and its interval is still [-1.5, +1.0], because the difference in medians has support below the observed value, while the difference in proportions with an empty arm is bounded below by 0. The amount of zeros is not what decides the collapse; the support of the statistic is.

4. A smaller version of the same read is worth flagging in the non-collapsed cells. In the 1/40 versus 0/40 cell the printed interval is [0.000, +0.075] under every seed I ran, and the 0 is not a quantile of the sampling distribution -- it is the support boundary plus the 36.3% tie mass. So a one-sided-looking interval like [0, +x] does not mean the effect was found to be non-negative; it is what the percentile method prints whenever one arm is empty, and the left end is the design, not the data.

5. The three-line check, in your units. Count the resamples whose statistic equals the observed one and print that fraction beside the interval: below the quoted percentile it is a normal cell; above it, the interval is being read off the tie mass and the honest output is the tie fraction itself. That one number separates your [0, 0] cells (tie fraction 1.0, any n) from the sparse cells that merely look like them (tie fraction 1/e), and it is cheap enough to sit in the same loop that builds the interval.