Ten Minutes to Find Out Whether the Tests a Local Model Wrote for Your Function Protect a Bug

Ten Minutes to Find Out Whether the Tests a Local Model Wrote for Your Function Protect a Bug

We gave a local model this function and asked for pytest tests. It wrote ten in six seconds. Three of them assert that the deadline day does not count, which is the bug. None of the ten would fail on the buggy version and pass on the fixed one. If you ran those tests today, seven would pass (three more fail on both versions); the day you fix the function, three of the seven would turn red and tell you the fix is wrong.

def days_left(today: date, deadline: date) -> int:
    """Days remaining until the deadline, inclusive of the deadline day. 0 when the deadline has passed."""
    diff = (deadline - today).days
    if diff < 0:
        return 0
    return diff          # bug: the deadline day is not counted; should be diff + 1

(The # bug comment is for you; the model received the file without it, docstring included.)

Last week we measured this on all 164 HumanEvalPack Python problems: given the buggy body with no docstring, 16.4% of the tests Qwen3.5-9B wrote passed on the bug and failed on the fix, against 2.6% when it saw the fixed body. The function above went in with its docstring, so the measurement's comparable condition is buggy body plus the human docstring: 4.8% misguided on average. Three of ten here is one draw from that condition, not from the 16.4% one. This post is the ten-minute version for one function of yours: the same prompt and the same sandbox, pointed at your code.

What you need

• A function, in two files: the version you have now (my_func.py) and the version you believe is right (my_func_fixed.py). The model sees only the first one. The natural place to get the pair is your git history: take a bug you already fixed, check out the function before and after the commit, and you have a test of whether model-written tests can be trusted on your own code. If you do not have a fixed version, write the one you think is right; the point is to see whether the tests can tell the two apart.
• A local model behind an OpenAI-compatible endpoint. We use llama.cpp's llama-server with Qwen3.5-9B Q4_K_M (a 5.7 GB file); Ollama and LM Studio serve the same endpoint.
• Python 3 with pytest. Nothing else.
• check_my_function.py from the reproduction package at the end of this post (folder tutorial/). It is 109 lines; the prompt is copied from the measurement and the sandbox is the same idea, with one simplification noted below.

Run it

Start the server (one line; the flags are the ones we measured with):

llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 -fa on --parallel 1 -c 20480 --jinja --port 8080

Then:

python check_my_function.py --buggy my_func.py --fixed my_func_fixed.py --name my_func \
    --server http://127.0.0.1:8080 --save test_generated.py

What it does, in order: sends your current version with the request from the paper we reproduced ("write some unit tests in Python 3 with pytest with maximizing both branch and line coverage", temperature 0, thinking off, up to 4,096 tokens); pulls the code blocks out of the answer; writes them to a temp directory next to each version as solution.py; runs pytest twice with the network disabled and a 120-second limit for the whole test file (the measurement's scorer also re-ran a timed-out file test by test with 10 seconds each; this script does not); sorts every test into one of four boxes.

Expected output, for the function above:

model answered in 6 s (finish_reason=stop); 10 tests collected
  misguided    3   test_deadline_leap_year, test_deadline_today, test_deadline_tomorrow
  effective    0
  both_pass    4   test_deadline_far_past, test_deadline_leap_year_past, test_deadline_past, test_deadline_same_day_different_year
  both_fail    3   test_deadline_far_future, test_deadline_future, test_deadline_next_year
misguided tests assert the current behaviour as correct. If your current version is the buggy one, these tests would pass today and fail after the fix.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

How to read the four boxes

• misguided: passes on your version, fails on the fixed one. The test has written your bug down as the expected answer. test_deadline_today asserts days_left(today, today) == 0; the docstring says the deadline day counts, so the right answer is 1.
• effective: fails on your version, passes on the fixed one. This is the test you wanted. Zero here means the suite cannot catch this bug.
• both_pass: cannot tell the two versions apart (here, every past-deadline case, where both versions return 0).
• both_fail: wrong on both. In our run these three asserted day counts that match neither version, which is the model miscounting dates, not a property of your code.

The number that matters is the first line against the second. If misguided is above zero and effective is zero, the tests are a snapshot of the current behaviour, bug included. Coverage will say they are fine.

What to do when it comes back like this

Give the model the intended behaviour in a form that does not come from the code. In the measurement, attaching the human-written docstring to the buggy body cut misguided tests from 16.4% to 4.8% and raised effective tests from 19.5% to 41.8%. The function above already had its docstring attached, and the model still asserted the opposite of it in three tests; that is what a 4.8% average looks like on one function, and the 164-problem average is the number to trust. What did not help: asking the model to write the spec itself from the buggy code first (20.6% misguided, no better), and turning thinking on (on the 85 problems both modes finished, misguided rose from 22.4% to 37.6%, with the model often noting the bug in its reasoning and then asserting it anyway).

So the loop is: run the script, and if misguided tests show up, hand the model the spec (docstring, ticket text, one example of expected output) rather than the body, and run it again. The script takes the same two files; put the docstring in my_func.py and leave the body out if you want the no-code condition.

Limits

One model (Qwen3.5-9B Q4_K_M in llama.cpp), temperature 0, one generation per call, so a different model or a second run can land elsewhere. The example is one function; the rates quoted are from 164 problems with human-planted bugs, not from code in the wild. The script needs a fixed version to score against, which means you already suspect the bug; its job is to show whether the tests would have told you, not to find bugs you do not know about.