To Yield, or Not to Yield: Making the Heisenbug Testify

To yield, or not to yield: that is the race—

Whether the task shall suffer the stale read,

The silent overwrite, and the green rerun,

Or summon every witness in the schedule,

Bind cause to consequence, and make the fault repeat.

The test fails once. CI runs it again. The second run passes, and the dashboard turns green with the solemnity of a judge dismissing a case for lack of evidence.

Nothing was proved. The timing changed. The program received a reprieve, not a diagnosis.

That is the strange economy of asynchronous failure: a defect can cost hours while occupying only a few hundred microseconds. Add a print statement and the schedule moves. Attach a debugger and the pauses change. Rerun the job and the event loop deals a new hand. The witness is also an accomplice; observation alters the conditions under which the crime occurred.

ChaosSloth is built around a different proposition: a flaky failure is not an inconvenience to be retried until it disappears. It is an execution history to be recovered, ordered, replayed, and challenged.

What I Built

ChaosSloth is an autonomous concurrency investigation engine for Python asyncio. It samples natural coroutine scheduling, analyzes causal relationships, searches scheduling telemetry for a failure boundary, and reconstructs the candidate interleaving deterministically. It then uses local model reasoning to inspect the critical section, synthesizes a focused AST patch, compiles that patch to in-memory bytecode, and subjects it to adversarial execution.

The tool’s governing idea is spare: chance may reveal a defect, but chance should not be required to explain it.

An asyncio task does not need a second operating-system thread to race another task. It needs shared mutable state and a yield point between reading and writing it. Each coroutine can behave exactly as written while their combined history violates the invariant.

Here, both tasks read zero. Task A yields. Task B reads zero and writes one. Task A resumes and writes one from its stale value. Two increments were attempted; only one survives.

flowchart LR
    A["Task A reads x = 0"] --> B["Task A awaits and yields"]
    B --> C["Task B reads x = 0"]
    C --> D["Task B writes x = 1"]
    D --> E["Task A resumes and writes stale x = 1"]
    E --> F["Two increments attempted; final x = 1"]

No coroutine has violated its local instructions. The bug lives in the interval between them: the read-modify-write sequence was allowed to overlap without an atomic boundary.

Demo

The benchmark begins with 25 natural sampling cycles. The observed execution exhibits 68.0% natural intermittent flakiness. TabPFN-3.5 inverts the scheduling telemetry and identifies a 240.1 µs delay at 69.64% posterior confidence. ChaosSloth then forces the candidate interleaving and reproduces the race 100.0% of the time (10/10).

A local Google Gemma 2B model diagnoses the unsynchronized read-modify-write path at 52.91 tokens/sec on Apple M4 via Metal/Ollama. ChaosSloth synthesizes a Python AST transformation that places the shared-state transition inside an asyncio.Lock, while leaving non-blocking delays and yields outside the critical section.

The modified AST is compiled into in-memory bytecode and run through adversarial cycles. Sandboxed bytecode verification reduces flakiness from 68.0% to 0.0% in 9.07 seconds.

The workflow is a chain of evidence. Each phase narrows the next question; the final phase makes the proposed repair answer to the same schedule that exposed the defect.

flowchart TD
    A["1. Sample 25 natural cycles"] --> B["2. Invert timing boundary with TabPFN-3.5"]
    B --> C["3. Force candidate interleaving"]
    C --> D["4. Audit Coffman lock DAG"]
    D --> E["5. Check event-loop starvation"]
    E --> F["6. Diagnose AST with local Gemma"]
    F --> G["7. Synthesize atomic AST patch"]
    G --> H["8. Verify bytecode across adversarial cycles"]
    H --> I{"Invariant holds?"}
    I -->|"Yes"| J["Report verified repair"]
    I -->|"No"| C

Link to Code

Repository: github.com/smtherer/ChaosSloth

Clone the project:

git clone https://github.com/smtherer/ChaosSloth.git

How I Built It

ChaosSloth combines runtime observation with formal reasoning and compiler transformation. Telemetry alone can show that a delay preceded a failure; it cannot establish causality. A model can suggest a critical section; it cannot prove that a patch protects the invariant. The system therefore treats each result as a distinct obligation: measure the schedule, recover causal order, identify a candidate boundary, build a repair, and replay the collision against it.

1. Preserve the evidence of execution

ChaosSloth instruments coroutine yield points non-invasively and records yield deltas, event-loop latency, and context switches across 25 natural cycles. The baseline matters: a schedule reconstructed from perturbed timing may be a different schedule wearing the same name.

2. Establish causal order

In concurrent systems there is no single global “now.” There are events and the causal edges that relate them. Lamport’s happens-before relation, written $e_1 \prec e_2$, captures that partial order. For conflicting events on the same memory location, the race predicate is:

Race Predicate: ¬(e₁ ≼ e₂) ∧ ¬(e₂ ≼ e₁) ∧ (Write(e₁) ∨ Write(e₂)) ⟹ Data Race

Here, $W(e)$ means that event $e$ writes the shared location. When neither event causally precedes the other, their observed order in one run is not a promise about the next. Vector clocks let ChaosSloth reason about that partial order instead of mistaking wall-clock proximity for causality.

3. Invert the temporal boundary

Prior Labs TabPFN-3.5 analyzes the scheduling telemetry as tabular state. Given a candidate timing hypothesis $H$ and observed telemetry $D$, Bayesian inversion updates the hypothesis by the evidence:

P(H | D) = [ P(D | H) · P(H) ] / P(D)
(where H is the candidate timing boundary and D is the telemetry record)

The inferred coordinate—240.1 µs delay at 69.64% posterior confidence—is a target for an experiment, not a vague suspicion. ChaosSloth injects the schedule at that boundary and makes the interleaving recur 100.0% of the time (10/10).

4. Audit liveness as a graph

The lock audit represents acquisition dependencies as a directed graph $G=(V,E)$. A cycle in the wait-for relation is circular wait: each participant is waiting on a resource held by the next. The graph-theoretic condition is:

Graph Liveness Condition: G is a DAG ⟺ there is no directed cycle in G

The Coffman audit checks the observed lock-acquisition graph for cycles. ChaosSloth also looks for synchronous blocking primitives that can stop the event loop from making progress. A program may preserve every value and still be dead if no task can advance.

5. Diagnose locally, transform the syntax tree

Google Gemma runs locally through Ollama. It receives the dynamic AST context and diagnoses the unsynchronized read-modify-write path. On Apple M4 via Metal/Ollama, local Google Gemma 2B inference runs at 52.91 tokens/sec.

Python’s ast.NodeTransformer then synthesizes an atomic lock around the complete shared-state transition. The boundary is the point: lock only the write and the stale read has already escaped; lock the whole coroutine and unrelated work queues behind needless serialization.

flowchart LR
    A["Natural delay or await"] --> B["Acquire asyncio.Lock"]
    subgraph CS["Atomic critical section"]
        B --> C["Read shared state"]
        C --> D["Modify state"]
        D --> E["Write shared state"]
    end
    E --> F["Release lock"]
    G["Unrelated coroutine work"]

The lock encloses the state transition. The delay remains outside. This is not minimalism for its own sake; it is the smallest boundary that makes the operation indivisible while allowing unrelated work to proceed.

If $P$ is the parallelizable fraction of a workload and $N$ is the number of workers, Amdahl’s Law bounds speedup:

S(N) = 1 / [ (1 - P) + (P / N) ]

Every instruction added to a critical section increases the serialized portion of the workload. The correct lock is not the largest lock. It is the lock that protects the invariant and no more.

6. Compile and challenge the repair

The transformed AST is compiled into in-memory bytecode and executed through adversarial cycles. The candidate is measured against the same interleaving that exposed the race. The observed flakiness falls from 68.0% to 0.0% in 9.07 seconds.

That result is more useful than a green rerun because it has a history: a measured baseline, a named timing boundary, a deterministic reproduction, a concrete patch, and a post-patch execution under the reconstructed schedule.

Why Open Innovation Matters

Flakiness takes more than CI minutes. It taxes attention. It teaches teams to accept uncertainty as routine and to treat a passing rerun as if it were an explanation. The cost is paid in stalled releases, weakened confidence, and engineers repeatedly doing by hand what a machine could establish with evidence.

Concurrency tools should expose their reasoning. A maintainer should be able to inspect the schedule, challenge the causal interpretation, read the AST change, and reproduce the verification. ChaosSloth makes those artifacts part of the investigation instead of asking the user to trust an opaque “fixed” label.

Open innovation matters because the hardest failures do not yield to authority. They yield to scrutiny: a clear model, a reproducible experiment, a repair whose scope can be reviewed, and a result others can question and improve.

The sloth is not slow because it cannot move. It refuses motion without evidence. While the engineer sleeps, the telemetry accumulates, the hypothesis narrows, and the schedule waits—patient as a witness who has been called back to testify.

Installation & Quickstart

Clone the repository and install ChaosSloth in editable mode:

git clone https://github.com/smtherer/ChaosSloth.git
cd ChaosSloth
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Run the analyzer:

chaossloth

Analyze a specific class and method:

chaossloth path/to/file.py --cls SharedSessionStore --method update_session

Run the pytest integration:

pytest --chaossloth

For local AST diagnosis, install Ollama and make the supported Gemma model available. TabPFN-3.5 and Gemma inference run locally; configure Render and Sentry separately for the dashboard and agent tracing.

Prize Categories

Deployment

The live telemetry explorer and dashboard are hosted on Render using render.yaml. The production deployment is available at chaossloth.tech. Sentry traces the agent workflow across TabPFN inversion, Gemma reasoning, and sandbox execution.