Back to Leaderboard

ZCodeglm-5.3

Run ID: zcode-glm-5-3-high-20261024-1237
Time Elapsed
1h 36m
Total wall-clock duration
Estimated Cost
$12.90
Total run cost (USD)
Plan Completion
7/7
100% of checkpoints done
Number of Tests
80/80
100% passed (120.9s)
Number of Commits
8
Git checkpoints recorded

Plan Completion (7/7)

View Stage Specs
  • S1 β€” Completed
    Launch the CLI
  • S2 β€” Completed
    Validate arguments
  • S3 β€” Completed
    Send one bounded GET
  • S4 β€” Completed
    Run a paced five-second loop
  • S5 β€” Completed
    Report outcomes and exit codes
  • S6 β€” Completed
    Report timing
  • S7 β€” Completed
    Handle Ctrl-C

Command / Interface Used

desktop
@PROMPT.md

ZCode desktop app β€’ Run window: 12:37–14:25 (96m active, 108m wall-clock excl. 12m rate-limit pauses) β€’ Session c316b383a14afdcc42b5e71bddca0c69 (account:zai-start-plan / GLM-5.3)

Token & Context Usage

Input (Uncached)
259.5k
Cached Input
44.75M
Output Tokens
204.6k
incl. 119.0k thinking
Total Tokens
45.22M
incl. cached reads
Cache Hit Rate
99.42%
Formula: CacheRead / (input + cacheRead + cacheWrite)
Context Window Usage
271.4k peak (26,846 β†’ 271,444 tokens)

Calls & Tool Breakdown

LLM API Calls
263
completed agent_response steps
Tool Calls
269
253 DONE β€’ 16 ERROR
  • Bash
    140(132 DONE, 8 ERROR)
  • Edit
    56(50 DONE, 6 ERROR)
  • Write
    38(36 DONE, 2 ERROR)
  • Read
    20(20 DONE)
  • TodoWrite
    8(8 DONE)
  • WebFetch
    4(4 DONE)
  • TaskOutput
    2(2 DONE)
  • WebSearch
    1(1 DONE)

Estimated Cost Breakdown

Token volume and estimated cost breakdown comparing 2026 and 2027 pricing rates
ComponentTokensRate (2026)Cost (2026 Rate)Rate (2027)Cost (2027 Rate)
Uncached Input259,545$1.40 / 1M$0.3634 ($0.363363)β€”β€”
Cached Input44,752,960$0.26 / 1M$11.6358 ($11.635770)β€”β€”
Output (incl. thinking)204,560$4.40 / 1M$0.9001 ($0.900064)β€”β€”
Total45,217,065β€”$12.90 ($12.899197)β€”β€”

Slice / Phase Breakdown

Per-stage slice breakdown of step ranges, LLM calls, task polls, thinking tokens, and cost
Slice / PhaseStep RangeLLM Callsmanage_task PollsThinking TokensCost (2026 Rate)
S0: Environment & architecture probing (SIGINT, +Bi, :httpc)1–3837231,426$0.88
S1: Launch CLI (wrapper + escript + --help)39–50120378$0.30
S2: Validate CLI arguments (exit 2)51–681704,768$0.52
S3: Bounded GET (stdlib HTTP/1.1 client, TLS, TestServer)69–15482021,950$3.80
S4: Paced 5-second loop & deterministic scheduler155–1923607,968$2.10
S5: Report HTTP outcomes & exit codes193–2132002,866$1.23
S6: Report elapsed time & exact latency percentiles214–2301602,163$1.03
S7: Handle terminal Ctrl-C via PTY & :erl_signal_server231–2633308,925$2.32
Post-S7: Release verification & final wrap-up264–273100819$0.72

Git Commits (8)

* 284c325  (7 minutes)  <Riza Fahmi>   Release verification: all checks green, Linux untested
* 4aa1f78  (11 minutes)  <Riza Fahmi>   S7: orderly Ctrl-C via pty-tested signal path with exit 130
* dda95cd  (25 minutes)  <Riza Fahmi>   S6: elapsed time and latency percentiles with exact ranks
* 77782e1  (32 minutes)  <Riza Fahmi>   S5: outcome counts by status, failures, and exit codes
* b26fae5  (41 minutes)  <Riza Fahmi>   S4: paced five-second loop with deterministic scheduler
* d5598a1  (55 minutes)  <Riza Fahmi>   S3: bounded GET via stdlib HTTP client with exact deadlines
* b1ff44c  (79 minutes)  <Riza Fahmi>   S2: validate cb arguments with exit 2 and stderr errors
* 2cc1106  (85 minutes)  <Riza Fahmi>   S1: launch cb CLI via wrapper + escript with help flags

Notes & Key Findings

7 findings
  1. Upfront Source-Code Probing Prevented Two Major Architecture Traps (S0)3 points
    • 20.0 Active Minutes of Probing Before S1 (Lines 1–38) The agent spent 37 LLM calls and 51.9k output tokens running empirical probes in /tmp and inspecting local Erlang/OTP 28 source files before touching project code.
    • 1. Uncovering an Erlang :httpc Streaming Blind Spot (Lines 25–37) After writing /tmp/probe_httpc.exs, the agent grepped ~/.nix-profile/lib/erlang/lib/inets-9.6.2.3/src/http_client/httpc_handler.erl and found -define(IS_STREAMED(Code), ((Code =:= 200) orelse (Code =:= 206))) (line 34). It proved that :httpc with stream: :self never delivers the HTTP status code on 200/206 responses ({RequestId, :stream_end, Headers}) and buffers the full body in memory for all other status codes. That led it to build a zero-dependency HTTP/1.1 client over :gen_tcp and :ssl (lib/capy_bench/http/conn.ex and lib/capy_bench/http.ex) with exact monotonic deadlines.
    • 2. Solving BEAM SIGINT & Escript Handling Upfront (Lines 7–23) It verified that OTP 28 rejects :os.set_signal(:sigint, :handle), proved in /tmp that +Bi -noinput disables the BEAM break handler cleanly, and validated replacing :erl_signal_server's default handler via :gen_event.delete_handler/3 + :gen_event.add_handler/3 so a POSIX ./cb wrapper could forward SIGINT as SIGTERM directly to the runner PID.
  2. Freemium Concurrency Limiter Triggered on Sub-85ms Turnarounds (10 Errors)3 points
    • 10 Failed Requests on zai-start-plan (Lines 24, 59, 73, 98, 115, 132, 160, 163, 202, 221) All 10 aborted turns failed with TerminalStreamChunkError: model concurrency limit exceeded on attempt: 1 despite zero parallel LLM calls.
    • Sub-85ms Turnaround Race: In 8 of the 10 errors (lines 59, 73, 98, 132, 160, 163, 202, 221), the preceding tool call was a fast local operation (Edit, Read, or grep) that completed in 21ms to 81ms (0.021s–0.081s). ZCode immediately opened the next SSE stream before the gateway finished releasing the prior stream's concurrency slot.
    • No Automatic Backoff Retry: Because ZCode treated TerminalStreamChunkError as fatal without retrying, each occurrence halted the loop until a manual 'continue' was entered (adding 11.75m / 705.2s of idle wait time excluded from the 96m active duration).
  3. Friction Between ZCode's Stateful File Guards and mix format (27 Python/Sed Mutations)3 points
    • Read-State Invalidated by mix format: ZCode's Edit and Write tools reject modifications if a file's mtime changed since its last Read, while Read rejects re-reading an unchanged file ('Wasted call β€” file unchanged since your last Read', triggered 3 times at lines 62, 159, 194).
    • 6 of 8 Edit/Write Errors from Linter/Formatter Drift Running mix format via Bash updated file mtimes on disk and caused 6 Edit/Write calls (lines 96, 139, 170, 178, 182, 226) to fail with 'File has been modified since read, either by the user or by a linter.'
    • Workaround via Inline Python Patching: To avoid extra Read round-trips after formatting, the agent executed 27 file mutations via python3 - <<'EOF' or sed -i inside Bash so it could patch, run mix format, and run mix test in a single turn.
  4. Extreme Front-Loaded Reasoning Spikes (75.5% of Active Time in LLM Inference)3 points
    • 75.5% LLM vs. 24.5% Tool Split Out of 95.99 active minutes, 72.44 minutes were spent in LLM inference (averaging 47.1 output tokens/sec across 263 calls) and only 23.54 minutes in tool execution.
    • Selective Reasoning: Only 121 of 263 turns (46.0%) emitted reasoningText (~119.0k estimated thinking tokens, or 58.2% of total output), while 142 tactical turns emitted 0 reasoning characters.
    • Top 6 Thinking Turns Took 18.05 Minutes (25% of All LLM Time) Line 5 (S0 plan: 355.0s, 16,763 output tokens), Line 155 (S4 scheduler design: 182.4s, 9,698 output tokens), Line 10 (S0 signal/httpc probes: 171.6s, 8,816 output tokens), Line 37 (S0 inets source analysis: 141.9s, 6,872 output tokens), Line 70 (S3 Conn layer: 118.2s, 6,823 output tokens), and Line 69 (S3 cert & test plan: 114.0s, 5,659 output tokens).
  5. Two Subtle Bugs Caught and Fixed via Systematic Isolation (S3 & S7)2 points
    • 1. The :binary.split/2 Single-Split Trap in HTTP Header Parsing (S3, Lines 111–129) Multi-header responses with Connection: close + Content-Length: 1 timed out after 2s while single-header responses passed. In lib/capy_bench/http.ex, [status_line | header_lines] = :binary.split(head, "\r\n") (without :global) split only once, collapsing all headers into a single 'connection' header with an embedded '\r\ncontent-length: 1' so no framing was recognized and the client waited for EOF. Switching to String.split(head, "\r\n") at line 128 fixed it.
    • 2. Interrupted In-Flight Elapsed-Time Accounting (S7, Lines 242–259) When Ctrl-C arrived at 1.2s during a stalled request that settled at its 2.0s timeout, Elapsed reported ~1.29s instead of ~2.0s. lib/capy_bench/runner.ex initially computed Elapsed from interrupted_at even when an in-flight request was still settling; adding interrupt_in_flight?: state.sched.in_flight (line 258) switched Elapsed to last_settle_at when interrupted mid-flight.
  6. Genuine Slice-by-Slice Discipline & 80-Test Suite (with Some Compiler Warnings)3 points
    • True Incremental Git History: The agent tracked phases with TodoWrite, updated PLAN.md verification notes, and created a genuine commit at the end of each slice (lines 50, 68, 154, 192, 213, 230, 263, 271).
    • Comprehensive 80-Test Suite: Built 80 passing tests (120.9s total runtime), combining fast deterministic unit tests for the pure scheduler (test/scheduler_test.exs), HTTP framing/TLS tests (test/http_test.exs), and end-to-end PTY/CLI integration tests, though leaving a few minor compiler warnings (unused import, etc.).
    • 99.42% Cache Hit Rate Saved $51.02 (79.8%) Out of 45,012,505 total input tokens across 263 calls, 44,752,960 were served from cache with zero cache invalidations as context grew from 26.8k to 271.4k tokens (reducing cost from $63.92 uncached to $12.90).
  7. Training-Data Fingerprint: GLM-5.3 Signed Every Commit as 'Claude Fable 5'2 points
    • Unprompted Commit Trailer: Even though the system prompt explicitly identified the harness and model as 'ZCode' and 'account:zai-start-plan/GLM-5.3'β€”and neither 'Claude' nor 'Co-Authored-By' appeared anywhere in the prompt, tool schemas, or user messages (lines 1–49 had zero matches)β€”at line 50 the model spontaneously signed its first git commit with 'Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>'.
    • Repeated Across All 8 Commits: Every subsequent commit (lines 68, 154, 192, 213, 230, 263, 271) included the same 'Claude Fable 5' co-author trailer, triggered by ZCode's Claude-Code-style system prompt and tool schemas.