Back to Leaderboard
ZCodeglm-5.3
Run ID:
zcode-glm-5-3-high-20261024-1237- Time Elapsed
- 1h 36m
- Total wall-clock duration
- Estimated Cost
- $12.90
- Total run cost (USD)
- Plan Completion
- 7/7
- 100% of checkpoints done
- Number of Tests
- 80/80
- 100% passed (120.9s)
- Number of Commits
- 8
- Git checkpoints recorded
Plan Completion (7/7)
View Stage Specs- S1 β CompletedLaunch the CLI
- S2 β CompletedValidate arguments
- S3 β CompletedSend one bounded GET
- S4 β CompletedRun a paced five-second loop
- S5 β CompletedReport outcomes and exit codes
- S6 β CompletedReport timing
- S7 β CompletedHandle Ctrl-C
Command / Interface Used
desktop@PROMPT.mdZCode desktop app β’ Run window: 12:37β14:25 (96m active, 108m wall-clock excl. 12m rate-limit pauses) β’ Session c316b383a14afdcc42b5e71bddca0c69 (account:zai-start-plan / GLM-5.3)
Token & Context Usage
- Input (Uncached)
- 259.5k
- Cached Input
- 44.75M
- Output Tokens
- 204.6k
- incl. 119.0k thinking
- Total Tokens
- 45.22M
- incl. cached reads
- Cache Hit Rate
- 99.42%
- Formula:
CacheRead / (input + cacheRead + cacheWrite) - Context Window Usage
- 271.4k peak (26,846 β 271,444 tokens)
Calls & Tool Breakdown
- LLM API Calls
- 263
- completed agent_response steps
- Tool Calls
- 269
- 253 DONE β’ 16 ERROR
- Bash140(132 DONE, 8 ERROR)
- Edit56(50 DONE, 6 ERROR)
- Write38(36 DONE, 2 ERROR)
- Read20(20 DONE)
- TodoWrite8(8 DONE)
- WebFetch4(4 DONE)
- TaskOutput2(2 DONE)
- WebSearch1(1 DONE)
Estimated Cost Breakdown
| Component | Tokens | Rate (2026) | Cost (2026 Rate) | Rate (2027) | Cost (2027 Rate) |
|---|---|---|---|---|---|
| Uncached Input | 259,545 | $1.40 / 1M | $0.3634 ($0.363363) | β | β |
| Cached Input | 44,752,960 | $0.26 / 1M | $11.6358 ($11.635770) | β | β |
| Output (incl. thinking) | 204,560 | $4.40 / 1M | $0.9001 ($0.900064) | β | β |
| Total | 45,217,065 | β | $12.90 ($12.899197) | β | β |
Slice / Phase Breakdown
| Slice / Phase | Step Range | LLM Calls | manage_task Polls | Thinking Tokens | Cost (2026 Rate) |
|---|---|---|---|---|---|
| S0: Environment & architecture probing (SIGINT, +Bi, :httpc) | 1β38 | 37 | 2 | 31,426 | $0.88 |
| S1: Launch CLI (wrapper + escript + --help) | 39β50 | 12 | 0 | 378 | $0.30 |
| S2: Validate CLI arguments (exit 2) | 51β68 | 17 | 0 | 4,768 | $0.52 |
| S3: Bounded GET (stdlib HTTP/1.1 client, TLS, TestServer) | 69β154 | 82 | 0 | 21,950 | $3.80 |
| S4: Paced 5-second loop & deterministic scheduler | 155β192 | 36 | 0 | 7,968 | $2.10 |
| S5: Report HTTP outcomes & exit codes | 193β213 | 20 | 0 | 2,866 | $1.23 |
| S6: Report elapsed time & exact latency percentiles | 214β230 | 16 | 0 | 2,163 | $1.03 |
| S7: Handle terminal Ctrl-C via PTY & :erl_signal_server | 231β263 | 33 | 0 | 8,925 | $2.32 |
| Post-S7: Release verification & final wrap-up | 264β273 | 10 | 0 | 819 | $0.72 |
Git Commits (8)
* 284c325 (7 minutes) <Riza Fahmi> Release verification: all checks green, Linux untested
* 4aa1f78 (11 minutes) <Riza Fahmi> S7: orderly Ctrl-C via pty-tested signal path with exit 130
* dda95cd (25 minutes) <Riza Fahmi> S6: elapsed time and latency percentiles with exact ranks
* 77782e1 (32 minutes) <Riza Fahmi> S5: outcome counts by status, failures, and exit codes
* b26fae5 (41 minutes) <Riza Fahmi> S4: paced five-second loop with deterministic scheduler
* d5598a1 (55 minutes) <Riza Fahmi> S3: bounded GET via stdlib HTTP client with exact deadlines
* b1ff44c (79 minutes) <Riza Fahmi> S2: validate cb arguments with exit 2 and stderr errors
* 2cc1106 (85 minutes) <Riza Fahmi> S1: launch cb CLI via wrapper + escript with help flagsNotes & Key Findings
7 findingsUpfront Source-Code Probing Prevented Two Major Architecture Traps (S0)
- 20.0 Active Minutes of Probing Before S1 (Lines 1β38) The agent spent 37 LLM calls and 51.9k output tokens running empirical probes in
/tmpand inspecting local Erlang/OTP 28 source files before touching project code. - 1. Uncovering an Erlang
:httpcStreaming Blind Spot (Lines 25β37) After writing/tmp/probe_httpc.exs, the agent grepped ~/.nix-profile/lib/erlang/lib/inets-9.6.2.3/src/http_client/httpc_handler.erland found -define(IS_STREAMED(Code), ((Code =:= 200) orelse (Code =:= 206))) (line 34). It proved that:httpcwith stream::selfnever delivers the HTTP status code on 200/206 responses ({RequestId,:stream_end, Headers}) and buffers the full body in memory for all other status codes. That led it to build a zero-dependency HTTP/1.1 client over:gen_tcpand:ssl(lib/capy_bench/http/conn.exandlib/capy_bench/http.ex) with exact monotonic deadlines. - 2. Solving BEAM SIGINT & Escript Handling Upfront (Lines 7β23) It verified that OTP 28 rejects
:os.set_signal(:sigint, :handle), proved in/tmpthat+Bi-noinput disables the BEAM break handler cleanly, and validated replacing:erl_signal_server's default handler via:gen_event.delete_handler/3+:gen_event.add_handler/3so a POSIX./cbwrapper could forward SIGINT as SIGTERM directly to the runner PID.
- 20.0 Active Minutes of Probing Before S1 (Lines 1β38) The agent spent 37 LLM calls and 51.9k output tokens running empirical probes in
Freemium Concurrency Limiter Triggered on Sub-85ms Turnarounds (10 Errors)
- 10 Failed Requests on zai-start-plan (Lines 24, 59, 73, 98, 115, 132, 160, 163, 202, 221) All 10 aborted turns failed with TerminalStreamChunkError: model concurrency limit exceeded on
attempt: 1despite zero parallel LLM calls. - Sub-85ms Turnaround Race: In 8 of the 10 errors (lines 59, 73, 98, 132, 160, 163, 202, 221), the preceding tool call was a fast local operation (Edit, Read, or grep) that completed in 21ms to 81ms (0.021sβ0.081s). ZCode immediately opened the next SSE stream before the gateway finished releasing the prior stream's concurrency slot.
- No Automatic Backoff Retry: Because ZCode treated TerminalStreamChunkError as fatal without retrying, each occurrence halted the loop until a manual 'continue' was entered (adding 11.75m / 705.2s of idle wait time excluded from the 96m active duration).
- 10 Failed Requests on zai-start-plan (Lines 24, 59, 73, 98, 115, 132, 160, 163, 202, 221) All 10 aborted turns failed with TerminalStreamChunkError: model concurrency limit exceeded on
Friction Between ZCode's Stateful File Guards and
mix format(27 Python/Sed Mutations)- Read-State Invalidated by
mix format: ZCode's Edit and Write tools reject modifications if a file's mtime changed since its last Read, while Read rejects re-reading an unchanged file ('Wasted call β file unchanged since your last Read', triggered 3 times at lines 62, 159, 194). - 6 of 8 Edit/Write Errors from Linter/Formatter Drift Running
mix formatvia Bash updated file mtimes on disk and caused 6 Edit/Write calls (lines 96, 139, 170, 178, 182, 226) to fail with 'File has been modified since read, either by the user or by a linter.' - Workaround via Inline Python Patching: To avoid extra Read round-trips after formatting, the agent executed 27 file mutations via python3 - <<'EOF' or sed -i inside Bash so it could patch, run
mix format, and runmix testin a single turn.
- Read-State Invalidated by
Extreme Front-Loaded Reasoning Spikes (75.5% of Active Time in LLM Inference)
- 75.5% LLM vs. 24.5% Tool Split Out of 95.99 active minutes, 72.44 minutes were spent in LLM inference (averaging 47.1 output tokens/sec across 263 calls) and only 23.54 minutes in tool execution.
- Selective Reasoning: Only 121 of 263 turns (46.0%) emitted
reasoningText(~119.0k estimated thinking tokens, or 58.2% of total output), while 142 tactical turns emitted 0 reasoning characters. - Top 6 Thinking Turns Took 18.05 Minutes (25% of All LLM Time) Line 5 (S0 plan: 355.0s, 16,763 output tokens), Line 155 (S4 scheduler design: 182.4s, 9,698 output tokens), Line 10 (S0 signal/httpc probes: 171.6s, 8,816 output tokens), Line 37 (S0 inets source analysis: 141.9s, 6,872 output tokens), Line 70 (S3 Conn layer: 118.2s, 6,823 output tokens), and Line 69 (S3 cert & test plan: 114.0s, 5,659 output tokens).
Two Subtle Bugs Caught and Fixed via Systematic Isolation (S3 & S7)
- 1. The
:binary.split/2Single-Split Trap in HTTP Header Parsing (S3, Lines 111β129) Multi-header responses with Connection: close +Content-Length: 1 timed out after 2s while single-header responses passed. Inlib/capy_bench/http.ex, [status_line | header_lines] =:binary.split(head, "\r\n")(without:global) split only once, collapsing all headers into a single 'connection' header with an embedded '\r\ncontent-length: 1' so no framing was recognized and the client waited for EOF. Switching toString.split(head, "\r\n")at line 128 fixed it. - 2. Interrupted In-Flight Elapsed-Time Accounting (S7, Lines 242β259) When Ctrl-C arrived at 1.2s during a stalled request that settled at its 2.0s timeout, Elapsed reported ~1.29s instead of ~2.0s.
lib/capy_bench/runner.exinitially computed Elapsed from interrupted_at even when an in-flight request was still settling; adding interrupt_in_flight?: state.sched.in_flight (line 258) switched Elapsed to last_settle_at when interrupted mid-flight.
- 1. The
Genuine Slice-by-Slice Discipline & 80-Test Suite (with Some Compiler Warnings)
- True Incremental Git History: The agent tracked phases with TodoWrite, updated
PLAN.mdverification notes, and created a genuine commit at the end of each slice (lines 50, 68, 154, 192, 213, 230, 263, 271). - Comprehensive 80-Test Suite: Built 80 passing tests (120.9s total runtime), combining fast deterministic unit tests for the pure scheduler (
test/scheduler_test.exs), HTTP framing/TLS tests (test/http_test.exs), and end-to-end PTY/CLI integration tests, though leaving a few minor compiler warnings (unused import, etc.). - 99.42% Cache Hit Rate Saved $51.02 (79.8%) Out of 45,012,505 total input tokens across 263 calls, 44,752,960 were served from cache with zero cache invalidations as context grew from 26.8k to 271.4k tokens (reducing cost from $63.92 uncached to $12.90).
- True Incremental Git History: The agent tracked phases with TodoWrite, updated
Training-Data Fingerprint: GLM-5.3 Signed Every Commit as 'Claude Fable 5'
- Unprompted Commit Trailer: Even though the system prompt explicitly identified the harness and model as 'ZCode' and 'account
:zai-start-plan/GLM-5.3'βand neither 'Claude' nor 'Co-Authored-By' appeared anywhere in the prompt, tool schemas, or user messages (lines 1β49 had zero matches)βat line 50 the model spontaneously signed its first git commit with 'Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>'. - Repeated Across All 8 Commits: Every subsequent commit (lines 68, 154, 192, 213, 230, 263, 271) included the same 'Claude Fable 5' co-author trailer, triggered by ZCode's Claude-Code-style system prompt and tool schemas.
- Unprompted Commit Trailer: Even though the system prompt explicitly identified the harness and model as 'ZCode' and 'account