Back to Leaderboard
ZCodeglm-5.3-flash
Run ID:
zcode-glm-5-3-flash-low-20261024-1852- Time Elapsed
- 1h 51m
- Total wall-clock duration
- Estimated Cost
- $1.10
- Total run cost (USD)
- Plan Completion
- 6/7
- 86% of checkpoints done
- Number of Tests
- 42/42
- 100% passed (41.4s)
- Number of Commits
- 8
- Git checkpoints recorded
Plan Completion (6/7)
View Stage Specs- S1 β CompletedLaunch the CLI
- S2 β CompletedValidate arguments
- S3 β CompletedSend one bounded GET
- S4 β CompletedRun a paced five-second loop
- S5 β IncompleteReport outcomes and exit codes
- S6 β CompletedReport timing
- S7 β CompletedHandle Ctrl-C
Command / Interface Used
desktop@PROMPT.mdZCode desktop app β’ Run window: 18:52β20:44 (111m active, 113m wall-clock excl. 1.5m manual 'continue' pauses) β’ Session 17499dedaeacc8c708d4a85a7cca7242 (account:zai-start-plan / GLM-5.3-Flash, effort: low)
Token & Context Usage
- Input (Uncached)
- 164.6k
- Cached Input
- 34.21M
- Output Tokens
- 103.0k
- incl. 42.2k thinking
- Total Tokens
- 34.48M
- incl. cached reads
- Cache Hit Rate
- 99.52%
- Formula:
CacheRead / (input + cacheRead + cacheWrite) - Context Window Usage
- 167.7k peak (26,398 β 167,669 tokens)
Calls & Tool Breakdown
- LLM API Calls
- 344
- completed agent_response steps
- Tool Calls
- 364
- 346 DONE β’ 18 ERROR
- Bash222(219 DONE, 3 ERROR)
- Edit94(80 DONE, 14 ERROR)
- Read21(21 DONE)
- Write20(19 DONE, 1 ERROR)
- TodoWrite7(7 DONE)
Estimated Cost Breakdown
| Component | Tokens | Rate (2026) | Cost (2026 Rate) | Rate (2027) | Cost (2027 Rate) |
|---|---|---|---|---|---|
| Uncached Input | 164,637 | $0.15 / 1M | $0.0247 ($0.024696) | β | β |
| Cached Input | 34,210,176 | $0.03 / 1M | $1.0263 ($1.026305) | β | β |
| Output (incl. thinking) | 102,973 | $0.50 / 1M | $0.0515 ($0.051486) | β | β |
| Total | 34,477,786 | β | $1.10 ($1.102487) | β | β |
Slice / Phase Breakdown
| Slice / Phase | Step Range | LLM Calls | manage_task Polls | Thinking Tokens | Cost (2026 Rate) |
|---|---|---|---|---|---|
| S1: Launch CLI (cb wrapper + escript + --help + Ctrl-C probe) | 1β35 | 30 | 0 | 9,362 | $0.04 |
| S2: Validate CLI arguments before any request (exit 2) | 36β66 | 30 | 0 | 1,226 | $0.05 |
| S3: Bounded streaming GET (TLS verification, timeouts, TestServer) | 67β245 | 156 | 0 | 14,008 | $0.43 |
| S4: Paced five-second loop with timeout cancellation | 246β271 | 26 | 0 | 8,187 | $0.11 |
| S5: Report outcome counts and exit 0/1 by completions | 272β290 | 19 | 0 | 588 | $0.08 |
| S6: Report elapsed time and completed-response p50/p95 | 291β317 | 27 | 0 | 1,264 | $0.12 |
| S7: Handle Ctrl-C via wrapper SIGTERM relay & exit 130 | 318β361 | 44 | 0 | 6,755 | $0.21 |
| Post-S7: Release verification & per-reason failure diagnostics | 362β373 | 12 | 0 | 765 | $0.06 |
Git Commits (8)
* 644cc85 (8 minutes) <Riza Fahmi> Release verification: record verified checks and failure diagnostics
* faaa62c (12 minutes) <Riza Fahmi> S7: handle Ctrl-C via wrapper SIGTERM relay with partial summary and exit 130
* 31c83ba (23 minutes) <Riza Fahmi> S6: report elapsed time and completed-response p50/p95
* c70ac56 (31 minutes) <Riza Fahmi> S5: report outcome counts and exit 0/1 by completions
* 96908f5 (37 minutes) <Riza Fahmi> S4: paced five-second loop with timeout cancellation
* 24c538b (45 minutes) <Riza Fahmi> S3: bounded streaming GET with TLS verification, timeouts, no redirects or retries
* fe53347 (2 hours) <Riza Fahmi> S2: validate CLI arguments before any request
* 8e68e2a (2 hours) <Riza Fahmi> S1: launch CLI with cb escript wrapper and help outputNotes & Key Findings
8 findings91.5% Cost Reduction vs. GLM-5.3 High ($1.10 vs. $12.90) with 6/7 Slices Verified (S5 Incomplete on Live Chunked 301)
- $1.10 Total Run Cost: Running GLM-5.3-Flash with effort: "low" completed the benchmark workflow for $1.10 ($1.1025)βa 91.5% cost reduction compared to GLM-5.3 High ($12.90) on the same ZCode harness.
- 99.52% Prefix Cache Hit Rate Saved $4.11 (78.8%) Out of 34,374,813 total input tokens across 344 successful turns, 34,210,176 were served from cache (
cacheReadTokens) with zero cache invalidations as context grew monotonically from 26,398 to 167,669 tokens. - Outside-In TDD Suite (42/42 Local Tests), but S5 Fails Against Live Chunked 301 Responses While the agent's 42 local ExUnit tests passed in 41.4s with zero warnings under
mix compile --warnings-as-errorsandmix credo --strict, manual S5 verification with./cb http://rizafahmi.comfailed due to a chunked transfer-encoding buffer bug inlib/capy_bench/http.ex.
S5 Verification Failure on '
./cb http://rizafahmi.com': Chunked-Body Buffer Drop Causes ':closed' Transport Failures- Failing Live 301 Redirect Check: Running
./cb http://rizafahmi.com(which returns a Cloudflare HTTP/1.1 301 Moved Permanently response withTransfer-Encoding: chunked) fails 40 of 41 requests with Transport failures: 40 (Failure reasons::closed: 40 on stderr) instead of counting completed 301 responses. - Root Cause in
CapyBench.HTTP.read_chunk/5(lib/capy_bench/http.ex) When reading a chunked body, read_chunk/5 calls read_n(socket, transport,String.to_integer(line, 16)+ 2, rest, deadline), which consumes the chunk payload from rest and returns bare:okβdiscarding the remaining bytes in rest (including the terminating 0\r\n\r\n chunk that arrived in the same TCP segment)βbefore callingread_chunked(socket, transport, "", deadline), which then blocks on transport.recv/3 until the server closes the connection ({:error,:closed}). - Skipped TodoWrite Updates on S5 & S6: In ZCode's TodoWrite state, the agent marked S5 in_progress at line 273 and never updated TodoWrite after committing S5 (line 290) or S6 (line 317), leaving S5 marked incomplete in the UI todo list until line 362.
- Failing Live 301 Redirect Check: Running
Upstream Stream Instability in S1βS3: 29 'TypeError: terminated' Aborts & 58 Multi-Attempt Retries (~21m Lost)
- 29 Mid-Stream SSE Aborts (10.58m Wasted): Between 18:52 and 20:00 (lines 1β213), 29 LLM responses aborted mid-stream after running for 9.4s to 67.0s each with TypeError: terminated inside Node's undici fetch stream, including a 4-error streak at lines 81β84 that burned 166.1s (2.8m) on a single turn.
- 58 Multi-Attempt Auto-Retries (~10.2m Unlogged Backoff) Across lines 1β223, 58 turns required internal retries before succeeding (46 on attempt 2, 10 on attempt 3, and 2 on attempt 4 at lines 80 and 145). On the 46 retry turns without a preceding error row, inter-turn gaps after fast local tools totaled 610.2s (10.17m) of unlogged failed-attempt and backoff latency.
- Complete Recovery After Line 224: Only 2 stream drops required manual 'continue' nudges (lines 102 and 213, plus 1 after a cancelled Bash run at line 358, totaling 87.2s / 1.45m of idle time). From line 224 (20:04) through line 373 (20:44), all 150 final turns across S3βS7 succeeded on attempt 1 with zero stream terminations.
S3 Consumed 46.5% of Active Time (51.7m, 156 LLM Calls) Due to Five OTP/GenServer Traps
- 1. macOS Keychain Ignores
SSL_CERT_FILE(Lines 67β85, ~8m) The agent spent 15 turns probing:public_key.cacerts_get()in/tmpbefore confirming that OTP on macOS reads the system Keychain and ignoresSSL_CERT_FILE, requiring explicit:public_key.cacerts_load(path)whenSSL_CERT_FILEis set. - 2. Misplacing extra_applications in
mix.exs(Lines 127β146, ~7m) At line 128, it added extra_applications: [:logger,:ssl,:public_key,:crypto] inside def project instead of def application, then spent 18 turns investigating why:code.lib_dir(:ssl)returned {:error,:bad_name} under Mix before spotting itsmix.exsplacement bug at line 145. - 3. Four TestServer GenServer Bugs (Lines 103β176, ~22m) In
test/support/test_server.ex, it sequentially debugged (a) handlingGenServer.call(:request)inside handle_info instead of handle_call (line 117), (b) blocking the GenServer inside:gen_tcp.accept/1so workerGenServer.callmessages timed out (line 149), (c) callingself()inside spawn(fn -> accept(...,self()) end) so workers messaged the acceptor PID instead of the GenServer (line 160), and (d) usingGenServer.start_link/2inTestServer.start/1which crashed ExUnit teardown afterTestServer.stop/1(line 176). - 4. Calling Removed
:ssl.accept/2on OTP 28 (Lines 183β198, ~4m) HTTPS test server connections crashed with UndefinedFunctionError on:ssl.accept/2until replaced with:ssl.transport_accept/1+:ssl.handshake/2at line 195.
- 1. macOS Keychain Ignores
5.5 MB
erl_crash.dumpCommitted Four Times via 'git add -A' (877k Diff Lines)- Broken Pipe Crash Dumps from head/tail Pipelines Piping Mix output into head or tail (such as
mix compile2>&1 | head -8 at line 295) closed stdout early while the BEAM was still writing to:standard_error, triggering an unhandled:badarg(:epipe) during VM termination and writing a 5.5 MBerl_crash.dumpto the repo root. - Unignored Artifact Staged by
git add -A: Because.gitignoreonly excluded /_build/, /deps/, and /cb.escript (line 31), runninggit add -A && git commit -qcommittederl_crash.dumpin S1 (8e68e2a, +215,252 lines) and updated it in S3 (24c538b, 284,165 lines changed), S6 (31c83ba, 205,614 lines changed), and S7 (faaa62c, 272,280 lines changed).
- Broken Pipe Crash Dumps from head/tail Pipelines Piping Mix output into head or tail (such as
S7 Signal-Relay & POSIX 'wait' Exit-Status Debugging
- Upfront S1 Probe (Lines 14β27): In
/tmp/cbprobe, the agent verified that OTP 28 rejects:os.set_signal(:sigint, :handle)and default escripts ignore SIGINT, then validated swapping:erl_signal_server's handler via:gen_event.swap_handler/3to catch:sigtermfrom a./cbwrapper. - Two S7 Integration Bugs Caught Live (Lines 342β353) When testing real Ctrl-C in S7, it fixed (1)
CapyBench.SignalHandler.init/1crashing because:gen_event.swap_handler/3passes {init_arg, term} rather than bare init_arg (line 344), and (2) the./cbwrapper's wait loop treating the escript's legitimate exit code 130 (> 128) as an interrupted wait builtin and re-invoking wait $child, which returned 127 (fixed at lines 346β349 via kill -0).
- Upfront S1 Probe (Lines 14β27): In
ZCode File-State Guard Friction (18 Inline Python Patches & 1 Binary Overwrite Error)
- 8 of 14 Edit Errors from Stale Read State: Running
mix formator inline scripts invalidated ZCode's file-mtime tracker, causing 8 Edit calls (lines 174, 185, 204, 226, 236, 313, 321, 338) to fail with 'File has been modified since read' and 1 Read call (line 152) to be rejected as a 'Wasted call'. To avoid extra Read turns, the agent executed 18 file mutations via python3 - <<'EOF' inside Bash. - Binary Escript Overwrite Guard (Line 28): When converting
./cbfrom a compiled escript binary to a POSIX shell wrapper in S1, ZCode's Write tool rejected overwriting./cbwith 'Unsupported or binary text encoding', prompting the agent to write the wrapper via cat > cb <<'EOF' in Bash (line 29).
- 8 of 14 Edit Errors from Stale Read State: Running
Low-Effort Reasoning Profile & Clean Git Commit Metadata (No Hallucinated Co-Author)
- Selective Reasoning on effort: "low": Only 140 of 344 successful turns (40.7%) emitted
reasoningText(~42.2k estimated thinking tokens, or 40.9% of the 103.0k output tokens), while 204 tactical turns emitted 0 reasoning characters. Architectural turns still scaled up thinking when needed, led by line 5 (S1 plan: 143.8s, 5,474 output tokens) and line 246 (S4 runner: 102.2s, 4,245 output tokens). - No Hallucinated Co-Author Trailers: Unlike GLM-5.3 High (which spontaneously appended 'Co-Authored-By: Claude Fable 5' to all 8 commits on the same ZCode harness), GLM-5.3-Flash Low committed cleanly with single-line
git commit -q -mmessages and zero co-author trailers.
- Selective Reasoning on effort: "low": Only 140 of 344 successful turns (40.7%) emitted