Metadata-Version: 2.1
Name: clew-custos
Version: 0.2.0
Summary: Detect wasteful cycles in multi-agent traces (Custos / Clew, S0 MVP).
Author: Custos
Project-URL: Homepage, https://github.com/JEONSEWON/Clew-by-Custos
Project-URL: Repository, https://github.com/JEONSEWON/Clew-by-Custos
Project-URL: Issues, https://github.com/JEONSEWON/Clew-by-Custos/issues
Keywords: agent,observability,llm,tokens,waste,tracing,opentelemetry,claude-code,cost,deterministic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Provides-Extra: adapter
Requires-Dist: langgraph>=0.2; extra == "adapter"
Requires-Dist: langchain-core>=0.3; extra == "adapter"
Requires-Dist: openinference-instrumentation-langchain>=0.1; extra == "adapter"
Requires-Dist: opentelemetry-sdk>=1.25; extra == "adapter"
Requires-Dist: opentelemetry-api>=1.25; extra == "adapter"
Provides-Extra: all
Requires-Dist: clew-custos[adapter,detect,semantic]; extra == "all"
Provides-Extra: detect
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: semantic
Requires-Dist: sentence-transformers==5.5.1; extra == "semantic"
Requires-Dist: transformers==5.0.0; extra == "semantic"
Requires-Dist: torch>=2.0; extra == "semantic"
Requires-Dist: huggingface-hub>=0.20; extra == "semantic"

# Clew

**Find where your coding agent wastes work — which file, which turn, why.**

```bash
pip install "clew-custos[detect]"   # published as clew-custos, not clew
python -m clew analyze ~/.claude/projects/<slug>/<uuid>.jsonl --out report.md
```

> Package name is **`clew-custos`** (the bare name `clew` on PyPI is an unrelated placeholder). The module still imports as `clew`.

Real excerpt from a public Claude Code session (`09d9abe9`, 258 turns; local path abbreviated, numbers unchanged):

```
Result: WASTE DETECTED

### 1. requery — Read on `.../boot.ts`
- turns: turn 50 → re-run at turn 58 (of 258 total)
- state: No modification of this file in between — re-read output is unchanged.
- re-consumed across 200 subsequent turns (≈439 tokens/turn → 87800 amplification tokens)
- estimated cost impact: $0.026340 ~ $0.263400 (cache-hit to cache-miss)
```

Deterministic, no LLM in the loop. Every span in the session was `200 OK`; the trace stayed green. Clew reads the finished session and points at the redundant step.

---

## Why diagnosis (and not another dashboard)

Observability tools (Langfuse, Phoenix, LangSmith) **show you the trace**.
Clew **tells you which spans are waste, and why** — the exact file, the turns, whether the file was modified in between.

**Clew diagnoses; it does not fix.** The output is a report you read. What to change in your agent — prompt, context caching, tool routing — is a call only you can make.

Scope is deliberately narrow: **one working pattern (`repeat` / `requery`)**, done precisely. A second pattern for reasoning-level ping-pong (`pingpong`) is implemented but has not fired on any trace format we've validated (tool-span only) — we don't advertise what we haven't observed.

---

## How it works

A two-stage cascade, fully deterministic:

1. **Structural gate** — group steps by `(node, normalized input)`; a group with ≥ 2 occurrences is a candidate.
2. **Identity gate** — require `sha256(output_A) == sha256(output_B)`. If outputs differ (state changed, a retry succeeded where one failed), it is **not** flagged.

Frozen parameters (never hand-tuned): `phi = 0.514345`, `N = 2`, embedding model `paraphrase-multilingual-MiniLM-L12-v2`, pinned to a git tag with a manifest `sha256`. `N = 2` began as an arbitrary default; we later verified it is F1-optimal across `N ∈ {2, 3, 5, ∞}` on RedundancyBench (F1 decreases monotonically as N grows).

A report entry also carries a per-file state check — "no modification in between" vs "**File was modified in between** — may be a legitimate re-read" — so you can tell forced re-reads (waste) from legitimate ones. Tool-error responses (`is_error: true` in Anthropic tool_result) are excluded from waste with an explicit count in the report.

---

## Where it stands

### RedundancyBench — labeled ground truth

Human-labeled benchmark for redundant-step detection ([arXiv:2605.29893](https://arxiv.org/abs/2605.29893)):

| | Clew (deterministic) | Best method in the paper (LLM-as-judge) |
|---|---|---|
| **Step-level F1** | **0.2642** | 0.2488 |
| Precision | 0.826 | — |

Same `evaluate.py` (imported directly from their repo), same scope (all four redundancy categories). Clew surpasses the paper's best reported method **without a single model call**.

**Read this honestly — three caveats:**

1. **0.2488 is the paper's reported number, not something we re-ran.** The repo ships the scorer we import, but not the baseline's prediction files, so we can't reproduce 24.88% in our environment. The scorer is identical; the comparison point is **cited, not reproduced**.
2. **This is one benchmark.** A 0.0154 F1 margin on a single dataset is a signal, not a verdict. Recall is low (0.157) — Clew catches the one kind of waste it's built for and deliberately ignores the rest.
3. **Precision 0.826 may be a lower bound.** Of the 22 false-positive spans, 21 were exact input-and-output repeats that no annotator labeled under any category, and 6 had zero state change in between — i.e. redundancies the annotators appear to have missed. Owner review pending; we don't claim all 22 are waste.

### trace-commons — 28 real public Claude Code sessions

Public dataset ([trace-commons/agent-traces](https://huggingface.co/datasets/trace-commons/agent-traces)), full scan on 2026-07-19:

- **28 / 28** sessions processed, **0 crashes**.
- **10 / 28** flagged as wasteful (34 waste spans in the cascade output; 32 kept after the tool-error gate).
- Aggregate saving potential (across all wasteful sessions): **$1.01 ~ $10.12** (cache-hit lower to cache-miss upper).
- Per-session range: $0 (no waste) up to $0.64 ~ $6.40 (one session, 18 waste spans).

**Honest scope:** trace-commons has **no step-level ground truth** — the 34 spans are cascade-flagged candidates plus a state-change check per file, not annotator-verified waste. Precision lives on RedundancyBench; scale on real data lives here.

### Toolathlon — 7,116 trajectories, cross-model scale

Beyond labeled and real-user data, Clew ran unmodified over **Toolathlon** — 22 frontier models × 3 runs ([arXiv:2510.25726](https://arxiv.org/abs/2510.25726), CC-BY-4.0):

- **8,042 waste candidates** flagged across 176,270 tool spans, in 32 seconds.
- Candidate density varies **54× across models** (0.157 – 8.463 per trajectory).

**Honest scope:** Toolathlon ships only pass/fail labels, not step-level ground truth. These are **candidates**, not verified waste — "model X wastes 54× more than model Y" would over-claim (no labels; task-mix and success-rate confounds uncontrolled).

---

## Cost estimation

Every wasted step in a Claude Code trace has a knock-on cost: the stale tool result stays in the trajectory and is **re-consumed as input on every subsequent turn**. Naive single-consumption math makes waste look trivial (~$0.001); the real number is the wasted output *times the remaining turns*.[^cache]

Formula per waste span:

```
amp_tokens_i  = waste_tokens_i × turns_after_i
lower_i (USD) = amp_tokens_i × cache_read_price  (fully cached re-consumption)
upper_i (USD) = amp_tokens_i × base_input_price  (uncached re-consumption)
```

Where `waste_tokens_i` comes directly from the vendor's `cache_creation_input_tokens` field on the assistant turn immediately after the waste, and `turns_after_i` is the number of assistant turns from the waste to the end of the session. Retry cascades where `prev.cache == next.cache` are filtered out with an explicit count.

**Read the estimate honestly:**

- It is **estimated saving potential, not measured cost.** The formula assumes the wasted output is re-consumed each subsequent turn (a structural upper bound).
- The range spans cache-hit (lower) to cache-miss (upper); the exact split is not observable from Anthropic usage.
- **Claude Code sessions only.** Other adapters (OTel, OpenInference, Toolathlon) still detect waste, but do not populate the cache-token fields the amplification calculator needs. Those reports show waste-detected without a dollar figure.
- Attribution assumes Sonnet 4.5 pricing.
- Clew reports **detected waste**, not intervention. Independent research on removing
  redundant trajectory content — e.g. AgentDiet[^agentdiet] — reports downstream
  savings in a different setup (21.1–35.9% total cost, 39.9–59.7% input tokens on two
  coding benchmarks); those are that paper's numbers, not Clew's, and are cited only
  to flag the detection→intervention gap that Clew's report leaves for the user.

[^cache]: Prompt-cache economics in long-horizon agent workloads — including the
41–80% cost reduction range from strategic cache placement — are measured empirically
in Lumer et al., *Don't Break the Cache*
([arXiv:2601.06007](https://arxiv.org/abs/2601.06007)). Those numbers describe optimal
caching, not Clew's estimator; Clew's `lower_i`–`upper_i` bracket assumes the wasted
output is re-consumed and prices it under the observed cache-hit vs cache-miss split.

[^agentdiet]: Xiao et al., *Reducing Cost of LLM Agents with Trajectory Reduction
(AgentDiet)*, [arXiv:2509.23586](https://arxiv.org/abs/2509.23586).

---

## Reads your existing traces

Clew is a **complement, not a replacement.** It doesn't store or visualize traces — it reads what your observability stack already produces and points at the waste. It also runs standalone on raw session logs.

Auto-detected input formats:

| Source | Detected by |
|---|---|
| Claude Code session logs (`.jsonl`) | `sessionId` |
| OpenTelemetry SDK JSON | `context` |
| OpenInference (Phoenix / TRAIL lineage) | `span_id` / nested `child_spans` |
| Clew native trace JSON | `trace_id` + `spans` |
| Toolathlon trajectories | `modelname_run` + `task_status` |
| RedundancyBench | `tasks` + `simulations` |

Because it ingests OpenTelemetry and OpenInference, it can read traces from Langfuse, Arize Phoenix, and any OTel-instrumented agent. *(OTLP protobuf-JSON is not yet supported; the error message points you to the SDK-JSON conversion.)*

---

## How we keep ourselves honest

This repo treats anti-self-deception as a working discipline, not a slogan:

- **Pre-registration.** Every detection change is committed *before* results are run, so the prediction carries an external timestamp. Predictions and stop-conditions are written first and not edited after seeing results.
- **Frozen parameters.** `phi`, `N`, and the embedding model are pinned to a git tag; changing them requires a documented recalibration, never a post-hoc nudge.
- **Published corrections.** When a small-sample number didn't survive a larger sample, we retracted it in the open. (An early "failed traces waste 2.6× more" held on 108 traces but collapsed across 7,116 — retracted. 18 of 22 models still show higher waste on failed traces, but no single multiplier holds.)
- **Fixes driven by real data.** The trace-commons scan surfaced two adapter issues that no synthetic test caught: session mid-run abort (3/28 crashes → recovered with `skip + warn`) and Anthropic `is_error: true` tool_result being sha256-identical (2 false-positives across 269 error responses → gated at the report layer, cascade unchanged). Both are recorded in `docs/CC_TRANSCRIPT.md` §29.
- **Disclosed limits.** The semantic embedding layer does not cleanly separate same-topic real-world outputs — the `sha256` structural gate carries the precision result, not the embedding. We say so rather than imply the model is doing the work.

**239 tests**, CI on every PR, frozen parameters enforced as failing tests.

---

## Install

> **Note:** the bare name `clew` on PyPI is an unrelated placeholder — don't install that. This project is published as `clew-custos` (the module still imports as `clew`).

```bash
pip install "clew-custos[detect]"
```

**What `[detect]` covers vs. what `[semantic]` adds:**

- **`[detect]`** (default, lightweight — no torch): tool-call repeat / requery
  detection via the sha256 structural gate. Works on Claude Code JSONL, Toolathlon,
  RedundancyBench, and any OTel/OpenInference trace whose duplicated work sits at the
  tool layer. This is where every empirically validated detection so far comes from.
- **`[semantic]`** (optional, ~2 GB with CUDA torch): adds the cosine gate for
  non-tool spans (cos ≥ φ), required for LangGraph chain-node paraphrase duplication
  (same node running twice with reworded but semantically overlapping output). The
  pingpong code path also flows through this gate — see the honesty note above; it has
  fired only on synthetic traces so far.

```bash
pip install "clew-custos[semantic]"
```

On Linux, PyPI's default torch wheel pulls the CUDA stack (~2 GB). To use CPU-only
torch, install it first from the PyTorch CPU index:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install "clew-custos[semantic]"
```

Or from source:

```bash
pip install "clew-custos[detect] @ git+https://github.com/JEONSEWON/Clew-by-Custos.git"
```

Requires Python ≥ 3.12.

## Use

```bash
python -m clew analyze path/to/trace.jsonl --out report.md
```

- Input: any auto-detected format from the table above.
- `--out` writes Markdown; `--json` writes structured output; `--no-snippets` omits output excerpts.
- Exit `0` whether or not waste is found; `1` on missing file / schema error / missing detect dependencies.

For your own Claude Code sessions, transcripts live at `~/.claude/projects/<slug>/<uuid>.jsonl`.

---

## License

MIT. Built under **Custos**.

External datasets referenced here (Toolathlon CC-BY-4.0, RedundancyBench MIT, trace-commons per its HF card) are analyzed locally and never redistributed in this repo.
