Metadata-Version: 2.4
Name: sciworld
Version: 0.1.0
Summary: Turn published papers into research environments an AI agent can work in
Author: Taiming Lu
License: MIT
Project-URL: Homepage, https://github.com/TaiMingLu/sciworld
Project-URL: Repository, https://github.com/TaiMingLu/sciworld
Project-URL: Issues, https://github.com/TaiMingLu/sciworld/issues
Keywords: ai,agents,research,reproducibility,benchmark,llm
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: prompt_toolkit>=3.0
Dynamic: license-file

# sciworld

Turn published papers into research environments an AI agent can actually work in.

Every paper with public code is a frozen research trajectory — problem, method,
baseline, metric, result. `sciworld` packages one of those into an isolated
workspace, hands the keys to an agent (Claude Code or Codex), and lets it try
to **reproduce**, **improve**, or **solve from scratch** — then scores the result.

```bash
pip install sciworld
```

Or from source:

```bash
git clone https://github.com/TaiMingLu/sciworld.git
cd sciworld
pip install -e .
```

## CLI

```bash
sw init                                         # one-time setup wizard
sw config [section]                             # view/edit (paths, compute, prompt, test-prompt, …)
sw select [--id ID]                             # browse papers; clones <data_dir>/envs/<id>/source/
sw info --id <arxiv_id> [--full]                # show paper overview/problem/experiments
sw research --id <arxiv_id> [flags]             # build + launch a research env
sw evaluate --id <arxiv_id> [--run NAME]        # compare run scores to paper scores
```

### `sw research` flags

| Flag | What it does |
|---|---|
| `--id ARXIV_ID` | paper to work on (required for non-interactive use) |
| `--mode improve \| scratch` | agent gets the method + results (`improve`) vs problem-only (`scratch`) |
| `--agent claude-code \| codex` | which CLI runs inside the container |
| `--tag TAG` | suffix for the run folder (e.g. `v2`, `ablation`) |
| `--path DIR` | override workspace destination |
| `--loop` | keep resuming the agent's session until `status.json.done=true` |
| `--max-loops N` | safety cap for `--loop` (default 10) |
| `--setup-only` | build the env + write `launch.sh` but **don't launch the agent** |
| `--refresh` | bypass paper cache |

`sw research` always writes a standalone `launch_<run_name>.sh` as a sibling
of the workspace. Normal runs execute it inside tmux/screen/nohup. With
`--setup-only`, nothing starts — the env is prepared and you can run
`bash <path>/launch_<run_name>.sh` yourself (inside your own job scheduler,
cron, or another session manager). Re-running the script is safe.

### Train / test split (anti-shortcut)

After the agent sets `status.json.done=true`, sciworld silently enters a
test phase: copies the hidden `source/eval/test/` + `scripts/evaluate_test.sh`
into the workspace, resumes the agent with a strict "be honest, do not
cheat, do not modify your method" prompt, and records the test-side scores
to `scoring/scores_test.json`. The test phase locks `method/` read-only.
Nothing test-related is present in the workspace during main-phase operation.

### Trajectory and token capture

Each CLI invocation writes raw JSONL events to
`<env_dir>/trajectory/turn_NNNN_raw.jsonl` and merges a unified summary
into `<env_dir>/trajectory/stats.json` with `{input_tokens, output_tokens,
num_turns, duration_s, cost_usd, tool_uses}` — identical shape for Claude
Code and Codex.

## Python API

Every CLI command is also a Python function with the same name and matching
kwargs. Required args missing → `TypeError`; non-zero exit → `RuntimeError`
(no interactive prompts — notebook/script-safe).

```python
import sciworld
from pathlib import Path

info = sciworld.research(
    id="2505.19252", agent="claude-code", mode="scratch",
    loop=True, max_loops=5,
    system_prompt=Path("~/my_prompts/strict.md").expanduser(),
    test_prompt="Run evaluate_test.sh, write reflection.md, stop.",
    setup_only=False,       # True → write launch.sh, don't launch
)
# → {"arxiv_id", "run_name", "env_dir", "source_dir", "launch_script"}

sciworld.evaluate(id="2505.19252", run=info["run_name"])
sciworld.info(id="2505.19252")
sciworld.select(id="2505.19252")
sciworld.config()                               # returns current config dict
sciworld.config(section="compute")              # opens the interactive editor
sciworld.init(force=False)                      # first-time setup wizard
```

Prompt kwargs (`system_prompt`, `mode_prompt`, `continue_prompt`,
`test_prompt`, `compute`) accept either a literal `str` or a
`pathlib.Path` — paths are read at call time. Use `import sciworld as sw`
if you prefer the two-letter alias.

## Where things live

| Path | What's there |
|---|---|
| `~/.sciworld/config.json` | your config (paths, container, session, env bindings) |
| `~/.sciworld/instructions/*.md` | your customized prompts (system, research_scratch, continue, test, compute) |
| `<data_dir>/envs/<id>/source/` | cloned paper repo |
| `<data_dir>/envs/<id>/<run_name>/` | per-run agent workspace (bind-mounted into container) |
| `<data_dir>/envs/<id>/launch_<run_name>.sh` | launcher — **outside** the workspace, not visible to the agent |
| `<run_workspace>/agent.log` | full pretty-printed session log |
| `<run_workspace>/trajectory/` | raw per-turn JSONL + merged stats.json |
| `<run_workspace>/scoring/scores_train.json` | agent's train-side scores |
| `<run_workspace>/scoring/scores_test.json` | agent's test-side scores (written during test phase) |
| `<run_workspace>/idea.md`, `reflection.md` | agent's end-of-run notes |

## Default prompts

All bundled prompts live as markdown under `sciworld/prompts/` — nothing
prompt-related is hardcoded in Python. Per-user overrides go into
`~/.sciworld/instructions/` via `sw config …`.

| File | When it's used |
|---|---|
| `system_prompt.md` | base role description, prepended to every launch |
| `improve.md`, `scratch.md` | mode-specific task instructions |
| `environment.md` | container housekeeping note (appended to every launch) |
| `continue.md` | follow-up message for each `--loop` iteration |
| `test.md` | strict test-phase prompt (sent only at test-phase transition) |

Run any command with `--help` for details.
