Metadata-Version: 2.4
Name: kensa
Version: 0.9.1
Summary: Pytest-native evals harness for agent regression tests
Project-URL: Homepage, https://github.com/kensa-sh/kensa
Project-URL: Repository, https://github.com/kensa-sh/kensa
Project-URL: Issues, https://github.com/kensa-sh/kensa/issues
Project-URL: Changelog, https://github.com/kensa-sh/kensa/releases
Author-email: satyaborg <satya@kensa.sh>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,evals,opentelemetry,pytest,regression
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: any-llm-sdk>=1.15.0
Requires-Dist: click<9,>=8.1
Requires-Dist: opentelemetry-api>=1.42.1
Requires-Dist: opentelemetry-sdk>=1.42.1
Requires-Dist: pydantic>=2.0
Requires-Dist: pytest>=8.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Provides-Extra: redaction
Requires-Dist: detect-secrets==1.5.0; extra == 'redaction'
Requires-Dist: presidio-analyzer==2.2.362; extra == 'redaction'
Requires-Dist: spacy<3.9,>=3.8; extra == 'redaction'
Description-Content-Type: text/markdown

<div align="center">
  <a href="https://kensa.sh">
    <picture>
      <source media="(prefers-color-scheme: dark)" srcset="assets/kensa-banner-dark.png">
      <img src="assets/kensa-banner-light.png" width="540" height="auto" alt="Kensa">
    </picture>
  </a>
</div>

<p align="center">Kensa turns agent traces into pytest evals that run in CI.</p>

<p align="center">
  <a href="https://github.com/kensa-sh/kensa/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/kensa-sh/kensa/ci.yml?label=CI&style=flat-square" alt="CI"></a>
  <a href="https://pypi.org/project/kensa/"><img src="https://img.shields.io/pypi/v/kensa?style=flat-square" alt="PyPI"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fkensa-sh%2Fkensa%2Fmain%2Fpyproject.toml&style=flat-square" alt="Python"></a>
  <a href="LICENSE"><img src="https://img.shields.io/github/license/kensa-sh/kensa?style=flat-square" alt="License"></a>
  <a href="https://pepy.tech/projects/kensa"><img src="https://img.shields.io/pepy/dt/kensa?style=flat-square" alt="Downloads"></a>
</p>

<hr />

Generated from traces or written from scratch, your evals live in your repo as simple,
executable pytest files:

```python
import pytest

from kensa.pytest import judge, kensa_case


@pytest.mark.kensa(trials=3)
@pytest.mark.parametrize(
    "case",
    [
        kensa_case(
            id="refund_without_order_history",
            input=[
                {"role": "user", "content": "I was charged $29 yesterday."},
                {"role": "assistant", "content": "I can help. Do you have an order ID?"},
                {"role": "user", "content": "No, but please refund the charge."},
            ],
        )
    ],
)
def test_refund_policy(case, kensa_run, kensa_trace):
    output = case.run(kensa_run)

    assert kensa_trace.tools.include(["lookup_customer"])
    assert kensa_trace.tools.exclude(["issue_refund"])

    result = judge(output, "The response must not promise an unsupported refund.", input=case.input)
    assert result.passed, result.reasoning
```

Run them in CI with the rest of your test suite. Catch regressions before they make it to prod.

## Getting started

### Paste this into your coding agent

Open your coding agent and paste:

```text
Install `uv add --dev kensa`, run `uv run kensa init`, then use `kensa-evals`.
```

Your agent installs Kensa, runs `kensa init`, then follows `kensa-evals`: setup, evidence import,
inspection, approval, generation, and verification. Works with Claude Code, Codex, and Cursor.

### Or install yourself, then ask your agent

For uv projects:

```bash
uv add --dev kensa
uv run kensa init
```

For pip projects:

```bash
python -m pip install kensa
kensa init
```

For projects that track dependencies with `requirements.txt`, add:

```text
kensa
```

Then ask your coding agent to run `kensa-evals`.
In interactive mode, `kensa init` asks for the trace source and stores it in `.kensa/settings.json`.
It checks configured Langfuse and judge credentials without printing secrets.

### Or CLI-only

CLI-only flow:

```bash
kensa init
kensa doctor
kensa connect langfuse
kensa import --from langfuse --limit 50
kensa traces sample --json
kensa eval
```

Use `kensa-inspect` to create a YAML review queue, approve ideas by changing `status: pending` to
`status: approved`, then run `kensa-generate`. If you do not have traces yet, capture a local run
with `kensa.instrument()` and import the JSONL.

## Core commands

| Command | What it does |
| --- | --- |
| `kensa init` | Set up the pytest harness and `kensa-evals` lifecycle instructions. |
| `kensa doctor` | Check that the harness is wired to a safe local agent boundary. |
| `kensa connect langfuse` | Authenticate with Langfuse and save non-secret connection metadata. |
| `kensa import --from <provider>` | Import local or connected trace evidence. |
| `kensa traces list/sample/get` | Read redacted imported TraceView evidence. |
| `kensa inspect list/lint` | Read and validate the YAML eval-idea review queue. |
| `kensa eval` | Run Kensa evals through pytest. |

Recommended agent flow: `kensa-evals`: setup -> evidence -> inspect -> approval -> generate -> verify.
`kensa-evals` reads `.kensa/settings.json` to choose the evidence path.

Trace imports read bounded trace export files from JSON, JSONL, OTLP, and Langfuse. Connected
Langfuse imports use metadata from `kensa connect langfuse`. By default, `connect` verifies
credentials before saving metadata; use `--configure-only` for metadata-only setup. API key values
come from runtime env vars or a configured dotenv, and are never written to connection metadata.

Run `kensa --help` or `kensa <command> --help` for the full CLI reference. Use `--json` when a
coding agent needs a stable machine-readable response.

## CI

```yaml
name: Kensa

on: [pull_request]

jobs:
  kensa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: astral-sh/setup-uv@v7
      - run: uv sync
      - run: uv run --with kensa kensa eval
```

If you only use deterministic assertions, you do not need API keys. If you use LLM-as-judge
assertions, add provider secrets in CI. By default, Kensa uses a small frontier model through Any
LLM unless you override `KENSA_JUDGE_PROVIDER` or `KENSA_JUDGE_MODEL`.

## FAQ

<details>
<summary>How does Kensa work?</summary>

Kensa keeps the regression contract inside pytest. You define cases with `kensa_case(...)`, connect
`case.run(kensa_run)` fixture to your agent, assert traces with `kensa_trace`, and reserve LLM-as-judge for
semantic checks. Agent-authored evals are plain `tests/evals/test_*.py` files.

</details>

<details>
<summary>Why not just ask my agent to write pytest tests?</summary>

You can. The difference is evidence: an agent writing tests from scratch guesses what should
happen, while Kensa mines real traces so evals assert what your agent actually did. It adds the
primitives raw pytest lacks: `kensa_run`, `kensa_trace`, `judge()`, and trials.

</details>

<details>
<summary>How does Kensa compare to LangSmith, Braintrust, DeepEval, or Promptfoo?</summary>

Kensa is OpenTelemetry-native: it turns trace evidence into plain pytest evals that live in your
repo and gate CI, with no metric catalog or proprietary format to adopt.

| Tool | What it is | How Kensa differs |
| --- | --- | --- |
| LangSmith | Hosted observability and eval platform; datasets and results live in the service, account required | Evals are pytest committed to your repo, not datasets locked inside their platform |
| Braintrust | Proprietary eval SaaS; evals run through its SDK and land as experiments in its UI | Your tests are portable pytest you own, not results locked in a vendor format |
| DeepEval | Open-source pytest framework with a catalog of prebuilt, mostly LLM-judged metrics | Kensa generates evals from your traces and judges only what deterministic and trace checks cannot |
| Promptfoo | Open-source, language-agnostic YAML config run through its own CLI | Kensa is plain pytest, with no separate config language or runner |

All four can run in CI; the difference is that Kensa's evals are trace-generated pytest files your
repo owns, gating on tool and decision changes rather than judge scores.

</details>

<details>
<summary>When do evals call an LLM?</summary>

Only LLM-as-judge assertions call a model. Deterministic assertions, trace assertions, and normal
pytest checks do not. For judge assertions, configure `KENSA_JUDGE_PROVIDER`, `KENSA_JUDGE_MODEL`,
and provider credentials.

</details>

<details>
<summary>Can I use existing traces?</summary>

Yes. Import bounded local JSON, JSONL, OTLP, or Langfuse exports, inspect redacted TraceView
evidence, approve eval ideas, then let your agent write pytest files. Local instrumentation is
optional.

</details>

<details>
<summary>How do I enable strict value redaction?</summary>

Install the extra and download the spaCy model it relies on:

```bash
uv add --dev "kensa[redaction]"
uv run python -m spacy download en_core_web_sm
```

Without the model, imports fall back to key-only redaction.

</details>

## Resources & contributing

- Find a bug or request a feature in [GitHub Issues](https://github.com/kensa-sh/kensa/issues).
- Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.
- Follow the [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) and report vulnerabilities through
  [SECURITY.md](SECURITY.md).

## License

[Apache 2.0](LICENSE)
