Metadata-Version: 2.5
Name: pyprocessors-reconciliation
Version: 1.6.59
Summary: Sherpa reconciliation processor
Author-email: Olivier Terrier <olivier.terrier@kairntech.com>
License-Expression: MIT
License-File: AUTHORS.md
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: collections-extended
Requires-Dist: log-with-context
Requires-Dist: pymultirole-plugins<1.7.0,>=1.6.0
Provides-Extra: dev
Requires-Dist: bump2version; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Provides-Extra: docs
Requires-Dist: lxml-html-clean; extra == 'docs'
Requires-Dist: m2r2; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-rtd-theme; extra == 'docs'
Requires-Dist: sphinxcontrib-apidoc; extra == 'docs'
Provides-Extra: sbom
Requires-Dist: cyclonedx-bom; extra == 'sbom'
Requires-Dist: pip-audit; extra == 'sbom'
Provides-Extra: test
Requires-Dist: dirty-equals; extra == 'test'
Requires-Dist: pytest-check; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-mypy; extra == 'test'
Requires-Dist: pytest>=6.0.0; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Description-Content-Type: text/markdown

# pyprocessors_reconciliation

[![license](https://img.shields.io/github/license/oterrier/pyprocessors_reconciliation)](https://github.com/oterrier/pyprocessors_reconciliation/blob/master/LICENSE)
[![tests](https://github.com/oterrier/pyprocessors_reconciliation/workflows/tests/badge.svg)](https://github.com/oterrier/pyprocessors_reconciliation/actions?query=workflow%3Atests)
[![codecov](https://img.shields.io/codecov/c/github/oterrier/pyprocessors_reconciliation)](https://codecov.io/gh/oterrier/pyprocessors_reconciliation)
[![docs](https://img.shields.io/readthedocs/pyprocessors_reconciliation)](https://pyprocessors_reconciliation.readthedocs.io)
[![version](https://img.shields.io/pypi/v/pyprocessors_reconciliation)](https://pypi.org/project/pyprocessors_reconciliation/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyprocessors_reconciliation)](https://pypi.org/project/pyprocessors_reconciliation/)

Reconciliation annotations coming from different annotators.

## Installation

```
pip install pyprocessors-reconciliation
```

## Overview

`ReconciliationProcessor` is a [pymultirole](https://github.com/kairntech/pymultirole-plugins) processor plugin that reconciles overlapping annotations produced by multiple annotators (NER models, knowledge-base linkers, white/kill lists) into a single coherent set.

The processor is registered under the `pyprocessors.plugins` entry point as `reconciliation`.

### Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `type` | `ReconciliationType` | `linker` | Reconciliation strategy (currently only `linker`) |
| `kill_label` | `str \| None` | `None` | Label whose annotations suppress matching model annotations |
| `white_label` | `str \| None` | `None` | Label treated as authoritative (terms stripped so it acts like a model annotation) |
| `whitelisted_lexicons` | `list[str] \| None` | `None` | Lexicons whose annotations are duplicated as term-free model candidates |
| `person_label` | `str \| None` | `None` | Label used to identify person annotations for last-name resolution |
| `remove_suspicious` | `bool` | `True` | Drop model annotations that contain no capitalised word (numbers, percentages, etc.) |
| `resolve_lastnames` | `bool` | `False` | Resolve isolated last names / first names using full names seen earlier in the document |

### How it works

1. **Sentence filtering** — `sentence`-labelled annotations are removed before processing.
2. **Whitelist marking** (`mark_whitelisted`) — annotations matching `white_label` have their terms cleared so they behave like model candidates; annotations from `whitelisted_lexicons` get a term-free duplicate added alongside the original.
3. **Grouping** (`group_annotations`) — annotations are grouped by their first term's lexicon (empty string = model / no-lexicon). Same-span annotations in the same group have their term lists merged and deduplicated.
4. **Linker consolidation** (`consolidate_linker`):
   - Suspicious model annotations (no capitalised word) are optionally dropped.
   - Kill-list annotations suppress matching model annotations.
   - KB annotations at the same span enrich the matching model annotation with their terms.
   - Overlapping or mismatched-label KB matches are logged as warnings and skipped.
5. **Last-name resolution** — when `resolve_lastnames=True`, isolated person names (single token) are resolved to the full-name annotation seen earliest in the document.

## Development

The build is driven by [Task](https://taskfile.dev) and [uv](https://docs.astral.sh/uv/),
with the shared stages coming from the `python-archetype` submodule.

### Getting started

The stages live in a Git submodule, so **clone with `--recurse-submodules`**:

```bash
git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_reconciliation.git
cd pyprocessors_reconciliation
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task
```

Already cloned without it? The submodule directory is empty, and `task` fails on:

```
task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"
```

which means exactly that, and nothing worse:

```bash
git submodule update --init
```

**Task is the only manual prerequisite.** An archetype cannot bootstrap itself: uv and the
Python interpreter install themselves on demand (every task that runs uv depends on an
internal `install-python` task), but the thing that runs them does not. Make sure
`~/.local/bin` is on your `PATH` — that is where `task` and `uv` both land.

### Running the pipeline

```bash
task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs
```

`task` with no argument is safe by construction: it runs every stage but the last, and that
bound is computed from the `STAGES` list rather than written down. The last stage is the
only one with an effect outside your machine.

`STAGES`, declared once in `Taskfile.yml`, is the single definition of the pipeline order —
so what you run locally is what Jenkins runs.

### Individual stages

| Task                            | Description                                           |
|---------------------------------|-------------------------------------------------------|
| `task py:sync`                  | Install the project and its dependencies (uv sync)    |
| `task py:lint`                  | `ruff check` and `ruff format --check`                |
| `task py:format`                | Reformat the code with ruff                           |
| `task py:test`                  | Run the test suite                                    |
| `task py:test-marker -- <m>`    | Run the tests carrying one pytest marker              |
| `task py:sbom`                  | Generate a CycloneDX SBOM of the resolved environment |
| `task py:check-vulnerabilities` | Check for known CVEs                                  |
| `task py:check-updates`         | Check for dependency updates                          |
| `task py:build`                 | Build the wheel and sdist (uv build)                  |
| `task py:publish`               | Publish the distributions (uv publish)                |
| `task py:version-file`          | Print the path of the file carrying `__version__`     |
| `task py:set-version VERSION=x` | Write that version into it                            |

`uv.lock` is not versioned here, so `py:sync` always resolves from scratch (`--upgrade`):
a stale lock lying around on a machine would otherwise make you test and audit versions the
CI never sees.

### SBOM & vulnerability check

`task py:sbom` and `task py:check-vulnerabilities` wrap the underlying tools. To run them
by hand:

```bash
uv sync --extra test --extra sbom
uv run cyclonedx-py environment -o sbom.cdx.json --output-format json
uv run pip-audit --skip-editable --format json --output audit-report.json
```

Ask for **all** the extras, not just `sbom`: `uv sync` synchronises rather than adds, so
whatever the requested extras do not pull in gets removed. `--extra sbom` on its own
uninstalls `ruff` and `pytest`, leaving an environment that can no longer lint or test.

`--skip-editable` skips the project itself, which no advisory database can know about. The
tasks do the same, and pip-audit still exits non-zero when it finds a real vulnerability —
which is the contract the CI relies on. Avoid `--strict`: it turns "dependency not found on
PyPI" into a fatal error, so it breaks as soon as the local version is not a published one.
