Metadata-Version: 2.4
Name: ragprepkit
Version: 0.1.3
Summary: Document preprocessing toolkit for RAG and LLM pipelines: text cleaning, chunking strategies, metadata extraction, and token counting.
Project-URL: Homepage, https://www.mindrops.com
Project-URL: Source, https://github.com/mindropsai/ragprep
Author: Mindrops
License: MIT
License-File: LICENSE
Keywords: chunking,document-processing,embeddings,llm,nlp,rag,text-preprocessing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: tokenizers
Requires-Dist: tiktoken>=0.5.0; extra == 'tokenizers'
Description-Content-Type: text/markdown

# RagPrep

[![PyPI](https://img.shields.io/pypi/v/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)
[![Python Versions](https://img.shields.io/pypi/pyversions/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)
[![License](https://img.shields.io/pypi/l/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)

Document preprocessing toolkit for RAG (retrieval-augmented generation) and
LLM pipelines. `ragprep` handles the unglamorous but high-leverage work that
sits between "raw document" and "ready to embed": cleaning noisy text,
splitting it into retrieval-sized chunks, pulling out lightweight structural
metadata, and estimating token counts before you ever call a model.

It has zero required dependencies. Everything works out of the box on
Python 3.9+; installing the optional `tiktoken` extra upgrades token counting
from a heuristic estimate to an exact count.

## Why this exists

Most RAG bugs aren't in the retrieval or the prompt — they're in the
preprocessing step nobody looked at closely: boilerplate leaking into
embeddings, sentences getting cut in half at chunk boundaries, or token
budgets blowing up because nobody counted before sending. `ragprep` is a
small, inspectable, dependency-light layer for that step, not a framework
that owns your whole pipeline.

## Features

- **Text cleaning** — unicode normalization, control-character stripping,
  configurable boilerplate-line removal (cookie notices, copyright footers,
  newsletter prompts), and whitespace normalization.
- **Three chunking strategies** — fixed-size character windows, sentence-safe
  chunking that never splits mid-sentence, and a recursive
  paragraph → sentence → fixed-size fallback for mixed long-form documents.
- **Lightweight metadata extraction** — word/sentence counts, estimated
  reading time, markdown headings, extracted URLs, and a coarse script-based
  language guess.
- **Token counting** — exact counts via an optional `tiktoken` integration,
  with a dependency-free heuristic fallback so the library always works.
- **Zero required dependencies** — the core package has no runtime
  dependencies; `tiktoken` is opt-in via an extra.
- **Fully typed** — type hints throughout, with a `py.typed` marker for
  PEP 561 compatibility with type checkers.

## Project structure

```
ragprep/
├── src/
│   └── ragprep/
│       ├── __init__.py      # public API exports, __version__
│       ├── cleaning.py      # clean_text, normalize_whitespace, strip_boilerplate
│       ├── chunking.py      # Chunk, fixed_size_chunks, sentence_chunks, recursive_chunks
│       ├── metadata.py      # DocumentMetadata, extract_metadata
│       ├── tokens.py        # count_tokens, estimate_cost
│       └── py.typed         # PEP 561 typed-package marker
├── tests/                   # pytest test suite (mirrors src/ragprep modules)
├── pyproject.toml           # build config, metadata, dependencies
├── LICENSE
└── README.md
```

## Installation

```bash
pip install ragprepkit
```

> **Note:** The package is published on PyPI as **`ragprepkit`**, while the Python import name is **`ragprep`**.

```python
from ragprep import clean_text
```

Requires Python 3.9 or later.

## Quick start

```python
from ragprep import clean_text, recursive_chunks, extract_metadata, count_tokens

raw = """
# Quarterly Report

Cookie Policy applies to this site.

Revenue grew 12% year over year, driven primarily by expansion
in the enterprise segment. Customer churn declined for the third
consecutive quarter.

## Outlook

Management expects continued growth into next year, though macro
headwinds remain a risk. See https://example.com/full-report for
the full filing.

All rights reserved 2026.
"""

text = clean_text(raw)
chunks = recursive_chunks(text, chunk_size=300, overlap=30)
meta = extract_metadata(text)

print(f"{len(chunks)} chunks, {meta.word_count} words, {meta.estimated_reading_time_minutes} min read")
for chunk in chunks:
    print(f"[chunk {chunk.index}] ({count_tokens(chunk.text)} tokens) {chunk.text[:60]}...")
```

## Usage examples

### 1. Cleaning scraped or exported documents

```python
from ragprep import clean_text

raw_html_text = extracted_text_from_your_scraper  # already stripped of tags
cleaned = clean_text(
    raw_html_text,
    remove_boilerplate=True,
    extra_boilerplate_patterns=[r"^sign up for our newsletter.*$"],
)
```

`clean_text` runs unicode normalization, control-character stripping,
boilerplate-line removal, and whitespace normalization in one pass. Each
step is also exposed individually (`normalize_whitespace`,
`strip_boilerplate`) if you want to compose your own pipeline.

### 2. Choosing a chunking strategy

```python
from ragprep import fixed_size_chunks, sentence_chunks, recursive_chunks

# Uniform windows — fastest, ignores structure. Good for short, dense text.
fixed = fixed_size_chunks(text, chunk_size=800, overlap=80)

# Never splits a sentence — good when exact quotes/citations matter.
by_sentence = sentence_chunks(text, max_chars=800, overlap_sentences=1)

# Paragraph-first, falling back to sentence- then fixed-size splitting.
# The best default for mixed long-form documents (reports, articles, docs).
by_structure = recursive_chunks(text, chunk_size=800, overlap=80)
```

Each strategy returns a list of `Chunk` objects carrying `text`,
`start_char`, `end_char`, `index`, and an open `metadata` dict you can
populate with your own fields (source document ID, page number, etc.)
before handing chunks to your embedding step.

```python
for chunk in by_structure:
    chunk.metadata["source"] = "quarterly_report.pdf"
    chunk.metadata["page"] = estimate_page_number(chunk.start_char)
```

### 3. Extracting metadata for filtering and routing

```python
from ragprep import extract_metadata

meta = extract_metadata(document_text)

print(meta.word_count)
print(meta.headings)                        # e.g. ["Quarterly Report", "Outlook"]
print(meta.urls)                             # links found in the body
print(meta.estimated_reading_time_minutes)
print(meta.likely_language)                  # coarse latin/non-latin script guess
```

Useful for building filters ("only index docs over 200 words"), surfacing
a table of contents from headings, or routing documents to a
language-specific pipeline before deeper processing.

### 4. Token counting and cost estimation

```python
from ragprep import count_tokens, estimate_cost

tokens = count_tokens(chunk.text)  # exact if tiktoken is installed, else an estimate

# Pass in your current provider's price per 1k tokens — pricing changes
# often, so this is deliberately not hardcoded into the library.
cost = estimate_cost(chunk.text, price_per_1k_tokens=0.003)
```

### 5. End-to-end pipeline sketch

```python
from ragprep import clean_text, recursive_chunks, count_tokens

def prepare_for_embedding(raw_text: str, source_id: str, max_tokens_per_chunk: int = 300):
    cleaned = clean_text(raw_text)
    chunks = recursive_chunks(cleaned, chunk_size=1200, overlap=120)

    prepared = []
    for chunk in chunks:
        if count_tokens(chunk.text) > max_tokens_per_chunk:
            # Fall back to a tighter split for oversized chunks.
            chunk_pieces = recursive_chunks(chunk.text, chunk_size=600, overlap=60)
        else:
            chunk_pieces = [chunk]

        for piece in chunk_pieces:
            prepared.append({
                "text": piece.text,
                "source_id": source_id,
                "tokens": count_tokens(piece.text),
            })
    return prepared
```

## API reference

| Function | Module | Purpose |
|---|---|---|
| `clean_text(text, **opts)` | `ragprep.cleaning` | Full cleaning pipeline |
| `normalize_whitespace(text)` | `ragprep.cleaning` | Collapse spaces/blank lines |
| `strip_boilerplate(text, extra_patterns=None)` | `ragprep.cleaning` | Remove boilerplate lines |
| `fixed_size_chunks(text, chunk_size, overlap)` | `ragprep.chunking` | Character-window chunking |
| `sentence_chunks(text, max_chars, overlap_sentences)` | `ragprep.chunking` | Sentence-safe chunking |
| `recursive_chunks(text, chunk_size, overlap)` | `ragprep.chunking` | Paragraph → sentence → fixed fallback |
| `extract_metadata(text, words_per_minute=200)` | `ragprep.metadata` | Word/sentence counts, headings, URLs, language guess |
| `count_tokens(text, encoding_name="cl100k_base")` | `ragprep.tokens` | Token count (exact w/ tiktoken, else estimate) |
| `estimate_cost(text, price_per_1k_tokens, encoding_name)` | `ragprep.tokens` | Rough cost estimate for a given price point |

## Development

```bash
git clone https://github.com/mindropsai/ragprep.git
cd ragprep
pip install -e ".[dev,tokenizers]"
pytest
ruff check .
```

## Design notes / limitations

- `extract_metadata`'s language detection is a coarse latin vs. non-latin
  script heuristic, not real language identification — pair with a proper
  library (e.g. `langdetect`, `fasttext`) if you need accurate ISO codes.
- `extract_metadata`'s sentence count is a simple punctuation-based
  heuristic and can overcount on things like decimal numbers or
  abbreviations — treat it as an estimate, not an exact linguistic count.
- `count_tokens` falls back to a `len(text) // 4` heuristic without
  `tiktoken` installed. This is a commonly cited rough average for English
  text, not an exact count for every tokenizer or language — install the
  `tokenizers` extra for precision.
- `estimate_cost` intentionally takes price as an argument rather than
  embedding a pricing table, since provider pricing changes frequently.

## License

MIT — see [LICENSE](LICENSE).

## About Mindrops

**Mindrops** is a global technology company that helps organizations accelerate digital transformation through intelligent software solutions. With expertise spanning Artificial Intelligence, cloud technologies, custom software development, mobile applications, and digital platforms, Mindrops partners with businesses to build scalable, secure, and high-performance technology solutions.

Driven by its philosophy of **"Thinking Together, Creating Together,"** Mindrops combines deep technical expertise with a collaborative, process-driven approach to solve complex business challenges. The company serves clients across North America, Europe, and Asia through its presence in India, the United Kingdom, Belgium, and the United States. 

**ragprep** is one of the open-source initiatives developed by Mindrops to support modern AI development. It reflects our commitment to building practical, production-ready tools that simplify document preprocessing for Retrieval-Augmented Generation (RAG), semantic search, and enterprise LLM applications.

Learn more about Mindrops at **https://www.mindrops.com**.

## Links

- 🌐 Homepage: https://www.mindrops.com
- 💻 Repository: https://github.com/mindropsai/ragprep
