piwheels

TokeNicer

A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install tokenicer

Dependencies

• None

Releases

Version	Released	Bullseye Python 3.9	Bookworm Python 3.11	Trixie Python 3.13	Files
0.0.13	2026-04-15
tokenicer-0.0.13-py3-none-any.whl (16 KB)
0.0.12	2026-03-26
tokenicer-0.0.12-py3-none-any.whl (16 KB)
0.0.11	2026-03-15
tokenicer-0.0.11-py3-none-any.whl (14 KB)
0.0.10	2026-03-15
tokenicer-0.0.10-py3-none-any.whl (14 KB)
0.0.9	2026-03-15
tokenicer-0.0.9-py3-none-any.whl (13 KB)
0.0.8	2026-03-06
tokenicer-0.0.8-py3-none-any.whl (11 KB)
0.0.7	2026-03-03
tokenicer-0.0.7-py3-none-any.whl (11 KB)
0.0.6	2026-02-09
tokenicer-0.0.6-py3-none-any.whl (11 KB)
0.0.5	2025-09-04
tokenicer-0.0.5-py3-none-any.whl (11 KB)
0.0.4	2025-02-21
tokenicer-0.0.4-py3-none-any.whl (11 KB)
0.0.3	2025-02-21
tokenicer-0.0.3-py3-none-any.whl (11 KB)
0.0.2	2025-02-10
tokenicer-0.0.2-py3-none-any.whl (11 KB)
0.0.1	2025-02-10
0.0.1.dev0 pre-release	2025-02-08
tokenicer-0.0.1.dev0-py3-none-any.whl (5 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue