piwheels

curated-tokenizers

Lightweight piece tokenization library

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install curated-tokenizers

Dependencies

• regex

Releases

Version	Released	Bookworm Python 3.11	Trixie Python 3.13	Files
2.0.1	2026-08-24
curated_tokenizers-2.0.1-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.1-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-2.0.1-cp313-cp313-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.1-cp313-cp313-linux_armv7l.whl (6 MB) curated_tokenizers-2.0.1-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.1-cp39-cp39-linux_armv7l.whl (6 MB)
2.0.0	2024-04-15
curated_tokenizers-2.0.0-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.0-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-2.0.0-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.0-cp39-cp39-linux_armv7l.whl (6 MB)
2.0.0.dev0 pre-release	2024-04-12
curated_tokenizers-2.0.0.dev0-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.0.dev0-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-2.0.0.dev0-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-2.0.0.dev0-cp39-cp39-linux_armv7l.whl (6 MB)
0.9.2	2024-02-07
curated_tokenizers-0.9.2-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.2-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.2-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.2-cp39-cp39-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.2-cp313-cp313-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.2-cp313-cp313-linux_armv6l.whl (6 MB)
0.9.1	2023-09-14
curated_tokenizers-0.9.1-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.1-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.1-cp37-cp37m-linux_armv6l.whl (4 MB) curated_tokenizers-0.9.1-cp37-cp37m-linux_armv7l.whl (4 MB) curated_tokenizers-0.9.1-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.1-cp39-cp39-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.1-cp313-cp313-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.1-cp313-cp313-linux_armv7l.whl (6 MB)
0.9.0	2023-07-11
curated_tokenizers-0.9.0-cp37-cp37m-linux_armv6l.whl (4 MB) curated_tokenizers-0.9.0-cp37-cp37m-linux_armv7l.whl (4 MB) curated_tokenizers-0.9.0-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.0-cp39-cp39-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.0-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.0-cp311-cp311-linux_armv7l.whl (6 MB)
0.9.0.dev0 pre-release	2023-07-06
curated_tokenizers-0.9.0.dev0-cp37-cp37m-linux_armv6l.whl (4 MB) curated_tokenizers-0.9.0.dev0-cp37-cp37m-linux_armv7l.whl (4 MB) curated_tokenizers-0.9.0.dev0-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.0.dev0-cp39-cp39-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.0.dev0-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.9.0.dev0-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.0.dev0-cp313-cp313-linux_armv7l.whl (6 MB) curated_tokenizers-0.9.0.dev0-cp313-cp313-linux_armv6l.whl (6 MB)
0.0.10	2026-08-24
curated_tokenizers-0.0.10-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.10-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.10-cp313-cp313-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.10-cp313-cp313-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.10-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.10-cp39-cp39-linux_armv7l.whl (6 MB)
0.0.9	2024-01-18
curated_tokenizers-0.0.9-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.9-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.9-cp37-cp37m-linux_armv6l.whl (3 MB) curated_tokenizers-0.0.9-cp37-cp37m-linux_armv7l.whl (3 MB) curated_tokenizers-0.0.9-cp39-cp39-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.9-cp39-cp39-linux_armv7l.whl (6 MB)
0.0.8	2023-07-06
curated_tokenizers-0.0.8-cp37-cp37m-linux_armv6l.whl (3 MB) curated_tokenizers-0.0.8-cp37-cp37m-linux_armv7l.whl (3 MB) curated_tokenizers-0.0.8-cp39-cp39-linux_armv6l.whl (5 MB) curated_tokenizers-0.0.8-cp39-cp39-linux_armv7l.whl (5 MB) curated_tokenizers-0.0.8-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.8-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.8-cp313-cp313-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.8-cp313-cp313-linux_armv7l.whl (6 MB)
0.0.7	2023-05-23
curated_tokenizers-0.0.7-cp37-cp37m-linux_armv6l.whl (3 MB) curated_tokenizers-0.0.7-cp37-cp37m-linux_armv7l.whl (3 MB) curated_tokenizers-0.0.7-cp39-cp39-linux_armv6l.whl (5 MB) curated_tokenizers-0.0.7-cp39-cp39-linux_armv7l.whl (5 MB) curated_tokenizers-0.0.7-cp313-cp313-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.7-cp313-cp313-linux_armv6l.whl (6 MB)
0.0.6	2023-05-23
curated_tokenizers-0.0.6-cp39-cp39-linux_armv6l.whl (5 MB) curated_tokenizers-0.0.6-cp39-cp39-linux_armv7l.whl (5 MB) curated_tokenizers-0.0.6-cp37-cp37m-linux_armv6l.whl (3 MB) curated_tokenizers-0.0.6-cp37-cp37m-linux_armv7l.whl (3 MB) curated_tokenizers-0.0.6-cp311-cp311-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.6-cp311-cp311-linux_armv7l.whl (6 MB) curated_tokenizers-0.0.6-cp313-cp313-linux_armv6l.whl (6 MB) curated_tokenizers-0.0.6-cp313-cp313-linux_armv7l.whl (6 MB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue