piwheels

pytokens

A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

Installation

In a virtualenv (see these instructions if you need to create one):

sudo apt install libc6
pip3 install pytokens

Dependencies

• None

Releases

Version	Released	Bookworm Python 3.11	Trixie Python 3.13	Files
0.4.1	2026-01-30
pytokens-0.4.1-cp313-cp313-linux_armv6l.whl (267 KB) pytokens-0.4.1-cp313-cp313-linux_armv7l.whl (267 KB)
0.4.0	2026-01-19
pytokens-0.4.0-cp311-cp311-linux_armv6l.whl (257 KB) pytokens-0.4.0-cp311-cp311-linux_armv7l.whl (257 KB) pytokens-0.4.0-cp313-cp313-linux_armv6l.whl (266 KB) pytokens-0.4.0-cp313-cp313-linux_armv7l.whl (266 KB)
0.3.0	2025-11-05
pytokens-0.3.0-py3-none-any.whl (12 KB)
0.2.0	2025-10-15
pytokens-0.2.0-py3-none-any.whl (12 KB)
0.1.10	2025-02-19
pytokens-0.1.10-py3-none-any.whl (12 KB)
0.1.9	2025-01-09
pytokens-0.1.9-py3-none-any.whl (11 KB)
0.1.8	2025-01-09
pytokens-0.1.8-py3-none-any.whl (11 KB)
0.1.7	2025-01-08
pytokens-0.1.7-py3-none-any.whl (11 KB)
0.1.5	2024-12-25
pytokens-0.1.5-py3-none-any.whl (11 KB)
0.1.4	2024-12-25
pytokens-0.1.4-py3-none-any.whl (11 KB)
0.1.3.post1	2024-12-23
pytokens-0.1.3.post1-py3-none-any.whl (11 KB)
0.1.3 yanked	2024-12-23
pytokens-0.1.3-py3-none-any.whl (11 KB)
0.1.2	2024-12-20
pytokens-0.1.2-py3-none-any.whl (11 KB)
0.1.1	2024-12-20
pytokens-0.1.1-py3-none-any.whl (11 KB)
0.1.0	2024-12-20
pytokens-0.1.0-py3-none-any.whl (10 KB)
0.0.0	2024-12-13
pytokens-0.0.0-py3-none-any.whl (4 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue