piwheels

pyTokenizer

A streaming tokenizer.

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install pytokenizer

Dependencies

• None

Releases

Version	Released	Bookworm Python 3.11	Trixie Python 3.13	Files
1.1.4	2020-09-06
pyTokenizer-1.1.4-py3-none-any.whl (9 KB)
1.1.3	2019-12-27
pyTokenizer-1.1.3-py3-none-any.whl (8 KB)
1.1.2	2019-12-25
pyTokenizer-1.1.2-py3-none-any.whl (8 KB)
1.1.1	2019-10-13
pyTokenizer-1.1.1-py3-none-any.whl (8 KB)
1.1.0	2019-10-13
pyTokenizer-1.1.0-py3-none-any.whl (8 KB)
0.1.3	2019-10-13
pyTokenizer-0.1.3-py3-none-any.whl (8 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue