piwheels

tok

Fast and customizable tokenizer

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install tok

Dependencies

• textsearch
• tldextract
• contractions

Releases

Version	Released	Bookworm Python 3.11	Trixie Python 3.13	Files
0.1.14	2019-07-09
tok-0.1.14-py2.py3-none-any.whl (8 KB)
0.1.13	2019-07-09
tok-0.1.13-py2.py3-none-any.whl (8 KB)
0.0.9	2019-07-04
tok-0.0.9-py2.py3-none-any.whl (8 KB)
0.0.8	2019-07-04
tok-0.0.8-py2.py3-none-any.whl (8 KB)
0.0.5	2019-07-04
tok-0.0.5-py2.py3-none-any.whl (8 KB)
0.0.4	2019-07-04
tok-0.0.4-py2.py3-none-any.whl (7 KB)
0.0.3	2019-07-04
tok-0.0.3-py2.py3-none-any.whl (7 KB)
0.0.2	2019-07-04
tok-0.0.2-py2.py3-none-any.whl (7 KB)
0.0.1	2019-07-04
tok-0.0.1-py2.py3-none-any.whl (7 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue