piwheels

lxtokenizer

LX-Tokenizer segments text into lexically relevant tokens.

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install lxtokenizer

Dependencies

• lxcommon
• openfile
• toolwrapper
• docopt

Releases

Version	Released	Bookworm Python 3.11	Trixie Python 3.13	Files
0.5.9	2022-07-11
lxtokenizer-0.5.9-py3-none-any.whl (63 KB)
0.5.8	2022-07-11
lxtokenizer-0.5.8-py3-none-any.whl (63 KB)
0.5.7	2022-07-11
lxtokenizer-0.5.7-py3-none-any.whl (63 KB)
0.5.6	2022-07-11
lxtokenizer-0.5.6-py3-none-any.whl (63 KB)
0.5.3	2022-07-11
lxtokenizer-0.5.3-py3-none-any.whl (63 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue