piwheels

code-tokenizers

Aligning BPE and AST

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install code-tokenizers

Dependencies

• fastcore
• pandas
• tree-sitter
• transformers

Releases

Version	Released	Bullseye Python 3.9	Bookworm Python 3.11	Trixie Python 3.13	Files
0.0.5	2023-02-06
code_tokenizers-0.0.5-py3-none-any.whl (12 KB)
0.0.4	2023-02-05
code_tokenizers-0.0.4-py3-none-any.whl (62 KB)
0.0.3	2022-11-28
code_tokenizers-0.0.3-py3-none-any.whl (61 KB)
0.0.2	2022-11-22
code_tokenizers-0.0.2-py3-none-any.whl (60 KB)
0.0.1	2022-11-17
code_tokenizers-0.0.1-py3-none-any.whl (108 KB)

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue