piwheels

tokenizers

Installation

In a virtualenv (see these instructions if you need to create one):

sudo apt install libstdc++6
pip3 install tokenizers

Dependencies

• huggingface-hub

Releases

Version	Released	Bullseye Python 3.9	Bookworm Python 3.11	Files
0.21.4	2025-07-28
0.21.2	2025-06-24
0.21.2rc0 pre-release	2025-06-24
0.21.1	2025-03-13
tokenizers-0.21.1-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.21.1-cp39-abi3-linux_armv7l.whl (3 MB)
0.21.1rc0 pre-release	2025-03-12
tokenizers-0.21.1rc0-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.21.1rc0-cp39-abi3-linux_armv7l.whl (3 MB)
0.21.0	2024-11-27
tokenizers-0.21.0-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.21.0-cp39-abi3-linux_armv7l.whl (3 MB)
0.21.0rc0 pre-release	2024-11-27
tokenizers-0.21.0rc0-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.21.0rc0-cp39-abi3-linux_armv7l.whl (3 MB)
0.20.4 yanked	2024-11-26
tokenizers-0.20.4-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.20.4-cp39-abi3-linux_armv7l.whl (3 MB)
0.20.4rc0 pre-release	2024-11-26
tokenizers-0.20.4rc0-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.20.4rc0-cp39-abi3-linux_armv7l.whl (3 MB)
0.20.3	2024-11-05
tokenizers-0.20.3-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.20.3-cp311-cp311-linux_armv7l.whl (3 MB) tokenizers-0.20.3-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.20.3-cp39-cp39-linux_armv7l.whl (3 MB)
0.20.3rc0 pre-release	2024-11-15
tokenizers-0.20.3rc0-cp39-abi3-linux_armv6l.whl (3 MB) tokenizers-0.20.3rc0-cp39-abi3-linux_armv7l.whl (3 MB)
0.20.2	2024-11-04
tokenizers-0.20.2-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.20.2-cp311-cp311-linux_armv7l.whl (3 MB) tokenizers-0.20.2-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.20.2-cp39-cp39-linux_armv7l.whl (3 MB)
0.20.1	2024-10-10
tokenizers-0.20.1-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.20.1-cp39-cp39-linux_armv7l.whl (3 MB)
0.20.1rc1 pre-release	2024-10-10
tokenizers-0.20.1rc1-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.20.1rc1-cp311-cp311-linux_armv7l.whl (3 MB) tokenizers-0.20.1rc1-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.20.1rc1-cp39-cp39-linux_armv7l.whl (3 MB)
0.20.0	2024-08-08
tokenizers-0.20.0-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.20.0-cp39-cp39-linux_armv7l.whl (3 MB) tokenizers-0.20.0-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.20.0-cp311-cp311-linux_armv7l.whl (3 MB)
0.20.0rc1 pre-release	2024-08-08
0.19.1	2024-04-17
tokenizers-0.19.1-cp37-cp37m-linux_armv6l.whl (3 MB) tokenizers-0.19.1-cp37-cp37m-linux_armv7l.whl (3 MB) tokenizers-0.19.1-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.19.1-cp311-cp311-linux_armv7l.whl (3 MB) tokenizers-0.19.1-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.19.1-cp39-cp39-linux_armv7l.whl (3 MB)
0.19.0	2024-04-17
tokenizers-0.19.0-cp37-cp37m-linux_armv6l.whl (3 MB) tokenizers-0.19.0-cp37-cp37m-linux_armv7l.whl (3 MB) tokenizers-0.19.0-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.19.0-cp39-cp39-linux_armv7l.whl (3 MB) tokenizers-0.19.0-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.19.0-cp311-cp311-linux_armv7l.whl (3 MB)
0.15.2	2024-02-12
tokenizers-0.15.2-cp311-cp311-linux_armv6l.whl (4 MB) tokenizers-0.15.2-cp311-cp311-linux_armv7l.whl (4 MB) tokenizers-0.15.2-cp39-cp39-linux_armv6l.whl (4 MB) tokenizers-0.15.2-cp39-cp39-linux_armv7l.whl (4 MB)
0.15.1	2024-01-22
tokenizers-0.15.1-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.15.1-cp311-cp311-linux_armv7l.whl (3 MB) tokenizers-0.15.1-cp37-cp37m-linux_armv6l.whl (4 MB) tokenizers-0.15.1-cp37-cp37m-linux_armv7l.whl (4 MB) tokenizers-0.15.1-cp39-cp39-linux_armv6l.whl (4 MB) tokenizers-0.15.1-cp39-cp39-linux_armv7l.whl (4 MB)
0.15.0	2023-11-14
tokenizers-0.15.0-cp311-cp311-linux_armv6l.whl (4 MB) tokenizers-0.15.0-cp311-cp311-linux_armv7l.whl (4 MB) tokenizers-0.15.0-cp37-cp37m-linux_armv6l.whl (4 MB) tokenizers-0.15.0-cp37-cp37m-linux_armv7l.whl (4 MB) tokenizers-0.15.0-cp39-cp39-linux_armv6l.whl (4 MB) tokenizers-0.15.0-cp39-cp39-linux_armv7l.whl (4 MB)
0.14.1	2023-10-06
tokenizers-0.14.1-cp311-cp311-linux_armv6l.whl (4 MB) tokenizers-0.14.1-cp311-cp311-linux_armv7l.whl (4 MB) tokenizers-0.14.1-cp37-cp37m-linux_armv6l.whl (4 MB) tokenizers-0.14.1-cp37-cp37m-linux_armv7l.whl (4 MB) tokenizers-0.14.1-cp39-cp39-linux_armv6l.whl (4 MB) tokenizers-0.14.1-cp39-cp39-linux_armv7l.whl (4 MB)
0.14.0	2023-09-07
tokenizers-0.14.0-cp39-cp39-linux_armv6l.whl (3 MB) tokenizers-0.14.0-cp39-cp39-linux_armv7l.whl (3 MB) tokenizers-0.14.0-cp37-cp37m-linux_armv6l.whl (3 MB) tokenizers-0.14.0-cp37-cp37m-linux_armv7l.whl (3 MB) tokenizers-0.14.0-cp311-cp311-linux_armv6l.whl (4 MB) tokenizers-0.14.0-cp311-cp311-linux_armv7l.whl (4 MB)
0.13.3	2023-04-05
tokenizers-0.13.3-cp37-cp37m-linux_armv6l.whl (5 MB) tokenizers-0.13.3-cp37-cp37m-linux_armv7l.whl (5 MB) tokenizers-0.13.3-cp39-cp39-linux_armv6l.whl (5 MB) tokenizers-0.13.3-cp39-cp39-linux_armv7l.whl (5 MB) tokenizers-0.13.3-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.13.3-cp311-cp311-linux_armv7l.whl (5 MB)
0.13.2	2022-11-07
tokenizers-0.13.2-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.13.2-cp311-cp311-linux_armv7l.whl (5 MB)
0.13.1	2022-10-06
0.13.0	2022-09-21
tokenizers-0.13.0-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.13.0-cp311-cp311-linux_armv7l.whl (5 MB)
0.12.1	2022-04-13
tokenizers-0.12.1-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.12.1-cp311-cp311-linux_armv7l.whl (5 MB)
0.12.0 yanked	2022-03-31
tokenizers-0.12.0-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.12.0-cp311-cp311-linux_armv7l.whl (5 MB)
0.11.6	2022-02-28
tokenizers-0.11.6-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.6-cp311-cp311-linux_armv7l.whl (5 MB)
0.11.5	2022-02-16
tokenizers-0.11.5-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.5-cp311-cp311-linux_armv7l.whl (5 MB)
0.11.4	2022-01-17
tokenizers-0.11.4-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.4-cp311-cp311-linux_armv7l.whl (5 MB) tokenizers-0.11.4-cp39-cp39-linux_armv6l.whl (6 MB) tokenizers-0.11.4-cp39-cp39-linux_armv7l.whl (6 MB)
0.11.3	2022-01-17
tokenizers-0.11.3-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.3-cp311-cp311-linux_armv7l.whl (5 MB) tokenizers-0.11.3-cp39-cp39-linux_armv6l.whl (6 MB) tokenizers-0.11.3-cp39-cp39-linux_armv7l.whl (6 MB)
0.11.2	2022-01-04
tokenizers-0.11.2-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.2-cp311-cp311-linux_armv7l.whl (5 MB) tokenizers-0.11.2-cp39-cp39-linux_armv6l.whl (6 MB) tokenizers-0.11.2-cp39-cp39-linux_armv7l.whl (6 MB)
0.11.1	2021-12-28
tokenizers-0.11.1-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.1-cp311-cp311-linux_armv7l.whl (5 MB) tokenizers-0.11.1-cp39-cp39-linux_armv6l.whl (6 MB) tokenizers-0.11.1-cp39-cp39-linux_armv7l.whl (6 MB)
0.11.0	2021-12-23
tokenizers-0.11.0-cp311-cp311-linux_armv6l.whl (5 MB) tokenizers-0.11.0-cp311-cp311-linux_armv7l.whl (5 MB) tokenizers-0.11.0-cp39-cp39-linux_armv6l.whl (6 MB) tokenizers-0.11.0-cp39-cp39-linux_armv7l.whl (6 MB)
0.10.3	2021-05-24
tokenizers-0.10.3-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.10.3-cp311-cp311-linux_armv7l.whl (3 MB)
0.10.2	2021-04-05
tokenizers-0.10.2-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.10.2-cp311-cp311-linux_armv7l.whl (3 MB)
0.10.1	2021-02-08
tokenizers-0.10.1-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.10.1-cp311-cp311-linux_armv7l.whl (3 MB)
0.10.1rc1 pre-release	2021-02-04
tokenizers-0.10.1rc1-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.10.1rc1-cp311-cp311-linux_armv7l.whl (3 MB)
0.10.0	2021-01-12
tokenizers-0.10.0-cp311-cp311-linux_armv6l.whl (3 MB) tokenizers-0.10.0-cp311-cp311-linux_armv7l.whl (3 MB)
0.10.0rc1 pre-release	2020-12-08
0.9.4	2020-11-10
0.9.3	2020-10-26
0.9.2	2020-10-15
0.9.1	2020-10-13
0.9.0	2020-10-09
0.9.0rc2 pre-release	2020-10-06
0.9.0rc1 pre-release	2020-09-29
0.9.0.dev4 pre-release	2020-09-24
0.9.0.dev3 pre-release	2020-09-18
0.9.0.dev2 pre-release	2020-09-14
0.9.0.dev1 pre-release	2020-09-14
0.9.0.dev0 pre-release	2020-08-21
0.8.1	2020-07-20
0.8.1rc2 pre-release	2020-07-17
0.8.1rc1 pre-release	2020-07-06
0.8.0	2020-06-26
0.8.0rc4 pre-release	2020-06-26
0.8.0rc3 pre-release	2020-06-22
0.8.0rc2 pre-release	2020-06-19
0.8.0rc1 pre-release	2020-06-11
0.8.0.dev2 pre-release	2020-06-03
0.8.0.dev1 pre-release	2020-05-27
0.8.0.dev0 pre-release	2020-05-21
0.7.0	2020-04-17
0.7.0rc7 pre-release	2020-04-16
0.7.0rc6 pre-release	2020-04-16
0.7.0rc5 pre-release	2020-04-09
0.7.0rc4 pre-release	2020-04-08
0.7.0rc3 pre-release	2020-03-31
0.7.0rc2 pre-release	2020-03-27
0.7.0rc1 pre-release	2020-03-26
0.6.0	2020-03-02
0.5.2	2020-02-24
tokenizers-0.5.2-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.5.2-cp311-cp311-linux_armv7l.whl (2 MB)
0.5.1	2020-02-24
tokenizers-0.5.1-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.5.1-cp311-cp311-linux_armv7l.whl (2 MB)
0.5.0	2020-02-19
tokenizers-0.5.0-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.5.0-cp311-cp311-linux_armv7l.whl (2 MB)
0.4.2	2020-02-11
tokenizers-0.4.2-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.4.2-cp311-cp311-linux_armv7l.whl (2 MB)
0.4.1	2020-02-11
tokenizers-0.4.1-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.4.1-cp311-cp311-linux_armv7l.whl (2 MB)
0.4.0	2020-02-10
tokenizers-0.4.0-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.4.0-cp311-cp311-linux_armv7l.whl (2 MB)
0.3.0	2020-02-05
tokenizers-0.3.0-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.3.0-cp311-cp311-linux_armv7l.whl (2 MB)
0.2.1	2020-01-22
tokenizers-0.2.1-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.2.1-cp311-cp311-linux_armv7l.whl (2 MB)
0.2.0	2020-01-20
tokenizers-0.2.0-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.2.0-cp311-cp311-linux_armv7l.whl (2 MB)
0.1.1	2020-01-12
tokenizers-0.1.1-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.1.1-cp311-cp311-linux_armv7l.whl (2 MB)
0.1.0	2020-01-10
tokenizers-0.1.0-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.1.0-cp311-cp311-linux_armv7l.whl (2 MB)
0.0.13	2020-01-08
tokenizers-0.0.13-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.0.13-cp311-cp311-linux_armv7l.whl (2 MB)
0.0.12	2020-01-07
tokenizers-0.0.12-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.0.12-cp311-cp311-linux_armv7l.whl (2 MB)
0.0.11	2019-12-27
tokenizers-0.0.11-cp311-cp311-linux_armv6l.whl (2 MB) tokenizers-0.0.11-cp311-cp311-linux_armv7l.whl (2 MB)
0.0.10	2019-12-26
0.0.9	2019-12-23
0.0.8	2019-12-20
0.0.7	2019-12-17
0.0.6	2019-12-17
0.0.5	2019-12-13
0.0.4	2019-12-10
0.0.3	2019-12-03
0.0.2	2019-12-02
0.0.1	2019-11-01

Issues with this package?

• Search issues for this package
• Package or version missing? Open a new issue
• Something else? Open a new issue