tokenizer
A fast, compact pure-Python tokenizer for Icelandic text with sentence segmentation
Installation
In a virtualenv (see these instructions if you need to create one):
pip3 install tokenizer
Dependencies
- None
Releases
| Version | Released | Bookworm Python 3.11 |
Trixie Python 3.13 |
Files |
|---|---|---|---|---|
| 3.6.4 | 2026-07-10 | |||
|
||||
| 3.6.3 | 2026-07-09 | |||
|
||||
| 3.6.2 | 2025-12-12 | |||
|
||||
| 3.6.0 | 2025-12-11 | |||
|
||||
| 3.5.5 | 2025-11-13 | |||
|
||||
| 3.5.4 | 2025-11-12 | |||
|
||||
| 3.5.3 | 2025-09-26 | |||
|
||||
| 3.5.2 | 2025-09-26 | |||
|
||||
| 3.5.1 | 2025-08-28 | |||
|
||||
| 3.5.0 | 2025-08-27 | |||
|
||||
| 3.4.5 | 2024-08-23 | |||
|
||||
| 3.4.4 | 2024-08-07 | |||
|
||||
| 3.4.3 | 2023-08-11 | |||
|
||||
| 3.4.2 | 2022-10-07 | |||
| 3.4.1 | 2022-05-03 | |||
| 3.4.0 | 2022-03-10 | |||
| 3.3.2 | 2021-09-27 | |||
| 3.3.1 | 2021-09-08 | |||
| 3.3.0 | 2021-09-08 | |||
| 3.2.0 | 2021-08-16 | |||
| 3.1.2 | 2021-06-02 | |||
| 3.1.1 | 2021-05-10 | |||
| 3.1.0 | 2021-04-14 | |||
| 3.0.0 | 2021-04-09 | |||
| 2.5.0 | 2021-03-08 | |||
|
||||
| 2.4.0 | 2020-10-08 | |||
|
||||
| 2.3.1 | 2020-09-21 | |||
|
||||
| 2.3.0 | 2020-09-03 | |||
|
||||
| 2.2.0 | 2020-08-20 | |||
|
||||
| 2.1.0 | 2020-07-02 | |||
|
||||
| 2.0.7 | 2020-06-24 | |||
|
||||
| 2.0.6 | 2020-05-29 | |||
|
||||
| 2.0.5 | 2020-03-26 | |||
|
||||
| 2.0.4 | 2020-02-17 | |||
|
||||
| 2.0.3 | 2019-12-17 | |||
|
||||
| 2.0.2 | 2019-12-11 | |||
|
||||
| 2.0.1 | 2019-12-09 | |||
|
||||
| 2.0.0 | 2019-12-04 | |||
|
||||
| 1.4.1 | 2019-10-22 | |||
| 1.4.0 | 2019-07-16 | |||
| 1.3.0 | 2019-05-21 | |||
| 1.2.3 | 2019-05-03 | |||
| 1.2.2 | 2019-04-26 | |||
| 1.2.1 | 2019-02-18 | |||
| 1.2.0 | 2019-02-07 | |||
| 1.1.2 | 2019-01-10 | |||
| 1.1.1 | 2019-01-04 | |||
| 1.1.0 | 2019-01-02 | |||
| 1.0.9 | 2018-12-29 | |||
| 1.0.8 | 2018-10-30 | |||
| 1.0.7 | 2018-09-25 | |||
| 1.0.6 | 2018-08-23 | |||
| 1.0.5 | 2018-07-23 | |||
| 1.0.4 | 2018-06-08 | |||
| 1.0.3 | 2018-04-30 | |||
| 1.0.2 | 2018-04-25 | |||
| 1.0.1 | 2018-04-24 | |||
| 1.0.0 | 2018-04-20 | |||
| 0.1.14 | 2018-04-11 | |||
| 0.1.12 | 2018-04-05 | |||
| 0.1.11 | 2018-03-27 | |||
| 0.1.10 | 2018-03-13 | |||
| 0.1.3 | 2017-10-01 | |||
| 0.1.2 | 2017-10-01 | |||
|
||||
| 0.1.1 | 2017-10-01 | |||
|
||||
Issues with this package?
- Search issues for this package
- Package or version missing? Open a new issue
- Something else? Open a new issue