tokenizer

A fast, compact pure-Python tokenizer for Icelandic text with sentence segmentation

Installation

In a virtualenv (see these instructions if you need to create one):

pip3 install tokenizer

Dependencies

  • None
PyPI page
pypi.org/project/tokenizer
Project JSON
piwheels.org/project/tokenizer/json
Versions
63
Files
63
Downloads (all time)
loading...
Downloads (last 30 days)
loading...

Releases

Version Released Bullseye
Python 3.9		 Bookworm
Python 3.11		 Trixie
Python 3.13		 Files
3.6.2 2025-12-12      
3.6.0 2025-12-11      
3.5.5 2025-11-13      
3.5.4 2025-11-12      
3.5.3 2025-09-26      
3.5.2 2025-09-26      
3.5.1 2025-08-28      
3.5.0 2025-08-27      
3.4.5 2024-08-23      
3.4.4 2024-08-07      
3.4.3 2023-08-11  
3.4.2 2022-10-07  
3.4.1 2022-05-03  
3.4.0 2022-03-10  
3.3.2 2021-09-27  
3.3.1 2021-09-08  
3.3.0 2021-09-08  
3.2.0 2021-08-16  
3.1.2 2021-06-02  
3.1.1 2021-05-10  
3.1.0 2021-04-14  
3.0.0 2021-04-09  
2.5.0 2021-03-08  
2.4.0 2020-10-08  
2.3.1 2020-09-21  
2.3.0 2020-09-03  
2.2.0 2020-08-20  
2.1.0 2020-07-02  
2.0.7 2020-06-24  
2.0.6 2020-05-29  
2.0.5 2020-03-26  
2.0.4 2020-02-17  
2.0.3 2019-12-17  
2.0.2 2019-12-11  
2.0.1 2019-12-09  
2.0.0 2019-12-04  
1.4.1 2019-10-22  
1.4.0 2019-07-16  
1.3.0 2019-05-21  
1.2.3 2019-05-03  
1.2.2 2019-04-26  
1.2.1 2019-02-18  
1.2.0 2019-02-07  
1.1.2 2019-01-10  
1.1.1 2019-01-04  
1.1.0 2019-01-02  
1.0.9 2018-12-29  
1.0.8 2018-10-30  
1.0.7 2018-09-25  
1.0.6 2018-08-23  
1.0.5 2018-07-23  
1.0.4 2018-06-08  
1.0.3 2018-04-30  
1.0.2 2018-04-25  
1.0.1 2018-04-24  
1.0.0 2018-04-20  
0.1.14 2018-04-11  
0.1.12 2018-04-05  
0.1.11 2018-03-27  
0.1.10 2018-03-13  
0.1.3 2017-10-01  
0.1.2 2017-10-01  
0.1.1 2017-10-01  

Issues with this package?

Key

Build succeeded
Build failed
Build skipped
Build pending

Page last updated 2025-12-12 16:55:15 UTC