2023-12-14T04:01:22,297 Created temporary directory: /tmp/pip-build-tracker-bu_lv4r9 2023-12-14T04:01:22,299 Initialized build tracking at /tmp/pip-build-tracker-bu_lv4r9 2023-12-14T04:01:22,300 Created build tracker: /tmp/pip-build-tracker-bu_lv4r9 2023-12-14T04:01:22,301 Entered build tracker: /tmp/pip-build-tracker-bu_lv4r9 2023-12-14T04:01:22,302 Created temporary directory: /tmp/pip-wheel-vh21srzp 2023-12-14T04:01:22,305 Created temporary directory: /tmp/pip-ephem-wheel-cache-4ou8a3k3 2023-12-14T04:01:22,332 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2023-12-14T04:01:22,335 2 location(s) to search for versions of i18n-json: 2023-12-14T04:01:22,335 * https://pypi.org/simple/i18n-json/ 2023-12-14T04:01:22,335 * https://www.piwheels.org/simple/i18n-json/ 2023-12-14T04:01:22,336 Fetching project page and analyzing links: https://pypi.org/simple/i18n-json/ 2023-12-14T04:01:22,337 Getting page https://pypi.org/simple/i18n-json/ 2023-12-14T04:01:22,338 Found index url https://pypi.org/simple/ 2023-12-14T04:01:22,472 Fetched page https://pypi.org/simple/i18n-json/ as application/vnd.pypi.simple.v1+json 2023-12-14T04:01:22,474 Skipping link: No binaries permitted for i18n-json: https://files.pythonhosted.org/packages/94/7e/e050c95102a9db3cb1efe7b0b2e821163c801569aa4040eea50f47167a85/i18n_json-0.0.8-cp310-cp310-win_amd64.whl (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T04:01:22,475 Found link https://files.pythonhosted.org/packages/52/f3/b4ee6454f577112fb5ce9650efa1a9358a204b4eb243bd1b22c9730ebec4/i18n_json-0.0.8.tar.gz (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8), version: 0.0.8 2023-12-14T04:01:22,476 Skipping link: No binaries permitted for i18n-json: https://files.pythonhosted.org/packages/d7/e2/f2e1e5e88725bea9939fbced144bee10fac4387c2c163be39d05a6ab955e/i18n_json-0.0.9-cp310-cp310-win_amd64.whl (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T04:01:22,477 Found link https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8), version: 0.0.9 2023-12-14T04:01:22,478 Fetching project page and analyzing links: https://www.piwheels.org/simple/i18n-json/ 2023-12-14T04:01:22,478 Getting page https://www.piwheels.org/simple/i18n-json/ 2023-12-14T04:01:22,480 Found index url https://www.piwheels.org/simple/ 2023-12-14T04:01:22,647 Fetched page https://www.piwheels.org/simple/i18n-json/ as text/html 2023-12-14T04:01:22,649 Skipping link: not a file: https://www.piwheels.org/simple/i18n-json/ 2023-12-14T04:01:22,650 Skipping link: not a file: https://pypi.org/simple/i18n-json/ 2023-12-14T04:01:22,671 Given no hashes to check 1 links for project 'i18n-json': discarding no candidates 2023-12-14T04:01:22,693 Collecting i18n-json==0.0.9 2023-12-14T04:01:22,696 Created temporary directory: /tmp/pip-unpack-du14yhag 2023-12-14T04:01:23,016 Downloading i18n_json-0.0.9.tar.gz (219 kB) 2023-12-14T04:01:23,996 Added i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz to build tracker '/tmp/pip-build-tracker-bu_lv4r9' 2023-12-14T04:01:24,003 Created temporary directory: /tmp/pip-build-env-jcnys5g_ 2023-12-14T04:01:24,008 Installing build dependencies: started 2023-12-14T04:01:24,010 Running command pip subprocess to install build dependencies 2023-12-14T04:01:25,216 Using pip 23.3.1 from /home/piwheels/.local/lib/python3.11/site-packages/pip (python 3.11) 2023-12-14T04:01:25,790 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2023-12-14T04:01:27,208 Collecting setuptools 2023-12-14T04:01:27,230 Using cached https://www.piwheels.org/simple/setuptools/setuptools-69.0.2-py3-none-any.whl (819 kB) 2023-12-14T04:01:27,484 Collecting wheel 2023-12-14T04:01:27,500 Using cached https://www.piwheels.org/simple/wheel/wheel-0.42.0-py3-none-any.whl (65 kB) 2023-12-14T04:01:28,696 Collecting cython 2023-12-14T04:01:28,713 Using cached https://www.piwheels.org/simple/cython/Cython-3.0.6-cp311-cp311-linux_armv7l.whl (11.7 MB) 2023-12-14T04:01:30,127 Link requires a different Python (3.11.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://files.pythonhosted.org/packages/36/3b/fa4025a424adafd85c6195001b1c130ecb8d8b30784a1c4cb68e7b5e5ae7/pylint-1.9.5-py2.py3-none-any.whl (from https://pypi.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T04:01:30,128 Link requires a different Python (3.11.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://files.pythonhosted.org/packages/3f/0b/4e7eeab1abf594b447385a340593c1a4244cdf8e54a78edcae1e2756d6fb/pylint-1.9.5.tar.gz (from https://pypi.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T04:01:30,304 Link requires a different Python (3.11.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://www.piwheels.org/simple/pylint/pylint-1.9.5-py2.py3-none-any.whl#sha256=367e3d49813d349a905390ac27989eff82ab84958731c5ef0bef867452cfdc42 (from https://www.piwheels.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T04:01:30,397 Collecting pylint 2023-12-14T04:01:30,411 Downloading https://www.piwheels.org/simple/pylint/pylint-3.0.3-py3-none-any.whl (510 kB) 2023-12-14T04:01:30,462 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 510.6/510.6 kB 12.1 MB/s eta 0:00:00 2023-12-14T04:01:31,133 Collecting psutil 2023-12-14T04:01:31,152 Using cached https://www.piwheels.org/simple/psutil/psutil-5.9.6-cp311-abi3-linux_armv7l.whl (278 kB) 2023-12-14T04:01:31,574 Collecting platformdirs>=2.2.0 (from pylint) 2023-12-14T04:01:31,590 Using cached https://www.piwheels.org/simple/platformdirs/platformdirs-4.1.0-py3-none-any.whl (17 kB) 2023-12-14T04:01:31,935 Collecting astroid<=3.1.0-dev0,>=3.0.1 (from pylint) 2023-12-14T04:01:31,947 Downloading https://www.piwheels.org/simple/astroid/astroid-3.0.2-py3-none-any.whl (275 kB) 2023-12-14T04:01:31,985 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 275.2/275.2 kB 8.8 MB/s eta 0:00:00 2023-12-14T04:01:32,630 Collecting isort!=5.13.0,<6,>=4.2.5 (from pylint) 2023-12-14T04:01:32,640 Obtaining dependency information for isort!=5.13.0,<6,>=4.2.5 from https://files.pythonhosted.org/packages/d1/b3/8def84f539e7d2289a02f0524b944b15d7c75dab7628bedf1c4f0992029c/isort-5.13.2-py3-none-any.whl.metadata 2023-12-14T04:01:32,794 Downloading isort-5.13.2-py3-none-any.whl.metadata (12 kB) 2023-12-14T04:01:32,889 Collecting mccabe<0.8,>=0.6 (from pylint) 2023-12-14T04:01:32,899 Downloading https://www.piwheels.org/simple/mccabe/mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB) 2023-12-14T04:01:33,100 Collecting tomlkit>=0.10.1 (from pylint) 2023-12-14T04:01:33,114 Using cached https://www.piwheels.org/simple/tomlkit/tomlkit-0.12.3-py3-none-any.whl (37 kB) 2023-12-14T04:01:33,219 Collecting dill>=0.3.6 (from pylint) 2023-12-14T04:01:33,229 Downloading https://www.piwheels.org/simple/dill/dill-0.3.7-py3-none-any.whl (115 kB) 2023-12-14T04:01:33,250 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 8.3 MB/s eta 0:00:00 2023-12-14T04:01:33,396 Downloading isort-5.13.2-py3-none-any.whl (92 kB) 2023-12-14T04:01:33,449 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.3/92.3 kB 1.8 MB/s eta 0:00:00 2023-12-14T04:01:36,131 Installing collected packages: wheel, tomlkit, setuptools, psutil, platformdirs, mccabe, isort, dill, cython, astroid, pylint 2023-12-14T04:01:36,372 Creating /tmp/pip-build-env-jcnys5g_/overlay/local/bin 2023-12-14T04:01:36,374 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/wheel to 755 2023-12-14T04:01:40,061 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/isort to 755 2023-12-14T04:01:40,063 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/isort-identify-imports to 755 2023-12-14T04:01:45,450 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/cygdb to 755 2023-12-14T04:01:45,453 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/cython to 755 2023-12-14T04:01:45,457 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/cythonize to 755 2023-12-14T04:01:47,903 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/pylint to 755 2023-12-14T04:01:47,905 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/pylint-config to 755 2023-12-14T04:01:47,907 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/pyreverse to 755 2023-12-14T04:01:47,909 changing mode of /tmp/pip-build-env-jcnys5g_/overlay/local/bin/symilar to 755 2023-12-14T04:01:48,037 Successfully installed astroid-3.0.2 cython-3.0.6 dill-0.3.7 isort-5.13.2 mccabe-0.7.0 platformdirs-4.1.0 psutil-5.9.6 pylint-3.0.3 setuptools-69.0.2 tomlkit-0.12.3 wheel-0.42.0 2023-12-14T04:01:48,742 Installing build dependencies: finished with status 'done' 2023-12-14T04:01:48,745 Getting requirements to build wheel: started 2023-12-14T04:01:48,746 Running command Getting requirements to build wheel 2023-12-14T04:01:49,382 # i18n_json 2023-12-14T04:01:49,383 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T04:01:49,384 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T04:01:49,385 Features: 2023-12-14T04:01:49,385 * Fast (compared to other sharing solutions) 2023-12-14T04:01:49,386 * No running manager processes 2023-12-14T04:01:49,386 * Works in spawn and fork context 2023-12-14T04:01:49,387 * Safe locking between independent processes 2023-12-14T04:01:49,388 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T04:01:49,388 * Convenient, no setter or getters necessary 2023-12-14T04:01:49,389 * Optional recursion for nested dicts 2023-12-14T04:01:49,390 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T04:01:49,390 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T04:01:49,391 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T04:01:49,392 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T04:01:49,393 ## General Concept 2023-12-14T04:01:49,394 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T04:01:49,396 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T04:01:49,397 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T04:01:49,397 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T04:01:49,398 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T04:01:49,398 streaming updates afterwards. 2023-12-14T04:01:49,399 ## Issues 2023-12-14T04:01:49,401 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T04:01:49,401 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T04:01:49,402 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T04:01:49,402 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T04:01:49,404 ## Alternatives 2023-12-14T04:01:49,405 There are many alternatives: 2023-12-14T04:01:49,405 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T04:01:49,406 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T04:01:49,406 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T04:01:49,407 * Redis 2023-12-14T04:01:49,407 * Memcached 2023-12-14T04:01:49,408 ## How to use? 2023-12-14T04:01:49,409 ### Simple 2023-12-14T04:01:49,409 In one Python REPL: 2023-12-14T04:01:49,410 ```python 2023-12-14T04:01:49,410 Python 3.9.2 on linux 2023-12-14T04:01:49,411 >>> 2023-12-14T04:01:49,411 >>> from i18n_json import i18n_json 2023-12-14T04:01:49,411 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T04:01:49,412 >>> ultra 2023-12-14T04:01:49,412 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:49,413 >>> 2023-12-14T04:01:49,413 >>> # We need the shared memory name in the other process. 2023-12-14T04:01:49,413 >>> ultra.name 2023-12-14T04:01:49,414 'psm_ad73da69' 2023-12-14T04:01:49,414 ``` 2023-12-14T04:01:49,415 In another Python REPL: 2023-12-14T04:01:49,415 ```python 2023-12-14T04:01:49,416 Python 3.9.2 on linux 2023-12-14T04:01:49,416 >>> 2023-12-14T04:01:49,416 >>> from i18n_json import i18n_json 2023-12-14T04:01:49,417 >>> # Connect to the shared memory with the name above 2023-12-14T04:01:49,417 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T04:01:49,418 >>> other 2023-12-14T04:01:49,418 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:49,419 >>> other[2] = 2 2023-12-14T04:01:49,420 ``` 2023-12-14T04:01:49,421 Back in the first Python REPL: 2023-12-14T04:01:49,421 ```python 2023-12-14T04:01:49,422 >>> ultra[2] 2023-12-14T04:01:49,422 2 2023-12-14T04:01:49,423 ``` 2023-12-14T04:01:49,424 ### Nested 2023-12-14T04:01:49,425 In one Python REPL: 2023-12-14T04:01:49,426 ```python 2023-12-14T04:01:49,426 Python 3.9.2 on linux 2023-12-14T04:01:49,427 >>> 2023-12-14T04:01:49,428 >>> from i18n_json import i18n_json 2023-12-14T04:01:49,428 >>> ultra = i18n_json(recurse=True) 2023-12-14T04:01:49,429 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T04:01:49,429 >>> type(ultra['nested']) 2023-12-14T04:01:49,430 2023-12-14T04:01:49,430 >>> ultra.name 2023-12-14T04:01:49,431 'psm_0a2713e4' 2023-12-14T04:01:49,431 ``` 2023-12-14T04:01:49,433 In another Python REPL: 2023-12-14T04:01:49,433 ```python 2023-12-14T04:01:49,434 Python 3.9.2 on linux 2023-12-14T04:01:49,434 >>> 2023-12-14T04:01:49,435 >>> from i18n_json import i18n_json 2023-12-14T04:01:49,435 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T04:01:49,436 >>> other['nested']['counter'] += 1 2023-12-14T04:01:49,436 ``` 2023-12-14T04:01:49,437 Back in the first Python REPL: 2023-12-14T04:01:49,437 ```python 2023-12-14T04:01:49,438 >>> ultra['nested']['counter'] 2023-12-14T04:01:49,439 1 2023-12-14T04:01:49,439 ``` 2023-12-14T04:01:49,440 ## Performance comparison 2023-12-14T04:01:49,441 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T04:01:49,442 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T04:01:49,443 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T04:01:49,444 ```python 2023-12-14T04:01:49,444 Python 3.9.2 on linux 2023-12-14T04:01:49,445 >>> 2023-12-14T04:01:49,445 >>> from i18n_json import i18n_json 2023-12-14T04:01:49,446 >>> ultra = i18n_json() 2023-12-14T04:01:49,446 >>> for i in range(10_000): ultra[i] = i 2023-12-14T04:01:49,447 ... 2023-12-14T04:01:49,447 >>> len(ultra) 2023-12-14T04:01:49,448 10000 2023-12-14T04:01:49,448 >>> ultra[500] 2023-12-14T04:01:49,449 500 2023-12-14T04:01:49,449 >>> # Now let's do some performance testing 2023-12-14T04:01:49,449 >>> import multiprocessing, redis, timeit 2023-12-14T04:01:49,450 >>> orig = dict(ultra) 2023-12-14T04:01:49,450 >>> len(orig) 2023-12-14T04:01:49,450 10000 2023-12-14T04:01:49,451 >>> orig[500] 2023-12-14T04:01:49,451 500 2023-12-14T04:01:49,452 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T04:01:49,452 >>> len(managed) 2023-12-14T04:01:49,452 10000 2023-12-14T04:01:49,453 >>> r = redis.Redis() 2023-12-14T04:01:49,453 >>> r.flushall() 2023-12-14T04:01:49,454 >>> r.mset(orig) 2023-12-14T04:01:49,454 ``` 2023-12-14T04:01:49,455 ### Read performance 2023-12-14T04:01:49,455 >>> 2023-12-14T04:01:49,456 ```python 2023-12-14T04:01:49,456 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T04:01:49,457 0.03832335816696286 2023-12-14T04:01:49,457 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T04:01:49,457 0.5248982920311391 2023-12-14T04:01:49,458 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T04:01:49,458 40.85506196087226 2023-12-14T04:01:49,459 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T04:01:49,459 49.3497632863 2023-12-14T04:01:49,459 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T04:01:49,460 0.04309639008715749 2023-12-14T04:01:49,460 ``` 2023-12-14T04:01:49,461 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T04:01:49,462 ### Write performance 2023-12-14T04:01:49,463 ```python 2023-12-14T04:01:49,463 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T04:01:49,464 0.028232071083039045 2023-12-14T04:01:49,465 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T04:01:49,465 2.911152713932097 2023-12-14T04:01:49,466 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T04:01:49,466 31.641707635018975 2023-12-14T04:01:49,467 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T04:01:49,468 124.3432381930761 2023-12-14T04:01:49,468 ``` 2023-12-14T04:01:49,469 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T04:01:49,470 ### Testing performance 2023-12-14T04:01:49,472 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T04:01:49,473 ```bash 2023-12-14T04:01:49,473 python ./tests/performance/performance.py 2023-12-14T04:01:49,474 Testing Performance with 1000000 operations each 2023-12-14T04:01:49,475 Redis (writes) = 24,351 ops per second 2023-12-14T04:01:49,476 Redis (reads) = 30,466 ops per second 2023-12-14T04:01:49,476 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T04:01:49,476 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T04:01:49,477 Python dict (writes) = 16,413,569 ops per second 2023-12-14T04:01:49,477 Python dict (reads) = 16,479,191 ops per second 2023-12-14T04:01:49,478 i18n_json (writes) = 479,860 ops per second 2023-12-14T04:01:49,479 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T04:01:49,479 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T04:01:49,480 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T04:01:49,481 Ranking: 2023-12-14T04:01:49,481 writes: 2023-12-14T04:01:49,482 Python dict = 16,413,569 (factor 1.0) 2023-12-14T04:01:49,483 i18n_json = 479,860 (factor 34.2) 2023-12-14T04:01:49,483 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T04:01:49,484 Redis = 24,351 (factor 674.04) 2023-12-14T04:01:49,484 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T04:01:49,485 reads: 2023-12-14T04:01:49,485 Python dict = 16,479,191 (factor 1.0) 2023-12-14T04:01:49,486 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T04:01:49,486 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T04:01:49,486 Redis = 30,466 (factor 540.9) 2023-12-14T04:01:49,487 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T04:01:49,487 ``` 2023-12-14T04:01:49,488 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T04:01:49,489 ## Parameters 2023-12-14T04:01:49,489 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T04:01:49,490 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T04:01:49,491 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T04:01:49,491 memory space is attached. 2023-12-14T04:01:49,492 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T04:01:49,492 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T04:01:49,493 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T04:01:49,494 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T04:01:49,495 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T04:01:49,496 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T04:01:49,496 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T04:01:49,497 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T04:01:49,498 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T04:01:49,499 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T04:01:49,499 dict will automatically load the full dump and continue streaming updates. 2023-12-14T04:01:49,501 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T04:01:49,502 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T04:01:49,502 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T04:01:49,503 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T04:01:49,504 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T04:01:49,504 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T04:01:49,505 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T04:01:49,506 (Also see the section [Locking](#locking) below!) 2023-12-14T04:01:49,507 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T04:01:49,508 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T04:01:49,508 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T04:01:49,509 be writing to the dict and therefore creating full dumps. 2023-12-14T04:01:49,510 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T04:01:49,511 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T04:01:49,511 dict. 2023-12-14T04:01:49,512 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T04:01:49,512 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T04:01:49,513 ## Memory management 2023-12-14T04:01:49,514 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T04:01:49,515 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T04:01:49,515 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T04:01:49,516 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T04:01:49,516 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T04:01:49,517 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T04:01:49,517 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T04:01:49,518 **Buffer sizes and read performance:** 2023-12-14T04:01:49,519 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T04:01:49,520 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T04:01:49,521 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T04:01:49,523 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T04:01:49,524 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T04:01:49,525 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T04:01:49,526 ## Locking 2023-12-14T04:01:49,527 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T04:01:49,528 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T04:01:49,529 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T04:01:49,530 ### How to use the locking? 2023-12-14T04:01:49,530 ```python 2023-12-14T04:01:49,531 ultra = i18n_json(shared_lock=True) 2023-12-14T04:01:49,531 with ultra.lock: 2023-12-14T04:01:49,532 ultra['counter']++ 2023-12-14T04:01:49,532 # The same as above with all default parameters 2023-12-14T04:01:49,533 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T04:01:49,533 ultra['counter']++ 2023-12-14T04:01:49,534 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T04:01:49,534 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T04:01:49,535 with ultra.lock(sleep_time=0): 2023-12-14T04:01:49,535 ultra['counter']++ 2023-12-14T04:01:49,536 try: 2023-12-14T04:01:49,536 result = ultra.lock.acquire(block=False) 2023-12-14T04:01:49,537 ultra.lock.release() 2023-12-14T04:01:49,537 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T04:01:49,537 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T04:01:49,538 try: 2023-12-14T04:01:49,539 with ultra.lock(timeout=1.5): 2023-12-14T04:01:49,540 ultra['counter']++ 2023-12-14T04:01:49,540 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T04:01:49,540 print('Stale lock?') 2023-12-14T04:01:49,542 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T04:01:49,542 ultra['counter']++ 2023-12-14T04:01:49,543 ``` 2023-12-14T04:01:49,544 ## Explicit cleanup 2023-12-14T04:01:49,546 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T04:01:49,547 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T04:01:49,548 Another way to do this in code is like this: 2023-12-14T04:01:49,549 ```python 2023-12-14T04:01:49,549 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T04:01:49,550 name = 'my-dict-name' 2023-12-14T04:01:49,551 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T04:01:49,551 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T04:01:49,552 ``` 2023-12-14T04:01:49,553 ## Advanced usage 2023-12-14T04:01:49,554 See [examples](/examples) folder 2023-12-14T04:01:49,555 ```python 2023-12-14T04:01:49,555 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T04:01:49,556 >>> # Let's use a value with 100k bytes length. 2023-12-14T04:01:49,556 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T04:01:49,557 >>> ultra[0] = ' ' * 100_000 2023-12-14T04:01:49,557 >>> ultra.print_status() 2023-12-14T04:01:49,558 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T04:01:49,558 'buffer_size': 100000, 2023-12-14T04:01:49,559 'control': SharedMemory('my-name', size=1000), 2023-12-14T04:01:49,560 'full_dump_counter': 1, 2023-12-14T04:01:49,560 'full_dump_counter_remote': 1, 2023-12-14T04:01:49,561 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T04:01:49,561 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T04:01:49,562 'full_dump_size': None, 2023-12-14T04:01:49,563 'full_dump_static_size_remote': , 2023-12-14T04:01:49,563 'lock': , 2023-12-14T04:01:49,564 'lock_pid_remote': 0, 2023-12-14T04:01:49,564 'lock_remote': 0, 2023-12-14T04:01:49,565 'name': 'my-name', 2023-12-14T04:01:49,565 'recurse': False, 2023-12-14T04:01:49,565 'recurse_remote': , 2023-12-14T04:01:49,566 'serializer': , 2023-12-14T04:01:49,566 'shared_lock_remote': , 2023-12-14T04:01:49,566 'update_stream_position': 0, 2023-12-14T04:01:49,567 'update_stream_position_remote': 0} 2023-12-14T04:01:49,567 ``` 2023-12-14T04:01:49,568 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T04:01:49,569 Other things you can do: 2023-12-14T04:01:49,569 ```python 2023-12-14T04:01:49,570 >>> # Create a full dump 2023-12-14T04:01:49,570 >>> ultra.dump() 2023-12-14T04:01:49,571 >>> # Load latest full dump if one is available 2023-12-14T04:01:49,571 >>> ultra.load() 2023-12-14T04:01:49,572 >>> # Show statistics 2023-12-14T04:01:49,573 >>> ultra.print_status() 2023-12-14T04:01:49,573 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T04:01:49,574 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T04:01:49,574 >>> ultra.load(force=True) 2023-12-14T04:01:49,575 >>> # Apply full dump and stream updates to 2023-12-14T04:01:49,575 >>> # underlying local dict, this is automatically 2023-12-14T04:01:49,576 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T04:01:49,576 >>> # but can be useful to call after a forced load. 2023-12-14T04:01:49,577 >>> ultra.apply_update() 2023-12-14T04:01:49,577 >>> # Access underlying local dict directly for maximum performance 2023-12-14T04:01:49,578 >>> ultra.data 2023-12-14T04:01:49,579 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T04:01:49,580 >>> import jsons 2023-12-14T04:01:49,580 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T04:01:49,581 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T04:01:49,582 >>> ultra.close() 2023-12-14T04:01:49,583 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T04:01:49,583 >>> ultra.unlink() 2023-12-14T04:01:49,584 ``` 2023-12-14T04:01:49,585 ## Contributing 2023-12-14T04:01:49,586 Contributions are always welcome! 2023-12-14T04:01:52,231 Compiling i18n_json.py because it changed. 2023-12-14T04:01:52,232 [1/1] Cythonizing i18n_json.py 2023-12-14T04:01:52,532 Getting requirements to build wheel: finished with status 'done' 2023-12-14T04:01:52,557 Created temporary directory: /tmp/pip-modern-metadata-b8n_lufg 2023-12-14T04:01:52,560 Preparing metadata (pyproject.toml): started 2023-12-14T04:01:52,561 Running command Preparing metadata (pyproject.toml) 2023-12-14T04:01:53,186 # i18n_json 2023-12-14T04:01:53,187 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T04:01:53,188 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T04:01:53,189 Features: 2023-12-14T04:01:53,190 * Fast (compared to other sharing solutions) 2023-12-14T04:01:53,190 * No running manager processes 2023-12-14T04:01:53,191 * Works in spawn and fork context 2023-12-14T04:01:53,192 * Safe locking between independent processes 2023-12-14T04:01:53,192 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T04:01:53,193 * Convenient, no setter or getters necessary 2023-12-14T04:01:53,193 * Optional recursion for nested dicts 2023-12-14T04:01:53,195 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T04:01:53,195 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T04:01:53,196 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T04:01:53,197 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T04:01:53,198 ## General Concept 2023-12-14T04:01:53,199 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T04:01:53,199 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T04:01:53,200 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T04:01:53,201 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T04:01:53,201 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T04:01:53,201 streaming updates afterwards. 2023-12-14T04:01:53,202 ## Issues 2023-12-14T04:01:53,203 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T04:01:53,203 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T04:01:53,203 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T04:01:53,204 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T04:01:53,205 ## Alternatives 2023-12-14T04:01:53,206 There are many alternatives: 2023-12-14T04:01:53,207 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T04:01:53,208 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T04:01:53,208 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T04:01:53,209 * Redis 2023-12-14T04:01:53,209 * Memcached 2023-12-14T04:01:53,211 ## How to use? 2023-12-14T04:01:53,212 ### Simple 2023-12-14T04:01:53,213 In one Python REPL: 2023-12-14T04:01:53,213 ```python 2023-12-14T04:01:53,214 Python 3.9.2 on linux 2023-12-14T04:01:53,214 >>> 2023-12-14T04:01:53,215 >>> from i18n_json import i18n_json 2023-12-14T04:01:53,215 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T04:01:53,216 >>> ultra 2023-12-14T04:01:53,217 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:53,217 >>> 2023-12-14T04:01:53,218 >>> # We need the shared memory name in the other process. 2023-12-14T04:01:53,218 >>> ultra.name 2023-12-14T04:01:53,219 'psm_ad73da69' 2023-12-14T04:01:53,219 ``` 2023-12-14T04:01:53,220 In another Python REPL: 2023-12-14T04:01:53,221 ```python 2023-12-14T04:01:53,221 Python 3.9.2 on linux 2023-12-14T04:01:53,221 >>> 2023-12-14T04:01:53,222 >>> from i18n_json import i18n_json 2023-12-14T04:01:53,222 >>> # Connect to the shared memory with the name above 2023-12-14T04:01:53,223 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T04:01:53,223 >>> other 2023-12-14T04:01:53,224 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:53,224 >>> other[2] = 2 2023-12-14T04:01:53,225 ``` 2023-12-14T04:01:53,226 Back in the first Python REPL: 2023-12-14T04:01:53,226 ```python 2023-12-14T04:01:53,227 >>> ultra[2] 2023-12-14T04:01:53,228 2 2023-12-14T04:01:53,228 ``` 2023-12-14T04:01:53,229 ### Nested 2023-12-14T04:01:53,230 In one Python REPL: 2023-12-14T04:01:53,231 ```python 2023-12-14T04:01:53,232 Python 3.9.2 on linux 2023-12-14T04:01:53,232 >>> 2023-12-14T04:01:53,233 >>> from i18n_json import i18n_json 2023-12-14T04:01:53,233 >>> ultra = i18n_json(recurse=True) 2023-12-14T04:01:53,234 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T04:01:53,235 >>> type(ultra['nested']) 2023-12-14T04:01:53,235 2023-12-14T04:01:53,236 >>> ultra.name 2023-12-14T04:01:53,236 'psm_0a2713e4' 2023-12-14T04:01:53,237 ``` 2023-12-14T04:01:53,237 In another Python REPL: 2023-12-14T04:01:53,238 ```python 2023-12-14T04:01:53,238 Python 3.9.2 on linux 2023-12-14T04:01:53,238 >>> 2023-12-14T04:01:53,239 >>> from i18n_json import i18n_json 2023-12-14T04:01:53,239 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T04:01:53,240 >>> other['nested']['counter'] += 1 2023-12-14T04:01:53,240 ``` 2023-12-14T04:01:53,241 Back in the first Python REPL: 2023-12-14T04:01:53,241 ```python 2023-12-14T04:01:53,242 >>> ultra['nested']['counter'] 2023-12-14T04:01:53,242 1 2023-12-14T04:01:53,242 ``` 2023-12-14T04:01:53,243 ## Performance comparison 2023-12-14T04:01:53,244 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T04:01:53,245 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T04:01:53,245 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T04:01:53,246 ```python 2023-12-14T04:01:53,246 Python 3.9.2 on linux 2023-12-14T04:01:53,247 >>> 2023-12-14T04:01:53,247 >>> from i18n_json import i18n_json 2023-12-14T04:01:53,248 >>> ultra = i18n_json() 2023-12-14T04:01:53,248 >>> for i in range(10_000): ultra[i] = i 2023-12-14T04:01:53,249 ... 2023-12-14T04:01:53,249 >>> len(ultra) 2023-12-14T04:01:53,249 10000 2023-12-14T04:01:53,250 >>> ultra[500] 2023-12-14T04:01:53,250 500 2023-12-14T04:01:53,251 >>> # Now let's do some performance testing 2023-12-14T04:01:53,251 >>> import multiprocessing, redis, timeit 2023-12-14T04:01:53,252 >>> orig = dict(ultra) 2023-12-14T04:01:53,253 >>> len(orig) 2023-12-14T04:01:53,253 10000 2023-12-14T04:01:53,254 >>> orig[500] 2023-12-14T04:01:53,254 500 2023-12-14T04:01:53,255 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T04:01:53,256 >>> len(managed) 2023-12-14T04:01:53,256 10000 2023-12-14T04:01:53,257 >>> r = redis.Redis() 2023-12-14T04:01:53,257 >>> r.flushall() 2023-12-14T04:01:53,258 >>> r.mset(orig) 2023-12-14T04:01:53,259 ``` 2023-12-14T04:01:53,260 ### Read performance 2023-12-14T04:01:53,261 >>> 2023-12-14T04:01:53,262 ```python 2023-12-14T04:01:53,262 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T04:01:53,263 0.03832335816696286 2023-12-14T04:01:53,264 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T04:01:53,264 0.5248982920311391 2023-12-14T04:01:53,265 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T04:01:53,265 40.85506196087226 2023-12-14T04:01:53,266 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T04:01:53,266 49.3497632863 2023-12-14T04:01:53,267 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T04:01:53,268 0.04309639008715749 2023-12-14T04:01:53,268 ``` 2023-12-14T04:01:53,269 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T04:01:53,270 ### Write performance 2023-12-14T04:01:53,271 ```python 2023-12-14T04:01:53,272 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T04:01:53,273 0.028232071083039045 2023-12-14T04:01:53,273 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T04:01:53,274 2.911152713932097 2023-12-14T04:01:53,274 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T04:01:53,275 31.641707635018975 2023-12-14T04:01:53,276 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T04:01:53,276 124.3432381930761 2023-12-14T04:01:53,277 ``` 2023-12-14T04:01:53,278 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T04:01:53,279 ### Testing performance 2023-12-14T04:01:53,279 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T04:01:53,280 ```bash 2023-12-14T04:01:53,280 python ./tests/performance/performance.py 2023-12-14T04:01:53,281 Testing Performance with 1000000 operations each 2023-12-14T04:01:53,282 Redis (writes) = 24,351 ops per second 2023-12-14T04:01:53,282 Redis (reads) = 30,466 ops per second 2023-12-14T04:01:53,283 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T04:01:53,283 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T04:01:53,283 Python dict (writes) = 16,413,569 ops per second 2023-12-14T04:01:53,284 Python dict (reads) = 16,479,191 ops per second 2023-12-14T04:01:53,284 i18n_json (writes) = 479,860 ops per second 2023-12-14T04:01:53,285 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T04:01:53,285 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T04:01:53,285 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T04:01:53,286 Ranking: 2023-12-14T04:01:53,287 writes: 2023-12-14T04:01:53,287 Python dict = 16,413,569 (factor 1.0) 2023-12-14T04:01:53,288 i18n_json = 479,860 (factor 34.2) 2023-12-14T04:01:53,288 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T04:01:53,288 Redis = 24,351 (factor 674.04) 2023-12-14T04:01:53,289 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T04:01:53,290 reads: 2023-12-14T04:01:53,290 Python dict = 16,479,191 (factor 1.0) 2023-12-14T04:01:53,291 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T04:01:53,292 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T04:01:53,292 Redis = 30,466 (factor 540.9) 2023-12-14T04:01:53,293 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T04:01:53,294 ``` 2023-12-14T04:01:53,295 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T04:01:53,296 ## Parameters 2023-12-14T04:01:53,297 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T04:01:53,298 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T04:01:53,299 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T04:01:53,300 memory space is attached. 2023-12-14T04:01:53,301 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T04:01:53,301 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T04:01:53,302 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T04:01:53,303 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T04:01:53,304 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T04:01:53,304 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T04:01:53,305 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T04:01:53,306 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T04:01:53,307 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T04:01:53,308 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T04:01:53,309 dict will automatically load the full dump and continue streaming updates. 2023-12-14T04:01:53,309 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T04:01:53,310 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T04:01:53,310 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T04:01:53,311 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T04:01:53,312 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T04:01:53,312 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T04:01:53,312 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T04:01:53,313 (Also see the section [Locking](#locking) below!) 2023-12-14T04:01:53,314 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T04:01:53,314 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T04:01:53,314 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T04:01:53,315 be writing to the dict and therefore creating full dumps. 2023-12-14T04:01:53,316 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T04:01:53,316 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T04:01:53,317 dict. 2023-12-14T04:01:53,318 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T04:01:53,319 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T04:01:53,320 ## Memory management 2023-12-14T04:01:53,321 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T04:01:53,322 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T04:01:53,323 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T04:01:53,323 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T04:01:53,324 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T04:01:53,325 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T04:01:53,325 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T04:01:53,326 **Buffer sizes and read performance:** 2023-12-14T04:01:53,327 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T04:01:53,329 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T04:01:53,330 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T04:01:53,331 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T04:01:53,332 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T04:01:53,332 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T04:01:53,333 ## Locking 2023-12-14T04:01:53,334 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T04:01:53,335 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T04:01:53,335 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T04:01:53,337 ### How to use the locking? 2023-12-14T04:01:53,337 ```python 2023-12-14T04:01:53,338 ultra = i18n_json(shared_lock=True) 2023-12-14T04:01:53,339 with ultra.lock: 2023-12-14T04:01:53,340 ultra['counter']++ 2023-12-14T04:01:53,341 # The same as above with all default parameters 2023-12-14T04:01:53,341 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T04:01:53,342 ultra['counter']++ 2023-12-14T04:01:53,343 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T04:01:53,343 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T04:01:53,344 with ultra.lock(sleep_time=0): 2023-12-14T04:01:53,345 ultra['counter']++ 2023-12-14T04:01:53,346 try: 2023-12-14T04:01:53,346 result = ultra.lock.acquire(block=False) 2023-12-14T04:01:53,346 ultra.lock.release() 2023-12-14T04:01:53,347 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T04:01:53,347 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T04:01:53,348 try: 2023-12-14T04:01:53,349 with ultra.lock(timeout=1.5): 2023-12-14T04:01:53,349 ultra['counter']++ 2023-12-14T04:01:53,350 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T04:01:53,350 print('Stale lock?') 2023-12-14T04:01:53,351 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T04:01:53,352 ultra['counter']++ 2023-12-14T04:01:53,353 ``` 2023-12-14T04:01:53,354 ## Explicit cleanup 2023-12-14T04:01:53,355 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T04:01:53,356 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T04:01:53,357 Another way to do this in code is like this: 2023-12-14T04:01:53,358 ```python 2023-12-14T04:01:53,359 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T04:01:53,359 name = 'my-dict-name' 2023-12-14T04:01:53,360 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T04:01:53,360 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T04:01:53,360 ``` 2023-12-14T04:01:53,361 ## Advanced usage 2023-12-14T04:01:53,362 See [examples](/examples) folder 2023-12-14T04:01:53,362 ```python 2023-12-14T04:01:53,363 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T04:01:53,363 >>> # Let's use a value with 100k bytes length. 2023-12-14T04:01:53,364 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T04:01:53,364 >>> ultra[0] = ' ' * 100_000 2023-12-14T04:01:53,364 >>> ultra.print_status() 2023-12-14T04:01:53,365 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T04:01:53,365 'buffer_size': 100000, 2023-12-14T04:01:53,365 'control': SharedMemory('my-name', size=1000), 2023-12-14T04:01:53,366 'full_dump_counter': 1, 2023-12-14T04:01:53,366 'full_dump_counter_remote': 1, 2023-12-14T04:01:53,366 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T04:01:53,367 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T04:01:53,367 'full_dump_size': None, 2023-12-14T04:01:53,368 'full_dump_static_size_remote': , 2023-12-14T04:01:53,368 'lock': , 2023-12-14T04:01:53,368 'lock_pid_remote': 0, 2023-12-14T04:01:53,369 'lock_remote': 0, 2023-12-14T04:01:53,370 'name': 'my-name', 2023-12-14T04:01:53,370 'recurse': False, 2023-12-14T04:01:53,371 'recurse_remote': , 2023-12-14T04:01:53,371 'serializer': , 2023-12-14T04:01:53,372 'shared_lock_remote': , 2023-12-14T04:01:53,372 'update_stream_position': 0, 2023-12-14T04:01:53,373 'update_stream_position_remote': 0} 2023-12-14T04:01:53,374 ``` 2023-12-14T04:01:53,375 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T04:01:53,376 Other things you can do: 2023-12-14T04:01:53,376 ```python 2023-12-14T04:01:53,377 >>> # Create a full dump 2023-12-14T04:01:53,378 >>> ultra.dump() 2023-12-14T04:01:53,379 >>> # Load latest full dump if one is available 2023-12-14T04:01:53,379 >>> ultra.load() 2023-12-14T04:01:53,380 >>> # Show statistics 2023-12-14T04:01:53,381 >>> ultra.print_status() 2023-12-14T04:01:53,382 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T04:01:53,383 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T04:01:53,383 >>> ultra.load(force=True) 2023-12-14T04:01:53,384 >>> # Apply full dump and stream updates to 2023-12-14T04:01:53,384 >>> # underlying local dict, this is automatically 2023-12-14T04:01:53,385 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T04:01:53,385 >>> # but can be useful to call after a forced load. 2023-12-14T04:01:53,386 >>> ultra.apply_update() 2023-12-14T04:01:53,387 >>> # Access underlying local dict directly for maximum performance 2023-12-14T04:01:53,387 >>> ultra.data 2023-12-14T04:01:53,388 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T04:01:53,389 >>> import jsons 2023-12-14T04:01:53,389 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T04:01:53,391 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T04:01:53,391 >>> ultra.close() 2023-12-14T04:01:53,392 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T04:01:53,393 >>> ultra.unlink() 2023-12-14T04:01:53,394 ``` 2023-12-14T04:01:53,395 ## Contributing 2023-12-14T04:01:53,397 Contributions are always welcome! 2023-12-14T04:01:53,750 running dist_info 2023-12-14T04:01:53,770 creating /tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info 2023-12-14T04:01:53,776 writing /tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/PKG-INFO 2023-12-14T04:01:53,780 writing dependency_links to /tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/dependency_links.txt 2023-12-14T04:01:53,782 writing top-level names to /tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/top_level.txt 2023-12-14T04:01:53,783 writing manifest file '/tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/SOURCES.txt' 2023-12-14T04:01:53,811 reading manifest file '/tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/SOURCES.txt' 2023-12-14T04:01:53,813 reading manifest template 'MANIFEST.in' 2023-12-14T04:01:53,814 adding license file 'LICENSE' 2023-12-14T04:01:53,816 writing manifest file '/tmp/pip-modern-metadata-b8n_lufg/i18n_json.egg-info/SOURCES.txt' 2023-12-14T04:01:53,817 creating '/tmp/pip-modern-metadata-b8n_lufg/i18n_json-0.0.9.dist-info' 2023-12-14T04:01:54,039 Preparing metadata (pyproject.toml): finished with status 'done' 2023-12-14T04:01:54,045 Source in /tmp/pip-wheel-vh21srzp/i18n-json_dc72aa8344e249bb96ceed55eb059b5a has version 0.0.9, which satisfies requirement i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz 2023-12-14T04:01:54,046 Removed i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz from build tracker '/tmp/pip-build-tracker-bu_lv4r9' 2023-12-14T04:01:54,052 Created temporary directory: /tmp/pip-unpack-rgjbfgaz 2023-12-14T04:01:54,053 Created temporary directory: /tmp/pip-unpack-cnhn7yal 2023-12-14T04:01:54,057 Building wheels for collected packages: i18n-json 2023-12-14T04:01:54,061 Created temporary directory: /tmp/pip-wheel-rfmx8pz_ 2023-12-14T04:01:54,062 Destination directory: /tmp/pip-wheel-rfmx8pz_ 2023-12-14T04:01:54,064 Building wheel for i18n-json (pyproject.toml): started 2023-12-14T04:01:54,065 Running command Building wheel for i18n-json (pyproject.toml) 2023-12-14T04:01:54,644 # i18n_json 2023-12-14T04:01:54,645 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T04:01:54,646 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T04:01:54,648 Features: 2023-12-14T04:01:54,648 * Fast (compared to other sharing solutions) 2023-12-14T04:01:54,649 * No running manager processes 2023-12-14T04:01:54,650 * Works in spawn and fork context 2023-12-14T04:01:54,650 * Safe locking between independent processes 2023-12-14T04:01:54,651 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T04:01:54,651 * Convenient, no setter or getters necessary 2023-12-14T04:01:54,652 * Optional recursion for nested dicts 2023-12-14T04:01:54,653 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T04:01:54,653 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T04:01:54,654 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T04:01:54,655 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T04:01:54,656 ## General Concept 2023-12-14T04:01:54,657 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T04:01:54,658 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T04:01:54,660 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T04:01:54,660 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T04:01:54,661 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T04:01:54,661 streaming updates afterwards. 2023-12-14T04:01:54,662 ## Issues 2023-12-14T04:01:54,662 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T04:01:54,663 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T04:01:54,663 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T04:01:54,663 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T04:01:54,664 ## Alternatives 2023-12-14T04:01:54,665 There are many alternatives: 2023-12-14T04:01:54,666 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T04:01:54,666 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T04:01:54,666 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T04:01:54,667 * Redis 2023-12-14T04:01:54,667 * Memcached 2023-12-14T04:01:54,668 ## How to use? 2023-12-14T04:01:54,669 ### Simple 2023-12-14T04:01:54,669 In one Python REPL: 2023-12-14T04:01:54,670 ```python 2023-12-14T04:01:54,670 Python 3.9.2 on linux 2023-12-14T04:01:54,671 >>> 2023-12-14T04:01:54,671 >>> from i18n_json import i18n_json 2023-12-14T04:01:54,672 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T04:01:54,673 >>> ultra 2023-12-14T04:01:54,673 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:54,674 >>> 2023-12-14T04:01:54,674 >>> # We need the shared memory name in the other process. 2023-12-14T04:01:54,675 >>> ultra.name 2023-12-14T04:01:54,675 'psm_ad73da69' 2023-12-14T04:01:54,676 ``` 2023-12-14T04:01:54,677 In another Python REPL: 2023-12-14T04:01:54,677 ```python 2023-12-14T04:01:54,678 Python 3.9.2 on linux 2023-12-14T04:01:54,679 >>> 2023-12-14T04:01:54,679 >>> from i18n_json import i18n_json 2023-12-14T04:01:54,680 >>> # Connect to the shared memory with the name above 2023-12-14T04:01:54,680 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T04:01:54,681 >>> other 2023-12-14T04:01:54,682 {1: 1, 'some_key': 'some_value'} 2023-12-14T04:01:54,682 >>> other[2] = 2 2023-12-14T04:01:54,683 ``` 2023-12-14T04:01:54,684 Back in the first Python REPL: 2023-12-14T04:01:54,684 ```python 2023-12-14T04:01:54,685 >>> ultra[2] 2023-12-14T04:01:54,685 2 2023-12-14T04:01:54,685 ``` 2023-12-14T04:01:54,686 ### Nested 2023-12-14T04:01:54,687 In one Python REPL: 2023-12-14T04:01:54,688 ```python 2023-12-14T04:01:54,688 Python 3.9.2 on linux 2023-12-14T04:01:54,689 >>> 2023-12-14T04:01:54,689 >>> from i18n_json import i18n_json 2023-12-14T04:01:54,690 >>> ultra = i18n_json(recurse=True) 2023-12-14T04:01:54,690 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T04:01:54,691 >>> type(ultra['nested']) 2023-12-14T04:01:54,691 2023-12-14T04:01:54,692 >>> ultra.name 2023-12-14T04:01:54,692 'psm_0a2713e4' 2023-12-14T04:01:54,693 ``` 2023-12-14T04:01:54,694 In another Python REPL: 2023-12-14T04:01:54,695 ```python 2023-12-14T04:01:54,695 Python 3.9.2 on linux 2023-12-14T04:01:54,696 >>> 2023-12-14T04:01:54,696 >>> from i18n_json import i18n_json 2023-12-14T04:01:54,697 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T04:01:54,697 >>> other['nested']['counter'] += 1 2023-12-14T04:01:54,698 ``` 2023-12-14T04:01:54,699 Back in the first Python REPL: 2023-12-14T04:01:54,700 ```python 2023-12-14T04:01:54,700 >>> ultra['nested']['counter'] 2023-12-14T04:01:54,701 1 2023-12-14T04:01:54,702 ``` 2023-12-14T04:01:54,702 ## Performance comparison 2023-12-14T04:01:54,703 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T04:01:54,704 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T04:01:54,704 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T04:01:54,705 ```python 2023-12-14T04:01:54,705 Python 3.9.2 on linux 2023-12-14T04:01:54,706 >>> 2023-12-14T04:01:54,706 >>> from i18n_json import i18n_json 2023-12-14T04:01:54,706 >>> ultra = i18n_json() 2023-12-14T04:01:54,707 >>> for i in range(10_000): ultra[i] = i 2023-12-14T04:01:54,707 ... 2023-12-14T04:01:54,707 >>> len(ultra) 2023-12-14T04:01:54,708 10000 2023-12-14T04:01:54,708 >>> ultra[500] 2023-12-14T04:01:54,708 500 2023-12-14T04:01:54,709 >>> # Now let's do some performance testing 2023-12-14T04:01:54,709 >>> import multiprocessing, redis, timeit 2023-12-14T04:01:54,710 >>> orig = dict(ultra) 2023-12-14T04:01:54,710 >>> len(orig) 2023-12-14T04:01:54,710 10000 2023-12-14T04:01:54,711 >>> orig[500] 2023-12-14T04:01:54,711 500 2023-12-14T04:01:54,711 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T04:01:54,712 >>> len(managed) 2023-12-14T04:01:54,712 10000 2023-12-14T04:01:54,713 >>> r = redis.Redis() 2023-12-14T04:01:54,713 >>> r.flushall() 2023-12-14T04:01:54,713 >>> r.mset(orig) 2023-12-14T04:01:54,714 ``` 2023-12-14T04:01:54,715 ### Read performance 2023-12-14T04:01:54,716 >>> 2023-12-14T04:01:54,716 ```python 2023-12-14T04:01:54,717 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T04:01:54,717 0.03832335816696286 2023-12-14T04:01:54,718 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T04:01:54,718 0.5248982920311391 2023-12-14T04:01:54,719 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T04:01:54,719 40.85506196087226 2023-12-14T04:01:54,720 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T04:01:54,720 49.3497632863 2023-12-14T04:01:54,721 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T04:01:54,721 0.04309639008715749 2023-12-14T04:01:54,722 ``` 2023-12-14T04:01:54,723 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T04:01:54,724 ### Write performance 2023-12-14T04:01:54,725 ```python 2023-12-14T04:01:54,725 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T04:01:54,726 0.028232071083039045 2023-12-14T04:01:54,726 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T04:01:54,727 2.911152713932097 2023-12-14T04:01:54,727 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T04:01:54,728 31.641707635018975 2023-12-14T04:01:54,728 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T04:01:54,729 124.3432381930761 2023-12-14T04:01:54,729 ``` 2023-12-14T04:01:54,730 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T04:01:54,731 ### Testing performance 2023-12-14T04:01:54,733 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T04:01:54,734 ```bash 2023-12-14T04:01:54,734 python ./tests/performance/performance.py 2023-12-14T04:01:54,735 Testing Performance with 1000000 operations each 2023-12-14T04:01:54,737 Redis (writes) = 24,351 ops per second 2023-12-14T04:01:54,737 Redis (reads) = 30,466 ops per second 2023-12-14T04:01:54,738 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T04:01:54,739 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T04:01:54,739 Python dict (writes) = 16,413,569 ops per second 2023-12-14T04:01:54,739 Python dict (reads) = 16,479,191 ops per second 2023-12-14T04:01:54,740 i18n_json (writes) = 479,860 ops per second 2023-12-14T04:01:54,740 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T04:01:54,741 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T04:01:54,741 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T04:01:54,742 Ranking: 2023-12-14T04:01:54,742 writes: 2023-12-14T04:01:54,742 Python dict = 16,413,569 (factor 1.0) 2023-12-14T04:01:54,743 i18n_json = 479,860 (factor 34.2) 2023-12-14T04:01:54,743 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T04:01:54,743 Redis = 24,351 (factor 674.04) 2023-12-14T04:01:54,744 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T04:01:54,744 reads: 2023-12-14T04:01:54,745 Python dict = 16,479,191 (factor 1.0) 2023-12-14T04:01:54,745 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T04:01:54,745 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T04:01:54,746 Redis = 30,466 (factor 540.9) 2023-12-14T04:01:54,746 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T04:01:54,746 ``` 2023-12-14T04:01:54,747 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T04:01:54,748 ## Parameters 2023-12-14T04:01:54,749 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T04:01:54,750 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T04:01:54,751 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T04:01:54,751 memory space is attached. 2023-12-14T04:01:54,753 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T04:01:54,753 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T04:01:54,754 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T04:01:54,755 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T04:01:54,756 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T04:01:54,756 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T04:01:54,757 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T04:01:54,757 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T04:01:54,758 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T04:01:54,759 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T04:01:54,759 dict will automatically load the full dump and continue streaming updates. 2023-12-14T04:01:54,760 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T04:01:54,761 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T04:01:54,762 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T04:01:54,763 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T04:01:54,764 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T04:01:54,764 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T04:01:54,765 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T04:01:54,766 (Also see the section [Locking](#locking) below!) 2023-12-14T04:01:54,767 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T04:01:54,767 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T04:01:54,768 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T04:01:54,768 be writing to the dict and therefore creating full dumps. 2023-12-14T04:01:54,769 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T04:01:54,769 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T04:01:54,769 dict. 2023-12-14T04:01:54,770 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T04:01:54,771 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T04:01:54,772 ## Memory management 2023-12-14T04:01:54,773 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T04:01:54,774 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T04:01:54,774 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T04:01:54,775 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T04:01:54,776 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T04:01:54,777 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T04:01:54,777 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T04:01:54,778 **Buffer sizes and read performance:** 2023-12-14T04:01:54,779 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T04:01:54,780 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T04:01:54,781 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T04:01:54,782 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T04:01:54,784 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T04:01:54,785 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T04:01:54,786 ## Locking 2023-12-14T04:01:54,787 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T04:01:54,787 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T04:01:54,788 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T04:01:54,789 ### How to use the locking? 2023-12-14T04:01:54,789 ```python 2023-12-14T04:01:54,790 ultra = i18n_json(shared_lock=True) 2023-12-14T04:01:54,790 with ultra.lock: 2023-12-14T04:01:54,791 ultra['counter']++ 2023-12-14T04:01:54,791 # The same as above with all default parameters 2023-12-14T04:01:54,792 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T04:01:54,793 ultra['counter']++ 2023-12-14T04:01:54,794 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T04:01:54,794 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T04:01:54,795 with ultra.lock(sleep_time=0): 2023-12-14T04:01:54,795 ultra['counter']++ 2023-12-14T04:01:54,796 try: 2023-12-14T04:01:54,797 result = ultra.lock.acquire(block=False) 2023-12-14T04:01:54,797 ultra.lock.release() 2023-12-14T04:01:54,798 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T04:01:54,799 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T04:01:54,800 try: 2023-12-14T04:01:54,800 with ultra.lock(timeout=1.5): 2023-12-14T04:01:54,801 ultra['counter']++ 2023-12-14T04:01:54,801 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T04:01:54,802 print('Stale lock?') 2023-12-14T04:01:54,803 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T04:01:54,804 ultra['counter']++ 2023-12-14T04:01:54,805 ``` 2023-12-14T04:01:54,805 ## Explicit cleanup 2023-12-14T04:01:54,806 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T04:01:54,807 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T04:01:54,808 Another way to do this in code is like this: 2023-12-14T04:01:54,809 ```python 2023-12-14T04:01:54,809 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T04:01:54,810 name = 'my-dict-name' 2023-12-14T04:01:54,810 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T04:01:54,811 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T04:01:54,812 ``` 2023-12-14T04:01:54,813 ## Advanced usage 2023-12-14T04:01:54,814 See [examples](/examples) folder 2023-12-14T04:01:54,815 ```python 2023-12-14T04:01:54,815 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T04:01:54,816 >>> # Let's use a value with 100k bytes length. 2023-12-14T04:01:54,817 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T04:01:54,818 >>> ultra[0] = ' ' * 100_000 2023-12-14T04:01:54,818 >>> ultra.print_status() 2023-12-14T04:01:54,818 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T04:01:54,819 'buffer_size': 100000, 2023-12-14T04:01:54,819 'control': SharedMemory('my-name', size=1000), 2023-12-14T04:01:54,819 'full_dump_counter': 1, 2023-12-14T04:01:54,820 'full_dump_counter_remote': 1, 2023-12-14T04:01:54,820 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T04:01:54,820 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T04:01:54,821 'full_dump_size': None, 2023-12-14T04:01:54,821 'full_dump_static_size_remote': , 2023-12-14T04:01:54,821 'lock': , 2023-12-14T04:01:54,822 'lock_pid_remote': 0, 2023-12-14T04:01:54,822 'lock_remote': 0, 2023-12-14T04:01:54,822 'name': 'my-name', 2023-12-14T04:01:54,823 'recurse': False, 2023-12-14T04:01:54,823 'recurse_remote': , 2023-12-14T04:01:54,823 'serializer': , 2023-12-14T04:01:54,824 'shared_lock_remote': , 2023-12-14T04:01:54,824 'update_stream_position': 0, 2023-12-14T04:01:54,825 'update_stream_position_remote': 0} 2023-12-14T04:01:54,825 ``` 2023-12-14T04:01:54,826 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T04:01:54,827 Other things you can do: 2023-12-14T04:01:54,827 ```python 2023-12-14T04:01:54,827 >>> # Create a full dump 2023-12-14T04:01:54,828 >>> ultra.dump() 2023-12-14T04:01:54,829 >>> # Load latest full dump if one is available 2023-12-14T04:01:54,829 >>> ultra.load() 2023-12-14T04:01:54,830 >>> # Show statistics 2023-12-14T04:01:54,831 >>> ultra.print_status() 2023-12-14T04:01:54,832 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T04:01:54,833 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T04:01:54,833 >>> ultra.load(force=True) 2023-12-14T04:01:54,834 >>> # Apply full dump and stream updates to 2023-12-14T04:01:54,835 >>> # underlying local dict, this is automatically 2023-12-14T04:01:54,835 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T04:01:54,836 >>> # but can be useful to call after a forced load. 2023-12-14T04:01:54,837 >>> ultra.apply_update() 2023-12-14T04:01:54,838 >>> # Access underlying local dict directly for maximum performance 2023-12-14T04:01:54,838 >>> ultra.data 2023-12-14T04:01:54,840 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T04:01:54,840 >>> import jsons 2023-12-14T04:01:54,840 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T04:01:54,842 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T04:01:54,842 >>> ultra.close() 2023-12-14T04:01:54,843 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T04:01:54,844 >>> ultra.unlink() 2023-12-14T04:01:54,845 ``` 2023-12-14T04:01:54,846 ## Contributing 2023-12-14T04:01:54,847 Contributions are always welcome! 2023-12-14T04:01:55,283 running bdist_wheel 2023-12-14T04:01:55,326 running build 2023-12-14T04:01:55,327 running build_py 2023-12-14T04:01:55,337 creating build 2023-12-14T04:01:55,337 creating build/lib.linux-armv7l-cpython-311 2023-12-14T04:01:55,338 creating build/lib.linux-armv7l-cpython-311/i18n_json 2023-12-14T04:01:55,339 copying ./__init__.py -> build/lib.linux-armv7l-cpython-311/i18n_json 2023-12-14T04:01:55,341 copying ./Exceptions.py -> build/lib.linux-armv7l-cpython-311/i18n_json 2023-12-14T04:01:55,343 copying ./i18n_json.py -> build/lib.linux-armv7l-cpython-311/i18n_json 2023-12-14T04:01:55,346 copying ./setup.py -> build/lib.linux-armv7l-cpython-311/i18n_json 2023-12-14T04:01:55,347 running egg_info 2023-12-14T04:01:55,353 writing i18n_json.egg-info/PKG-INFO 2023-12-14T04:01:55,357 writing dependency_links to i18n_json.egg-info/dependency_links.txt 2023-12-14T04:01:55,359 writing top-level names to i18n_json.egg-info/top_level.txt 2023-12-14T04:01:55,372 reading manifest file 'i18n_json.egg-info/SOURCES.txt' 2023-12-14T04:01:55,374 reading manifest template 'MANIFEST.in' 2023-12-14T04:01:55,375 adding license file 'LICENSE' 2023-12-14T04:01:55,377 writing manifest file 'i18n_json.egg-info/SOURCES.txt' 2023-12-14T04:01:55,379 running build_ext 2023-12-14T04:01:55,386 building 'i18n_json' extension 2023-12-14T04:01:55,387 creating build/temp.linux-armv7l-cpython-311 2023-12-14T04:01:55,387 arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.11 -c i18n_json.c -o build/temp.linux-armv7l-cpython-311/i18n_json.o 2023-12-14T04:03:12,918 arm-linux-gnueabihf-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-armv7l-cpython-311/i18n_json.o -L/usr/lib/arm-linux-gnueabihf -o build/lib.linux-armv7l-cpython-311/i18n_json.cpython-311-arm-linux-gnueabihf.so 2023-12-14T04:03:13,112 installing to build/bdist.linux-armv7l/wheel 2023-12-14T04:03:13,112 running install 2023-12-14T04:03:13,136 running install_lib 2023-12-14T04:03:13,145 creating build/bdist.linux-armv7l 2023-12-14T04:03:13,145 creating build/bdist.linux-armv7l/wheel 2023-12-14T04:03:13,147 creating build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T04:03:13,148 copying build/lib.linux-armv7l-cpython-311/i18n_json/__init__.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T04:03:13,150 copying build/lib.linux-armv7l-cpython-311/i18n_json/Exceptions.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T04:03:13,152 copying build/lib.linux-armv7l-cpython-311/i18n_json/i18n_json.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T04:03:13,155 copying build/lib.linux-armv7l-cpython-311/i18n_json/setup.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T04:03:13,157 copying build/lib.linux-armv7l-cpython-311/i18n_json.cpython-311-arm-linux-gnueabihf.so -> build/bdist.linux-armv7l/wheel 2023-12-14T04:03:13,193 running install_egg_info 2023-12-14T04:03:13,206 Copying i18n_json.egg-info to build/bdist.linux-armv7l/wheel/i18n_json-0.0.9-py3.11.egg-info 2023-12-14T04:03:13,217 running install_scripts 2023-12-14T04:03:13,263 creating build/bdist.linux-armv7l/wheel/i18n_json-0.0.9.dist-info/WHEEL 2023-12-14T04:03:13,266 creating '/tmp/pip-wheel-rfmx8pz_/.tmp-3umnsc24/i18n_json-0.0.9-cp311-cp311-linux_armv7l.whl' and adding 'build/bdist.linux-armv7l/wheel' to it 2023-12-14T04:03:13,660 adding 'i18n_json.cpython-311-arm-linux-gnueabihf.so' 2023-12-14T04:03:13,673 adding 'i18n_json/Exceptions.py' 2023-12-14T04:03:13,674 adding 'i18n_json/__init__.py' 2023-12-14T04:03:13,680 adding 'i18n_json/i18n_json.py' 2023-12-14T04:03:13,681 adding 'i18n_json/setup.py' 2023-12-14T04:03:13,684 adding 'i18n_json-0.0.9.dist-info/LICENSE' 2023-12-14T04:03:13,687 adding 'i18n_json-0.0.9.dist-info/METADATA' 2023-12-14T04:03:13,688 adding 'i18n_json-0.0.9.dist-info/WHEEL' 2023-12-14T04:03:13,689 adding 'i18n_json-0.0.9.dist-info/top_level.txt' 2023-12-14T04:03:13,690 adding 'i18n_json-0.0.9.dist-info/RECORD' 2023-12-14T04:03:13,700 removing build/bdist.linux-armv7l/wheel 2023-12-14T04:03:13,841 Building wheel for i18n-json (pyproject.toml): finished with status 'done' 2023-12-14T04:03:13,864 Created wheel for i18n-json: filename=i18n_json-0.0.9-cp311-cp311-linux_armv7l.whl size=817643 sha256=38773b051f5f0040648f0726fb95c5e2c4f12c6fe7eca9f2deb0bc854222b0bc 2023-12-14T04:03:13,865 Stored in directory: /tmp/pip-ephem-wheel-cache-4ou8a3k3/wheels/d5/8a/f0/fa4e81216345214855471034c5d7bd011bdc8c36de70a73203 2023-12-14T04:03:13,878 Successfully built i18n-json 2023-12-14T04:03:13,904 Removed build tracker: '/tmp/pip-build-tracker-bu_lv4r9'