2023-12-14T06:30:25,934 Created temporary directory: /tmp/pip-build-tracker-wm5p1b0q 2023-12-14T06:30:25,935 Initialized build tracking at /tmp/pip-build-tracker-wm5p1b0q 2023-12-14T06:30:25,936 Created build tracker: /tmp/pip-build-tracker-wm5p1b0q 2023-12-14T06:30:25,936 Entered build tracker: /tmp/pip-build-tracker-wm5p1b0q 2023-12-14T06:30:25,936 Created temporary directory: /tmp/pip-wheel-3_nzz3dz 2023-12-14T06:30:25,941 Created temporary directory: /tmp/pip-ephem-wheel-cache-et6pc6z5 2023-12-14T06:30:25,965 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2023-12-14T06:30:25,968 2 location(s) to search for versions of i18n-json: 2023-12-14T06:30:25,968 * https://pypi.org/simple/i18n-json/ 2023-12-14T06:30:25,968 * https://www.piwheels.org/simple/i18n-json/ 2023-12-14T06:30:25,969 Fetching project page and analyzing links: https://pypi.org/simple/i18n-json/ 2023-12-14T06:30:25,969 Getting page https://pypi.org/simple/i18n-json/ 2023-12-14T06:30:25,971 Found index url https://pypi.org/simple/ 2023-12-14T06:30:26,022 Fetched page https://pypi.org/simple/i18n-json/ as application/vnd.pypi.simple.v1+json 2023-12-14T06:30:26,024 Skipping link: No binaries permitted for i18n-json: https://files.pythonhosted.org/packages/94/7e/e050c95102a9db3cb1efe7b0b2e821163c801569aa4040eea50f47167a85/i18n_json-0.0.8-cp310-cp310-win_amd64.whl (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T06:30:26,025 Found link https://files.pythonhosted.org/packages/52/f3/b4ee6454f577112fb5ce9650efa1a9358a204b4eb243bd1b22c9730ebec4/i18n_json-0.0.8.tar.gz (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8), version: 0.0.8 2023-12-14T06:30:26,025 Skipping link: No binaries permitted for i18n-json: https://files.pythonhosted.org/packages/d7/e2/f2e1e5e88725bea9939fbced144bee10fac4387c2c163be39d05a6ab955e/i18n_json-0.0.9-cp310-cp310-win_amd64.whl (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T06:30:26,025 Found link https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz (from https://pypi.org/simple/i18n-json/) (requires-python:>=3.8), version: 0.0.9 2023-12-14T06:30:26,026 Fetching project page and analyzing links: https://www.piwheels.org/simple/i18n-json/ 2023-12-14T06:30:26,026 Getting page https://www.piwheels.org/simple/i18n-json/ 2023-12-14T06:30:26,027 Found index url https://www.piwheels.org/simple/ 2023-12-14T06:30:26,093 Fetched page https://www.piwheels.org/simple/i18n-json/ as text/html 2023-12-14T06:30:26,094 Skipping link: No binaries permitted for i18n-json: https://www.piwheels.org/simple/i18n-json/i18n_json-0.0.9-cp311-cp311-linux_armv6l.whl#sha256=38773b051f5f0040648f0726fb95c5e2c4f12c6fe7eca9f2deb0bc854222b0bc (from https://www.piwheels.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T06:30:26,094 Skipping link: No binaries permitted for i18n-json: https://www.piwheels.org/simple/i18n-json/i18n_json-0.0.9-cp311-cp311-linux_armv7l.whl#sha256=38773b051f5f0040648f0726fb95c5e2c4f12c6fe7eca9f2deb0bc854222b0bc (from https://www.piwheels.org/simple/i18n-json/) (requires-python:>=3.8) 2023-12-14T06:30:26,095 Skipping link: not a file: https://www.piwheels.org/simple/i18n-json/ 2023-12-14T06:30:26,095 Skipping link: not a file: https://pypi.org/simple/i18n-json/ 2023-12-14T06:30:26,112 Given no hashes to check 1 links for project 'i18n-json': discarding no candidates 2023-12-14T06:30:26,128 Collecting i18n-json==0.0.9 2023-12-14T06:30:26,131 Created temporary directory: /tmp/pip-unpack-n4v65qu3 2023-12-14T06:30:26,173 Downloading i18n_json-0.0.9.tar.gz (219 kB) 2023-12-14T06:30:26,299 Added i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz to build tracker '/tmp/pip-build-tracker-wm5p1b0q' 2023-12-14T06:30:26,305 Created temporary directory: /tmp/pip-build-env-5kkr3kh9 2023-12-14T06:30:26,315 Installing build dependencies: started 2023-12-14T06:30:26,316 Running command pip subprocess to install build dependencies 2023-12-14T06:30:27,472 Using pip 23.3.1 from /home/piwheels/.local/lib/python3.9/site-packages/pip (python 3.9) 2023-12-14T06:30:28,010 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2023-12-14T06:30:29,430 Collecting setuptools 2023-12-14T06:30:29,447 Using cached https://www.piwheels.org/simple/setuptools/setuptools-69.0.2-py3-none-any.whl (819 kB) 2023-12-14T06:30:29,690 Collecting wheel 2023-12-14T06:30:29,706 Using cached https://www.piwheels.org/simple/wheel/wheel-0.42.0-py3-none-any.whl (65 kB) 2023-12-14T06:30:30,912 Collecting cython 2023-12-14T06:30:30,929 Using cached https://www.piwheels.org/simple/cython/Cython-3.0.6-cp39-cp39-linux_armv7l.whl (10.7 MB) 2023-12-14T06:30:31,461 Link requires a different Python (3.9.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://files.pythonhosted.org/packages/36/3b/fa4025a424adafd85c6195001b1c130ecb8d8b30784a1c4cb68e7b5e5ae7/pylint-1.9.5-py2.py3-none-any.whl (from https://pypi.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T06:30:31,463 Link requires a different Python (3.9.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://files.pythonhosted.org/packages/3f/0b/4e7eeab1abf594b447385a340593c1a4244cdf8e54a78edcae1e2756d6fb/pylint-1.9.5.tar.gz (from https://pypi.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T06:30:31,657 Link requires a different Python (3.9.2 not in: '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7'): https://www.piwheels.org/simple/pylint/pylint-1.9.5-py2.py3-none-any.whl#sha256=367e3d49813d349a905390ac27989eff82ab84958731c5ef0bef867452cfdc42 (from https://www.piwheels.org/simple/pylint/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7) 2023-12-14T06:30:31,768 Collecting pylint 2023-12-14T06:30:31,779 Downloading https://www.piwheels.org/simple/pylint/pylint-3.0.3-py3-none-any.whl (510 kB) 2023-12-14T06:30:31,838 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 510.6/510.6 kB 10.5 MB/s eta 0:00:00 2023-12-14T06:30:32,572 Collecting psutil 2023-12-14T06:30:32,594 Using cached https://www.piwheels.org/simple/psutil/psutil-5.9.6-cp39-abi3-linux_armv7l.whl (278 kB) 2023-12-14T06:30:33,058 Collecting platformdirs>=2.2.0 (from pylint) 2023-12-14T06:30:33,076 Using cached https://www.piwheels.org/simple/platformdirs/platformdirs-4.1.0-py3-none-any.whl (17 kB) 2023-12-14T06:30:33,451 Collecting astroid<=3.1.0-dev0,>=3.0.1 (from pylint) 2023-12-14T06:30:33,462 Downloading https://www.piwheels.org/simple/astroid/astroid-3.0.2-py3-none-any.whl (275 kB) 2023-12-14T06:30:33,503 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 275.2/275.2 kB 8.6 MB/s eta 0:00:00 2023-12-14T06:30:33,952 Collecting isort!=5.13.0,<6,>=4.2.5 (from pylint) 2023-12-14T06:30:33,954 Obtaining dependency information for isort!=5.13.0,<6,>=4.2.5 from https://files.pythonhosted.org/packages/d1/b3/8def84f539e7d2289a02f0524b944b15d7c75dab7628bedf1c4f0992029c/isort-5.13.2-py3-none-any.whl.metadata 2023-12-14T06:30:33,998 Downloading isort-5.13.2-py3-none-any.whl.metadata (12 kB) 2023-12-14T06:30:34,095 Collecting mccabe<0.8,>=0.6 (from pylint) 2023-12-14T06:30:34,115 Using cached https://www.piwheels.org/simple/mccabe/mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB) 2023-12-14T06:30:34,328 Collecting tomlkit>=0.10.1 (from pylint) 2023-12-14T06:30:34,344 Using cached https://www.piwheels.org/simple/tomlkit/tomlkit-0.12.3-py3-none-any.whl (37 kB) 2023-12-14T06:30:34,481 Collecting typing-extensions>=3.10.0 (from pylint) 2023-12-14T06:30:34,497 Using cached https://www.piwheels.org/simple/typing-extensions/typing_extensions-4.9.0-py3-none-any.whl (32 kB) 2023-12-14T06:30:34,606 Collecting dill>=0.2 (from pylint) 2023-12-14T06:30:34,618 Downloading https://www.piwheels.org/simple/dill/dill-0.3.7-py3-none-any.whl (115 kB) 2023-12-14T06:30:34,643 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 6.8 MB/s eta 0:00:00 2023-12-14T06:30:34,757 Collecting tomli>=1.1.0 (from pylint) 2023-12-14T06:30:34,773 Using cached https://www.piwheels.org/simple/tomli/tomli-2.0.1-py3-none-any.whl (12 kB) 2023-12-14T06:30:34,991 Downloading isort-5.13.2-py3-none-any.whl (92 kB) 2023-12-14T06:30:35,016 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.3/92.3 kB 5.4 MB/s eta 0:00:00 2023-12-14T06:30:36,759 Installing collected packages: wheel, typing-extensions, tomlkit, tomli, setuptools, psutil, platformdirs, mccabe, isort, dill, cython, astroid, pylint 2023-12-14T06:30:36,978 Creating /tmp/pip-build-env-5kkr3kh9/overlay/bin 2023-12-14T06:30:36,980 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/wheel to 755 2023-12-14T06:30:40,689 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/isort to 755 2023-12-14T06:30:40,691 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/isort-identify-imports to 755 2023-12-14T06:30:44,570 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/cygdb to 755 2023-12-14T06:30:44,573 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/cython to 755 2023-12-14T06:30:44,575 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/cythonize to 755 2023-12-14T06:30:47,037 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/pylint to 755 2023-12-14T06:30:47,040 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/pylint-config to 755 2023-12-14T06:30:47,042 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/pyreverse to 755 2023-12-14T06:30:47,045 changing mode of /tmp/pip-build-env-5kkr3kh9/overlay/bin/symilar to 755 2023-12-14T06:30:47,079 Successfully installed astroid-3.0.2 cython-3.0.6 dill-0.3.7 isort-5.13.2 mccabe-0.7.0 platformdirs-4.1.0 psutil-5.9.6 pylint-3.0.3 setuptools-69.0.2 tomli-2.0.1 tomlkit-0.12.3 typing-extensions-4.9.0 wheel-0.42.0 2023-12-14T06:30:47,726 Installing build dependencies: finished with status 'done' 2023-12-14T06:30:47,731 Getting requirements to build wheel: started 2023-12-14T06:30:47,732 Running command Getting requirements to build wheel 2023-12-14T06:30:48,373 # i18n_json 2023-12-14T06:30:48,373 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T06:30:48,374 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T06:30:48,374 Features: 2023-12-14T06:30:48,374 * Fast (compared to other sharing solutions) 2023-12-14T06:30:48,374 * No running manager processes 2023-12-14T06:30:48,374 * Works in spawn and fork context 2023-12-14T06:30:48,374 * Safe locking between independent processes 2023-12-14T06:30:48,375 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T06:30:48,375 * Convenient, no setter or getters necessary 2023-12-14T06:30:48,375 * Optional recursion for nested dicts 2023-12-14T06:30:48,375 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T06:30:48,375 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T06:30:48,375 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T06:30:48,376 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T06:30:48,376 ## General Concept 2023-12-14T06:30:48,376 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T06:30:48,376 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T06:30:48,377 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T06:30:48,377 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T06:30:48,377 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T06:30:48,377 streaming updates afterwards. 2023-12-14T06:30:48,377 ## Issues 2023-12-14T06:30:48,377 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T06:30:48,378 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T06:30:48,378 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T06:30:48,378 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T06:30:48,378 ## Alternatives 2023-12-14T06:30:48,378 There are many alternatives: 2023-12-14T06:30:48,379 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T06:30:48,379 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T06:30:48,379 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T06:30:48,379 * Redis 2023-12-14T06:30:48,379 * Memcached 2023-12-14T06:30:48,379 ## How to use? 2023-12-14T06:30:48,380 ### Simple 2023-12-14T06:30:48,380 In one Python REPL: 2023-12-14T06:30:48,380 ```python 2023-12-14T06:30:48,380 Python 3.9.2 on linux 2023-12-14T06:30:48,380 >>> 2023-12-14T06:30:48,380 >>> from i18n_json import i18n_json 2023-12-14T06:30:48,380 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T06:30:48,381 >>> ultra 2023-12-14T06:30:48,381 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:48,381 >>> 2023-12-14T06:30:48,381 >>> # We need the shared memory name in the other process. 2023-12-14T06:30:48,381 >>> ultra.name 2023-12-14T06:30:48,381 'psm_ad73da69' 2023-12-14T06:30:48,381 ``` 2023-12-14T06:30:48,382 In another Python REPL: 2023-12-14T06:30:48,382 ```python 2023-12-14T06:30:48,382 Python 3.9.2 on linux 2023-12-14T06:30:48,382 >>> 2023-12-14T06:30:48,382 >>> from i18n_json import i18n_json 2023-12-14T06:30:48,382 >>> # Connect to the shared memory with the name above 2023-12-14T06:30:48,382 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T06:30:48,382 >>> other 2023-12-14T06:30:48,383 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:48,383 >>> other[2] = 2 2023-12-14T06:30:48,383 ``` 2023-12-14T06:30:48,383 Back in the first Python REPL: 2023-12-14T06:30:48,383 ```python 2023-12-14T06:30:48,383 >>> ultra[2] 2023-12-14T06:30:48,383 2 2023-12-14T06:30:48,384 ``` 2023-12-14T06:30:48,384 ### Nested 2023-12-14T06:30:48,384 In one Python REPL: 2023-12-14T06:30:48,384 ```python 2023-12-14T06:30:48,384 Python 3.9.2 on linux 2023-12-14T06:30:48,384 >>> 2023-12-14T06:30:48,385 >>> from i18n_json import i18n_json 2023-12-14T06:30:48,385 >>> ultra = i18n_json(recurse=True) 2023-12-14T06:30:48,385 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T06:30:48,385 >>> type(ultra['nested']) 2023-12-14T06:30:48,385 2023-12-14T06:30:48,385 >>> ultra.name 2023-12-14T06:30:48,385 'psm_0a2713e4' 2023-12-14T06:30:48,385 ``` 2023-12-14T06:30:48,386 In another Python REPL: 2023-12-14T06:30:48,386 ```python 2023-12-14T06:30:48,386 Python 3.9.2 on linux 2023-12-14T06:30:48,386 >>> 2023-12-14T06:30:48,386 >>> from i18n_json import i18n_json 2023-12-14T06:30:48,386 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T06:30:48,386 >>> other['nested']['counter'] += 1 2023-12-14T06:30:48,386 ``` 2023-12-14T06:30:48,387 Back in the first Python REPL: 2023-12-14T06:30:48,387 ```python 2023-12-14T06:30:48,387 >>> ultra['nested']['counter'] 2023-12-14T06:30:48,387 1 2023-12-14T06:30:48,387 ``` 2023-12-14T06:30:48,387 ## Performance comparison 2023-12-14T06:30:48,388 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T06:30:48,388 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T06:30:48,388 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T06:30:48,388 ```python 2023-12-14T06:30:48,388 Python 3.9.2 on linux 2023-12-14T06:30:48,388 >>> 2023-12-14T06:30:48,389 >>> from i18n_json import i18n_json 2023-12-14T06:30:48,389 >>> ultra = i18n_json() 2023-12-14T06:30:48,389 >>> for i in range(10_000): ultra[i] = i 2023-12-14T06:30:48,389 ... 2023-12-14T06:30:48,389 >>> len(ultra) 2023-12-14T06:30:48,389 10000 2023-12-14T06:30:48,389 >>> ultra[500] 2023-12-14T06:30:48,389 500 2023-12-14T06:30:48,389 >>> # Now let's do some performance testing 2023-12-14T06:30:48,390 >>> import multiprocessing, redis, timeit 2023-12-14T06:30:48,390 >>> orig = dict(ultra) 2023-12-14T06:30:48,390 >>> len(orig) 2023-12-14T06:30:48,390 10000 2023-12-14T06:30:48,390 >>> orig[500] 2023-12-14T06:30:48,390 500 2023-12-14T06:30:48,390 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T06:30:48,390 >>> len(managed) 2023-12-14T06:30:48,390 10000 2023-12-14T06:30:48,391 >>> r = redis.Redis() 2023-12-14T06:30:48,391 >>> r.flushall() 2023-12-14T06:30:48,391 >>> r.mset(orig) 2023-12-14T06:30:48,391 ``` 2023-12-14T06:30:48,391 ### Read performance 2023-12-14T06:30:48,391 >>> 2023-12-14T06:30:48,391 ```python 2023-12-14T06:30:48,392 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T06:30:48,392 0.03832335816696286 2023-12-14T06:30:48,392 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T06:30:48,392 0.5248982920311391 2023-12-14T06:30:48,392 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T06:30:48,392 40.85506196087226 2023-12-14T06:30:48,392 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T06:30:48,392 49.3497632863 2023-12-14T06:30:48,392 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T06:30:48,393 0.04309639008715749 2023-12-14T06:30:48,393 ``` 2023-12-14T06:30:48,393 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T06:30:48,393 ### Write performance 2023-12-14T06:30:48,393 ```python 2023-12-14T06:30:48,393 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T06:30:48,394 0.028232071083039045 2023-12-14T06:30:48,394 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T06:30:48,394 2.911152713932097 2023-12-14T06:30:48,394 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T06:30:48,394 31.641707635018975 2023-12-14T06:30:48,394 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T06:30:48,394 124.3432381930761 2023-12-14T06:30:48,394 ``` 2023-12-14T06:30:48,394 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T06:30:48,395 ### Testing performance 2023-12-14T06:30:48,395 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T06:30:48,395 ```bash 2023-12-14T06:30:48,395 python ./tests/performance/performance.py 2023-12-14T06:30:48,395 Testing Performance with 1000000 operations each 2023-12-14T06:30:48,396 Redis (writes) = 24,351 ops per second 2023-12-14T06:30:48,396 Redis (reads) = 30,466 ops per second 2023-12-14T06:30:48,396 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T06:30:48,396 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T06:30:48,396 Python dict (writes) = 16,413,569 ops per second 2023-12-14T06:30:48,396 Python dict (reads) = 16,479,191 ops per second 2023-12-14T06:30:48,396 i18n_json (writes) = 479,860 ops per second 2023-12-14T06:30:48,396 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T06:30:48,397 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T06:30:48,397 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T06:30:48,397 Ranking: 2023-12-14T06:30:48,397 writes: 2023-12-14T06:30:48,397 Python dict = 16,413,569 (factor 1.0) 2023-12-14T06:30:48,397 i18n_json = 479,860 (factor 34.2) 2023-12-14T06:30:48,397 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T06:30:48,397 Redis = 24,351 (factor 674.04) 2023-12-14T06:30:48,397 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T06:30:48,398 reads: 2023-12-14T06:30:48,398 Python dict = 16,479,191 (factor 1.0) 2023-12-14T06:30:48,398 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T06:30:48,398 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T06:30:48,398 Redis = 30,466 (factor 540.9) 2023-12-14T06:30:48,398 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T06:30:48,398 ``` 2023-12-14T06:30:48,398 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T06:30:48,399 ## Parameters 2023-12-14T06:30:48,399 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T06:30:48,399 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T06:30:48,399 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T06:30:48,399 memory space is attached. 2023-12-14T06:30:48,400 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T06:30:48,400 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T06:30:48,400 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T06:30:48,400 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T06:30:48,400 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T06:30:48,400 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T06:30:48,400 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T06:30:48,401 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T06:30:48,401 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T06:30:48,401 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T06:30:48,401 dict will automatically load the full dump and continue streaming updates. 2023-12-14T06:30:48,401 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T06:30:48,401 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T06:30:48,402 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T06:30:48,402 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T06:30:48,402 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T06:30:48,402 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T06:30:48,402 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T06:30:48,402 (Also see the section [Locking](#locking) below!) 2023-12-14T06:30:48,403 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T06:30:48,403 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T06:30:48,403 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T06:30:48,403 be writing to the dict and therefore creating full dumps. 2023-12-14T06:30:48,403 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T06:30:48,403 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T06:30:48,403 dict. 2023-12-14T06:30:48,403 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T06:30:48,404 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T06:30:48,404 ## Memory management 2023-12-14T06:30:48,404 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T06:30:48,404 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T06:30:48,404 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T06:30:48,405 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T06:30:48,405 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T06:30:48,405 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T06:30:48,405 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T06:30:48,405 **Buffer sizes and read performance:** 2023-12-14T06:30:48,405 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T06:30:48,406 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T06:30:48,406 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T06:30:48,406 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T06:30:48,406 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T06:30:48,407 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T06:30:48,407 ## Locking 2023-12-14T06:30:48,407 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T06:30:48,407 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T06:30:48,407 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T06:30:48,408 ### How to use the locking? 2023-12-14T06:30:48,408 ```python 2023-12-14T06:30:48,408 ultra = i18n_json(shared_lock=True) 2023-12-14T06:30:48,408 with ultra.lock: 2023-12-14T06:30:48,408 ultra['counter']++ 2023-12-14T06:30:48,408 # The same as above with all default parameters 2023-12-14T06:30:48,408 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T06:30:48,409 ultra['counter']++ 2023-12-14T06:30:48,409 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T06:30:48,409 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T06:30:48,409 with ultra.lock(sleep_time=0): 2023-12-14T06:30:48,409 ultra['counter']++ 2023-12-14T06:30:48,409 try: 2023-12-14T06:30:48,409 result = ultra.lock.acquire(block=False) 2023-12-14T06:30:48,410 ultra.lock.release() 2023-12-14T06:30:48,410 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T06:30:48,410 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T06:30:48,410 try: 2023-12-14T06:30:48,410 with ultra.lock(timeout=1.5): 2023-12-14T06:30:48,410 ultra['counter']++ 2023-12-14T06:30:48,410 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T06:30:48,410 print('Stale lock?') 2023-12-14T06:30:48,411 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T06:30:48,411 ultra['counter']++ 2023-12-14T06:30:48,411 ``` 2023-12-14T06:30:48,411 ## Explicit cleanup 2023-12-14T06:30:48,411 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T06:30:48,412 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T06:30:48,412 Another way to do this in code is like this: 2023-12-14T06:30:48,412 ```python 2023-12-14T06:30:48,412 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T06:30:48,412 name = 'my-dict-name' 2023-12-14T06:30:48,412 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T06:30:48,412 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T06:30:48,412 ``` 2023-12-14T06:30:48,413 ## Advanced usage 2023-12-14T06:30:48,413 See [examples](/examples) folder 2023-12-14T06:30:48,413 ```python 2023-12-14T06:30:48,413 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T06:30:48,413 >>> # Let's use a value with 100k bytes length. 2023-12-14T06:30:48,413 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T06:30:48,414 >>> ultra[0] = ' ' * 100_000 2023-12-14T06:30:48,414 >>> ultra.print_status() 2023-12-14T06:30:48,414 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T06:30:48,414 'buffer_size': 100000, 2023-12-14T06:30:48,414 'control': SharedMemory('my-name', size=1000), 2023-12-14T06:30:48,414 'full_dump_counter': 1, 2023-12-14T06:30:48,414 'full_dump_counter_remote': 1, 2023-12-14T06:30:48,414 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T06:30:48,414 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T06:30:48,415 'full_dump_size': None, 2023-12-14T06:30:48,415 'full_dump_static_size_remote': , 2023-12-14T06:30:48,415 'lock': , 2023-12-14T06:30:48,415 'lock_pid_remote': 0, 2023-12-14T06:30:48,415 'lock_remote': 0, 2023-12-14T06:30:48,415 'name': 'my-name', 2023-12-14T06:30:48,415 'recurse': False, 2023-12-14T06:30:48,415 'recurse_remote': , 2023-12-14T06:30:48,415 'serializer': , 2023-12-14T06:30:48,415 'shared_lock_remote': , 2023-12-14T06:30:48,416 'update_stream_position': 0, 2023-12-14T06:30:48,416 'update_stream_position_remote': 0} 2023-12-14T06:30:48,416 ``` 2023-12-14T06:30:48,416 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T06:30:48,416 Other things you can do: 2023-12-14T06:30:48,416 ```python 2023-12-14T06:30:48,416 >>> # Create a full dump 2023-12-14T06:30:48,417 >>> ultra.dump() 2023-12-14T06:30:48,417 >>> # Load latest full dump if one is available 2023-12-14T06:30:48,417 >>> ultra.load() 2023-12-14T06:30:48,417 >>> # Show statistics 2023-12-14T06:30:48,417 >>> ultra.print_status() 2023-12-14T06:30:48,417 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T06:30:48,418 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T06:30:48,418 >>> ultra.load(force=True) 2023-12-14T06:30:48,418 >>> # Apply full dump and stream updates to 2023-12-14T06:30:48,418 >>> # underlying local dict, this is automatically 2023-12-14T06:30:48,418 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T06:30:48,418 >>> # but can be useful to call after a forced load. 2023-12-14T06:30:48,418 >>> ultra.apply_update() 2023-12-14T06:30:48,419 >>> # Access underlying local dict directly for maximum performance 2023-12-14T06:30:48,419 >>> ultra.data 2023-12-14T06:30:48,419 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T06:30:48,419 >>> import jsons 2023-12-14T06:30:48,419 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T06:30:48,419 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T06:30:48,419 >>> ultra.close() 2023-12-14T06:30:48,420 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T06:30:48,420 >>> ultra.unlink() 2023-12-14T06:30:48,420 ``` 2023-12-14T06:30:48,420 ## Contributing 2023-12-14T06:30:48,420 Contributions are always welcome! 2023-12-14T06:30:51,251 Compiling i18n_json.py because it changed. 2023-12-14T06:30:51,251 [1/1] Cythonizing i18n_json.py 2023-12-14T06:30:51,484 Getting requirements to build wheel: finished with status 'done' 2023-12-14T06:30:51,499 Created temporary directory: /tmp/pip-modern-metadata-r02dz1c1 2023-12-14T06:30:51,501 Preparing metadata (pyproject.toml): started 2023-12-14T06:30:51,502 Running command Preparing metadata (pyproject.toml) 2023-12-14T06:30:52,119 # i18n_json 2023-12-14T06:30:52,120 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T06:30:52,120 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T06:30:52,121 Features: 2023-12-14T06:30:52,121 * Fast (compared to other sharing solutions) 2023-12-14T06:30:52,121 * No running manager processes 2023-12-14T06:30:52,121 * Works in spawn and fork context 2023-12-14T06:30:52,121 * Safe locking between independent processes 2023-12-14T06:30:52,121 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T06:30:52,121 * Convenient, no setter or getters necessary 2023-12-14T06:30:52,121 * Optional recursion for nested dicts 2023-12-14T06:30:52,122 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T06:30:52,122 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T06:30:52,122 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T06:30:52,122 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T06:30:52,122 ## General Concept 2023-12-14T06:30:52,123 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T06:30:52,123 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T06:30:52,123 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T06:30:52,123 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T06:30:52,123 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T06:30:52,124 streaming updates afterwards. 2023-12-14T06:30:52,124 ## Issues 2023-12-14T06:30:52,124 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T06:30:52,124 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T06:30:52,124 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T06:30:52,124 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T06:30:52,125 ## Alternatives 2023-12-14T06:30:52,125 There are many alternatives: 2023-12-14T06:30:52,125 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T06:30:52,125 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T06:30:52,125 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T06:30:52,125 * Redis 2023-12-14T06:30:52,126 * Memcached 2023-12-14T06:30:52,126 ## How to use? 2023-12-14T06:30:52,126 ### Simple 2023-12-14T06:30:52,126 In one Python REPL: 2023-12-14T06:30:52,126 ```python 2023-12-14T06:30:52,126 Python 3.9.2 on linux 2023-12-14T06:30:52,127 >>> 2023-12-14T06:30:52,127 >>> from i18n_json import i18n_json 2023-12-14T06:30:52,127 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T06:30:52,127 >>> ultra 2023-12-14T06:30:52,127 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:52,127 >>> 2023-12-14T06:30:52,127 >>> # We need the shared memory name in the other process. 2023-12-14T06:30:52,127 >>> ultra.name 2023-12-14T06:30:52,128 'psm_ad73da69' 2023-12-14T06:30:52,128 ``` 2023-12-14T06:30:52,128 In another Python REPL: 2023-12-14T06:30:52,128 ```python 2023-12-14T06:30:52,128 Python 3.9.2 on linux 2023-12-14T06:30:52,128 >>> 2023-12-14T06:30:52,128 >>> from i18n_json import i18n_json 2023-12-14T06:30:52,129 >>> # Connect to the shared memory with the name above 2023-12-14T06:30:52,129 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T06:30:52,129 >>> other 2023-12-14T06:30:52,129 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:52,129 >>> other[2] = 2 2023-12-14T06:30:52,129 ``` 2023-12-14T06:30:52,129 Back in the first Python REPL: 2023-12-14T06:30:52,130 ```python 2023-12-14T06:30:52,130 >>> ultra[2] 2023-12-14T06:30:52,130 2 2023-12-14T06:30:52,130 ``` 2023-12-14T06:30:52,130 ### Nested 2023-12-14T06:30:52,130 In one Python REPL: 2023-12-14T06:30:52,131 ```python 2023-12-14T06:30:52,131 Python 3.9.2 on linux 2023-12-14T06:30:52,131 >>> 2023-12-14T06:30:52,131 >>> from i18n_json import i18n_json 2023-12-14T06:30:52,131 >>> ultra = i18n_json(recurse=True) 2023-12-14T06:30:52,131 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T06:30:52,131 >>> type(ultra['nested']) 2023-12-14T06:30:52,131 2023-12-14T06:30:52,131 >>> ultra.name 2023-12-14T06:30:52,132 'psm_0a2713e4' 2023-12-14T06:30:52,132 ``` 2023-12-14T06:30:52,132 In another Python REPL: 2023-12-14T06:30:52,132 ```python 2023-12-14T06:30:52,132 Python 3.9.2 on linux 2023-12-14T06:30:52,132 >>> 2023-12-14T06:30:52,132 >>> from i18n_json import i18n_json 2023-12-14T06:30:52,133 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T06:30:52,133 >>> other['nested']['counter'] += 1 2023-12-14T06:30:52,133 ``` 2023-12-14T06:30:52,133 Back in the first Python REPL: 2023-12-14T06:30:52,133 ```python 2023-12-14T06:30:52,133 >>> ultra['nested']['counter'] 2023-12-14T06:30:52,133 1 2023-12-14T06:30:52,133 ``` 2023-12-14T06:30:52,134 ## Performance comparison 2023-12-14T06:30:52,134 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T06:30:52,134 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T06:30:52,134 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T06:30:52,134 ```python 2023-12-14T06:30:52,135 Python 3.9.2 on linux 2023-12-14T06:30:52,135 >>> 2023-12-14T06:30:52,135 >>> from i18n_json import i18n_json 2023-12-14T06:30:52,135 >>> ultra = i18n_json() 2023-12-14T06:30:52,135 >>> for i in range(10_000): ultra[i] = i 2023-12-14T06:30:52,135 ... 2023-12-14T06:30:52,135 >>> len(ultra) 2023-12-14T06:30:52,135 10000 2023-12-14T06:30:52,135 >>> ultra[500] 2023-12-14T06:30:52,136 500 2023-12-14T06:30:52,136 >>> # Now let's do some performance testing 2023-12-14T06:30:52,136 >>> import multiprocessing, redis, timeit 2023-12-14T06:30:52,136 >>> orig = dict(ultra) 2023-12-14T06:30:52,136 >>> len(orig) 2023-12-14T06:30:52,136 10000 2023-12-14T06:30:52,136 >>> orig[500] 2023-12-14T06:30:52,136 500 2023-12-14T06:30:52,136 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T06:30:52,137 >>> len(managed) 2023-12-14T06:30:52,137 10000 2023-12-14T06:30:52,137 >>> r = redis.Redis() 2023-12-14T06:30:52,137 >>> r.flushall() 2023-12-14T06:30:52,137 >>> r.mset(orig) 2023-12-14T06:30:52,137 ``` 2023-12-14T06:30:52,137 ### Read performance 2023-12-14T06:30:52,138 >>> 2023-12-14T06:30:52,138 ```python 2023-12-14T06:30:52,138 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T06:30:52,138 0.03832335816696286 2023-12-14T06:30:52,138 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T06:30:52,138 0.5248982920311391 2023-12-14T06:30:52,138 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T06:30:52,138 40.85506196087226 2023-12-14T06:30:52,138 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T06:30:52,139 49.3497632863 2023-12-14T06:30:52,139 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T06:30:52,139 0.04309639008715749 2023-12-14T06:30:52,139 ``` 2023-12-14T06:30:52,139 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T06:30:52,139 ### Write performance 2023-12-14T06:30:52,140 ```python 2023-12-14T06:30:52,140 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T06:30:52,140 0.028232071083039045 2023-12-14T06:30:52,140 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T06:30:52,140 2.911152713932097 2023-12-14T06:30:52,140 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T06:30:52,140 31.641707635018975 2023-12-14T06:30:52,140 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T06:30:52,140 124.3432381930761 2023-12-14T06:30:52,141 ``` 2023-12-14T06:30:52,141 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T06:30:52,141 ### Testing performance 2023-12-14T06:30:52,141 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T06:30:52,141 ```bash 2023-12-14T06:30:52,142 python ./tests/performance/performance.py 2023-12-14T06:30:52,142 Testing Performance with 1000000 operations each 2023-12-14T06:30:52,142 Redis (writes) = 24,351 ops per second 2023-12-14T06:30:52,142 Redis (reads) = 30,466 ops per second 2023-12-14T06:30:52,142 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T06:30:52,142 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T06:30:52,142 Python dict (writes) = 16,413,569 ops per second 2023-12-14T06:30:52,143 Python dict (reads) = 16,479,191 ops per second 2023-12-14T06:30:52,143 i18n_json (writes) = 479,860 ops per second 2023-12-14T06:30:52,143 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T06:30:52,143 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T06:30:52,143 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T06:30:52,143 Ranking: 2023-12-14T06:30:52,143 writes: 2023-12-14T06:30:52,143 Python dict = 16,413,569 (factor 1.0) 2023-12-14T06:30:52,143 i18n_json = 479,860 (factor 34.2) 2023-12-14T06:30:52,144 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T06:30:52,144 Redis = 24,351 (factor 674.04) 2023-12-14T06:30:52,144 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T06:30:52,144 reads: 2023-12-14T06:30:52,144 Python dict = 16,479,191 (factor 1.0) 2023-12-14T06:30:52,144 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T06:30:52,144 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T06:30:52,144 Redis = 30,466 (factor 540.9) 2023-12-14T06:30:52,144 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T06:30:52,145 ``` 2023-12-14T06:30:52,145 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T06:30:52,145 ## Parameters 2023-12-14T06:30:52,145 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T06:30:52,145 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T06:30:52,146 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T06:30:52,146 memory space is attached. 2023-12-14T06:30:52,146 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T06:30:52,146 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T06:30:52,146 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T06:30:52,146 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T06:30:52,146 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T06:30:52,147 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T06:30:52,147 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T06:30:52,147 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T06:30:52,147 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T06:30:52,147 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T06:30:52,147 dict will automatically load the full dump and continue streaming updates. 2023-12-14T06:30:52,148 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T06:30:52,148 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T06:30:52,148 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T06:30:52,148 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T06:30:52,148 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T06:30:52,148 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T06:30:52,148 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T06:30:52,149 (Also see the section [Locking](#locking) below!) 2023-12-14T06:30:52,149 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T06:30:52,149 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T06:30:52,149 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T06:30:52,149 be writing to the dict and therefore creating full dumps. 2023-12-14T06:30:52,149 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T06:30:52,149 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T06:30:52,150 dict. 2023-12-14T06:30:52,150 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T06:30:52,150 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T06:30:52,150 ## Memory management 2023-12-14T06:30:52,150 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T06:30:52,151 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T06:30:52,151 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T06:30:52,151 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T06:30:52,151 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T06:30:52,151 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T06:30:52,151 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T06:30:52,152 **Buffer sizes and read performance:** 2023-12-14T06:30:52,152 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T06:30:52,152 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T06:30:52,152 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T06:30:52,152 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T06:30:52,153 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T06:30:52,153 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T06:30:52,153 ## Locking 2023-12-14T06:30:52,153 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T06:30:52,154 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T06:30:52,154 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T06:30:52,154 ### How to use the locking? 2023-12-14T06:30:52,154 ```python 2023-12-14T06:30:52,154 ultra = i18n_json(shared_lock=True) 2023-12-14T06:30:52,154 with ultra.lock: 2023-12-14T06:30:52,154 ultra['counter']++ 2023-12-14T06:30:52,155 # The same as above with all default parameters 2023-12-14T06:30:52,155 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T06:30:52,155 ultra['counter']++ 2023-12-14T06:30:52,155 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T06:30:52,155 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T06:30:52,155 with ultra.lock(sleep_time=0): 2023-12-14T06:30:52,155 ultra['counter']++ 2023-12-14T06:30:52,156 try: 2023-12-14T06:30:52,156 result = ultra.lock.acquire(block=False) 2023-12-14T06:30:52,156 ultra.lock.release() 2023-12-14T06:30:52,156 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T06:30:52,156 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T06:30:52,156 try: 2023-12-14T06:30:52,156 with ultra.lock(timeout=1.5): 2023-12-14T06:30:52,157 ultra['counter']++ 2023-12-14T06:30:52,157 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T06:30:52,157 print('Stale lock?') 2023-12-14T06:30:52,157 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T06:30:52,157 ultra['counter']++ 2023-12-14T06:30:52,157 ``` 2023-12-14T06:30:52,158 ## Explicit cleanup 2023-12-14T06:30:52,158 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T06:30:52,158 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T06:30:52,158 Another way to do this in code is like this: 2023-12-14T06:30:52,158 ```python 2023-12-14T06:30:52,158 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T06:30:52,158 name = 'my-dict-name' 2023-12-14T06:30:52,159 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T06:30:52,159 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T06:30:52,159 ``` 2023-12-14T06:30:52,159 ## Advanced usage 2023-12-14T06:30:52,159 See [examples](/examples) folder 2023-12-14T06:30:52,159 ```python 2023-12-14T06:30:52,160 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T06:30:52,160 >>> # Let's use a value with 100k bytes length. 2023-12-14T06:30:52,160 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T06:30:52,160 >>> ultra[0] = ' ' * 100_000 2023-12-14T06:30:52,160 >>> ultra.print_status() 2023-12-14T06:30:52,160 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T06:30:52,160 'buffer_size': 100000, 2023-12-14T06:30:52,160 'control': SharedMemory('my-name', size=1000), 2023-12-14T06:30:52,160 'full_dump_counter': 1, 2023-12-14T06:30:52,161 'full_dump_counter_remote': 1, 2023-12-14T06:30:52,161 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T06:30:52,161 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T06:30:52,161 'full_dump_size': None, 2023-12-14T06:30:52,161 'full_dump_static_size_remote': , 2023-12-14T06:30:52,161 'lock': , 2023-12-14T06:30:52,161 'lock_pid_remote': 0, 2023-12-14T06:30:52,161 'lock_remote': 0, 2023-12-14T06:30:52,161 'name': 'my-name', 2023-12-14T06:30:52,161 'recurse': False, 2023-12-14T06:30:52,162 'recurse_remote': , 2023-12-14T06:30:52,162 'serializer': , 2023-12-14T06:30:52,162 'shared_lock_remote': , 2023-12-14T06:30:52,162 'update_stream_position': 0, 2023-12-14T06:30:52,162 'update_stream_position_remote': 0} 2023-12-14T06:30:52,162 ``` 2023-12-14T06:30:52,162 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T06:30:52,163 Other things you can do: 2023-12-14T06:30:52,163 ```python 2023-12-14T06:30:52,163 >>> # Create a full dump 2023-12-14T06:30:52,163 >>> ultra.dump() 2023-12-14T06:30:52,163 >>> # Load latest full dump if one is available 2023-12-14T06:30:52,163 >>> ultra.load() 2023-12-14T06:30:52,163 >>> # Show statistics 2023-12-14T06:30:52,164 >>> ultra.print_status() 2023-12-14T06:30:52,164 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T06:30:52,164 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T06:30:52,164 >>> ultra.load(force=True) 2023-12-14T06:30:52,164 >>> # Apply full dump and stream updates to 2023-12-14T06:30:52,164 >>> # underlying local dict, this is automatically 2023-12-14T06:30:52,164 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T06:30:52,165 >>> # but can be useful to call after a forced load. 2023-12-14T06:30:52,165 >>> ultra.apply_update() 2023-12-14T06:30:52,165 >>> # Access underlying local dict directly for maximum performance 2023-12-14T06:30:52,165 >>> ultra.data 2023-12-14T06:30:52,165 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T06:30:52,165 >>> import jsons 2023-12-14T06:30:52,165 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T06:30:52,166 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T06:30:52,166 >>> ultra.close() 2023-12-14T06:30:52,166 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T06:30:52,166 >>> ultra.unlink() 2023-12-14T06:30:52,166 ``` 2023-12-14T06:30:52,166 ## Contributing 2023-12-14T06:30:52,167 Contributions are always welcome! 2023-12-14T06:30:52,719 running dist_info 2023-12-14T06:30:52,744 creating /tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info 2023-12-14T06:30:52,752 writing /tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/PKG-INFO 2023-12-14T06:30:52,756 writing dependency_links to /tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/dependency_links.txt 2023-12-14T06:30:52,759 writing top-level names to /tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/top_level.txt 2023-12-14T06:30:52,760 writing manifest file '/tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/SOURCES.txt' 2023-12-14T06:30:52,791 reading manifest file '/tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/SOURCES.txt' 2023-12-14T06:30:52,793 reading manifest template 'MANIFEST.in' 2023-12-14T06:30:52,794 adding license file 'LICENSE' 2023-12-14T06:30:52,796 writing manifest file '/tmp/pip-modern-metadata-r02dz1c1/i18n_json.egg-info/SOURCES.txt' 2023-12-14T06:30:52,797 creating '/tmp/pip-modern-metadata-r02dz1c1/i18n_json-0.0.9.dist-info' 2023-12-14T06:30:53,003 Preparing metadata (pyproject.toml): finished with status 'done' 2023-12-14T06:30:53,010 Source in /tmp/pip-wheel-3_nzz3dz/i18n-json_40b15a505ed34cbaa863df6dc5ff1387 has version 0.0.9, which satisfies requirement i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz 2023-12-14T06:30:53,011 Removed i18n-json==0.0.9 from https://files.pythonhosted.org/packages/47/31/ce995b6b9e834d8978f1c20c702f74e3feeab62d10143b8b61090cfe192a/i18n_json-0.0.9.tar.gz from build tracker '/tmp/pip-build-tracker-wm5p1b0q' 2023-12-14T06:30:53,018 Created temporary directory: /tmp/pip-unpack-q84twpdk 2023-12-14T06:30:53,019 Created temporary directory: /tmp/pip-unpack-rs0pe7lh 2023-12-14T06:30:53,022 Building wheels for collected packages: i18n-json 2023-12-14T06:30:53,026 Created temporary directory: /tmp/pip-wheel-vqfsb_9d 2023-12-14T06:30:53,026 Destination directory: /tmp/pip-wheel-vqfsb_9d 2023-12-14T06:30:53,028 Building wheel for i18n-json (pyproject.toml): started 2023-12-14T06:30:53,029 Running command Building wheel for i18n-json (pyproject.toml) 2023-12-14T06:30:53,670 # i18n_json 2023-12-14T06:30:53,671 Sychronized, streaming Python dictionary that uses shared memory as a backend 2023-12-14T06:30:53,671 **Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!** 2023-12-14T06:30:53,671 Features: 2023-12-14T06:30:53,672 * Fast (compared to other sharing solutions) 2023-12-14T06:30:53,672 * No running manager processes 2023-12-14T06:30:53,672 * Works in spawn and fork context 2023-12-14T06:30:53,672 * Safe locking between independent processes 2023-12-14T06:30:53,672 * Tested with Python >= v3.8 on Linux, Windows and Mac 2023-12-14T06:30:53,672 * Convenient, no setter or getters necessary 2023-12-14T06:30:53,672 * Optional recursion for nested dicts 2023-12-14T06:30:53,673 [![PyPI Package](https://img.shields.io/pypi/v/i18n_json.svg)](https://pypi.org/project/i18n_json) 2023-12-14T06:30:53,673 [![Run Python Tests](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ronny-rentner/i18n_json/actions/workflows/ci.yml) 2023-12-14T06:30:53,673 [![Python >=3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) 2023-12-14T06:30:53,673 [![License](https://img.shields.io/github/license/ronny-rentner/i18n_json.svg)](https://github.com/ronny-rentner/i18n_json/blob/master/LICENSE.md) 2023-12-14T06:30:53,673 ## General Concept 2023-12-14T06:30:53,674 `i18n_json` uses [multiprocessing.shared_memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory) to synchronize a dict between multiple processes. 2023-12-14T06:30:53,674 It does so by using a *stream of updates* in a shared memory buffer. This is efficient because only changes have to be serialized and transferred. 2023-12-14T06:30:53,674 If the buffer is full, `i18n_json` will automatically do a full dump to a new shared 2023-12-14T06:30:53,674 memory space, reset the streaming buffer and continue to stream further updates. All users 2023-12-14T06:30:53,674 of the `i18n_json` will automatically load full dumps and continue using 2023-12-14T06:30:53,674 streaming updates afterwards. 2023-12-14T06:30:53,675 ## Issues 2023-12-14T06:30:53,675 On Windows, if no process has any handles on the shared memory, the OS will gc all of the shared memory making it inaccessible for 2023-12-14T06:30:53,675 future processes. To work around this issue you can currently set `full_dump_size` which will cause the creator 2023-12-14T06:30:53,675 of the dict to set a static full dump memory of the requested size. This full dump memory will live as long as the creator lives. 2023-12-14T06:30:53,675 This approach has the downside that you need to plan ahead for your data size and if it does not fit into the full dump memory, it will break. 2023-12-14T06:30:53,675 ## Alternatives 2023-12-14T06:30:53,676 There are many alternatives: 2023-12-14T06:30:53,676 * [multiprocessing.Manager](https://docs.python.org/3/library/multiprocessing.html#managers) 2023-12-14T06:30:53,676 * [shared-memory-dict](https://github.com/luizalabs/shared-memory-dict) 2023-12-14T06:30:53,676 * [mpdict](https://github.com/gatopeich/mpdict) 2023-12-14T06:30:53,676 * Redis 2023-12-14T06:30:53,676 * Memcached 2023-12-14T06:30:53,677 ## How to use? 2023-12-14T06:30:53,677 ### Simple 2023-12-14T06:30:53,677 In one Python REPL: 2023-12-14T06:30:53,677 ```python 2023-12-14T06:30:53,677 Python 3.9.2 on linux 2023-12-14T06:30:53,677 >>> 2023-12-14T06:30:53,678 >>> from i18n_json import i18n_json 2023-12-14T06:30:53,678 >>> ultra = i18n_json({ 1:1 }, some_key='some_value') 2023-12-14T06:30:53,678 >>> ultra 2023-12-14T06:30:53,678 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:53,678 >>> 2023-12-14T06:30:53,678 >>> # We need the shared memory name in the other process. 2023-12-14T06:30:53,678 >>> ultra.name 2023-12-14T06:30:53,678 'psm_ad73da69' 2023-12-14T06:30:53,679 ``` 2023-12-14T06:30:53,679 In another Python REPL: 2023-12-14T06:30:53,679 ```python 2023-12-14T06:30:53,679 Python 3.9.2 on linux 2023-12-14T06:30:53,679 >>> 2023-12-14T06:30:53,679 >>> from i18n_json import i18n_json 2023-12-14T06:30:53,679 >>> # Connect to the shared memory with the name above 2023-12-14T06:30:53,680 >>> other = i18n_json(name='psm_ad73da69') 2023-12-14T06:30:53,680 >>> other 2023-12-14T06:30:53,680 {1: 1, 'some_key': 'some_value'} 2023-12-14T06:30:53,680 >>> other[2] = 2 2023-12-14T06:30:53,680 ``` 2023-12-14T06:30:53,680 Back in the first Python REPL: 2023-12-14T06:30:53,680 ```python 2023-12-14T06:30:53,681 >>> ultra[2] 2023-12-14T06:30:53,681 2 2023-12-14T06:30:53,681 ``` 2023-12-14T06:30:53,681 ### Nested 2023-12-14T06:30:53,681 In one Python REPL: 2023-12-14T06:30:53,681 ```python 2023-12-14T06:30:53,682 Python 3.9.2 on linux 2023-12-14T06:30:53,682 >>> 2023-12-14T06:30:53,682 >>> from i18n_json import i18n_json 2023-12-14T06:30:53,682 >>> ultra = i18n_json(recurse=True) 2023-12-14T06:30:53,682 >>> ultra['nested'] = { 'counter': 0 } 2023-12-14T06:30:53,682 >>> type(ultra['nested']) 2023-12-14T06:30:53,682 2023-12-14T06:30:53,682 >>> ultra.name 2023-12-14T06:30:53,682 'psm_0a2713e4' 2023-12-14T06:30:53,683 ``` 2023-12-14T06:30:53,683 In another Python REPL: 2023-12-14T06:30:53,683 ```python 2023-12-14T06:30:53,683 Python 3.9.2 on linux 2023-12-14T06:30:53,683 >>> 2023-12-14T06:30:53,683 >>> from i18n_json import i18n_json 2023-12-14T06:30:53,683 >>> other = i18n_json(name='psm_0a2713e4') 2023-12-14T06:30:53,683 >>> other['nested']['counter'] += 1 2023-12-14T06:30:53,684 ``` 2023-12-14T06:30:53,684 Back in the first Python REPL: 2023-12-14T06:30:53,684 ```python 2023-12-14T06:30:53,684 >>> ultra['nested']['counter'] 2023-12-14T06:30:53,684 1 2023-12-14T06:30:53,684 ``` 2023-12-14T06:30:53,684 ## Performance comparison 2023-12-14T06:30:53,685 Lets compare a classical Python dict, i18n_json, multiprocessing.Manager and Redis. 2023-12-14T06:30:53,685 Note that this comparison is not a real life workload. It was executed on Debian Linux 11 2023-12-14T06:30:53,685 with Redis installed from the Debian package and with the default configuration of Redis. 2023-12-14T06:30:53,685 ```python 2023-12-14T06:30:53,685 Python 3.9.2 on linux 2023-12-14T06:30:53,685 >>> 2023-12-14T06:30:53,686 >>> from i18n_json import i18n_json 2023-12-14T06:30:53,686 >>> ultra = i18n_json() 2023-12-14T06:30:53,686 >>> for i in range(10_000): ultra[i] = i 2023-12-14T06:30:53,686 ... 2023-12-14T06:30:53,686 >>> len(ultra) 2023-12-14T06:30:53,686 10000 2023-12-14T06:30:53,686 >>> ultra[500] 2023-12-14T06:30:53,686 500 2023-12-14T06:30:53,686 >>> # Now let's do some performance testing 2023-12-14T06:30:53,687 >>> import multiprocessing, redis, timeit 2023-12-14T06:30:53,687 >>> orig = dict(ultra) 2023-12-14T06:30:53,687 >>> len(orig) 2023-12-14T06:30:53,687 10000 2023-12-14T06:30:53,687 >>> orig[500] 2023-12-14T06:30:53,687 500 2023-12-14T06:30:53,687 >>> managed = multiprocessing.Manager().dict(orig) 2023-12-14T06:30:53,687 >>> len(managed) 2023-12-14T06:30:53,687 10000 2023-12-14T06:30:53,687 >>> r = redis.Redis() 2023-12-14T06:30:53,688 >>> r.flushall() 2023-12-14T06:30:53,688 >>> r.mset(orig) 2023-12-14T06:30:53,688 ``` 2023-12-14T06:30:53,688 ### Read performance 2023-12-14T06:30:53,688 >>> 2023-12-14T06:30:53,688 ```python 2023-12-14T06:30:53,688 >>> timeit.timeit('orig[1]', globals=globals()) # original 2023-12-14T06:30:53,689 0.03832335816696286 2023-12-14T06:30:53,689 >>> timeit.timeit('ultra[1]', globals=globals()) # i18n_json 2023-12-14T06:30:53,689 0.5248982920311391 2023-12-14T06:30:53,689 >>> timeit.timeit('managed[1]', globals=globals()) # Manager 2023-12-14T06:30:53,689 40.85506196087226 2023-12-14T06:30:53,689 >>> timeit.timeit('r.get(1)', globals=globals()) # Redis 2023-12-14T06:30:53,689 49.3497632863 2023-12-14T06:30:53,689 >>> timeit.timeit('ultra.data[1]', globals=globals()) # i18n_json data cache 2023-12-14T06:30:53,689 0.04309639008715749 2023-12-14T06:30:53,690 ``` 2023-12-14T06:30:53,690 We are factor 15 slower than a real, local dict, but way faster than using a Manager. If you need full read performance, you can access the underlying cache `ultra.data` directly and get almost original dict performance, of course at the cost of not having real-time updates anymore. 2023-12-14T06:30:53,690 ### Write performance 2023-12-14T06:30:53,690 ```python 2023-12-14T06:30:53,690 >>> min(timeit.repeat('orig[1] = 1', globals=globals())) # original 2023-12-14T06:30:53,690 0.028232071083039045 2023-12-14T06:30:53,691 >>> min(timeit.repeat('ultra[1] = 1', globals=globals())) # i18n_json 2023-12-14T06:30:53,691 2.911152713932097 2023-12-14T06:30:53,691 >>> min(timeit.repeat('managed[1] = 1', globals=globals())) # Manager 2023-12-14T06:30:53,691 31.641707635018975 2023-12-14T06:30:53,691 >>> min(timeit.repeat('r.set(1, 1)', globals=globals())) # Redis 2023-12-14T06:30:53,691 124.3432381930761 2023-12-14T06:30:53,691 ``` 2023-12-14T06:30:53,691 We are factor 100 slower than a real, local Python dict, but still factor 10 faster than using a Manager and much fast than Redis. 2023-12-14T06:30:53,692 ### Testing performance 2023-12-14T06:30:53,692 There is an automated performance test in `tests/performance/performance.py`. If you run it, you get something like this: 2023-12-14T06:30:53,692 ```bash 2023-12-14T06:30:53,692 python ./tests/performance/performance.py 2023-12-14T06:30:53,692 Testing Performance with 1000000 operations each 2023-12-14T06:30:53,693 Redis (writes) = 24,351 ops per second 2023-12-14T06:30:53,693 Redis (reads) = 30,466 ops per second 2023-12-14T06:30:53,693 Python MPM dict (writes) = 19,371 ops per second 2023-12-14T06:30:53,693 Python MPM dict (reads) = 22,290 ops per second 2023-12-14T06:30:53,693 Python dict (writes) = 16,413,569 ops per second 2023-12-14T06:30:53,693 Python dict (reads) = 16,479,191 ops per second 2023-12-14T06:30:53,693 i18n_json (writes) = 479,860 ops per second 2023-12-14T06:30:53,693 i18n_json (reads) = 2,337,944 ops per second 2023-12-14T06:30:53,693 i18n_json (shared_lock=True) (writes) = 41,176 ops per second 2023-12-14T06:30:53,694 i18n_json (shared_lock=True) (reads) = 1,518,652 ops per second 2023-12-14T06:30:53,694 Ranking: 2023-12-14T06:30:53,694 writes: 2023-12-14T06:30:53,694 Python dict = 16,413,569 (factor 1.0) 2023-12-14T06:30:53,694 i18n_json = 479,860 (factor 34.2) 2023-12-14T06:30:53,694 i18n_json (shared_lock=True) = 41,176 (factor 398.62) 2023-12-14T06:30:53,694 Redis = 24,351 (factor 674.04) 2023-12-14T06:30:53,694 Python MPM dict = 19,371 (factor 847.33) 2023-12-14T06:30:53,694 reads: 2023-12-14T06:30:53,695 Python dict = 16,479,191 (factor 1.0) 2023-12-14T06:30:53,695 i18n_json = 2,337,944 (factor 7.05) 2023-12-14T06:30:53,695 i18n_json (shared_lock=True) = 1,518,652 (factor 10.85) 2023-12-14T06:30:53,695 Redis = 30,466 (factor 540.9) 2023-12-14T06:30:53,695 Python MPM dict = 22,290 (factor 739.31) 2023-12-14T06:30:53,695 ``` 2023-12-14T06:30:53,695 I am interested in extending the performance testing to other solutions (like sqlite, memcached, etc.) and to more complex use cases with multiple processes working in parallel. 2023-12-14T06:30:53,695 ## Parameters 2023-12-14T06:30:53,696 `i18n_json(*arg, name=None, create=None, buffer_size=10000, serializer=pickle, shared_lock=False, full_dump_size=None, auto_unlink=None, recurse=False, recurse_register=None, **kwargs)` 2023-12-14T06:30:53,696 `name`: Name of the shared memory. A random name will be chosen if not set. By default, if a name is given 2023-12-14T06:30:53,696 a new shared memory space is created if it does not exist yet. Otherwise the existing shared 2023-12-14T06:30:53,696 memory space is attached. 2023-12-14T06:30:53,696 `create`: Can be either `True` or `False` or `None`. If set to `True`, a new i18n_json will be created 2023-12-14T06:30:53,696 and an exception is thrown if one exists already with the given name. If kept at the default value `None`, 2023-12-14T06:30:53,697 either a new i18n_json will be created if the name is not taken or an existing i18n_json will be attached. 2023-12-14T06:30:53,697 Setting `create=True` does ensure not accidentally attaching to an existing i18n_json that might be left over. 2023-12-14T06:30:53,697 `buffer_size`: Size of the shared memory buffer used for streaming changes of the dict. 2023-12-14T06:30:53,697 The buffer size limits the biggest change that can be streamed, so when you use large values or 2023-12-14T06:30:53,697 deeply nested dicts you might need a bigger buffer. Otherwise, if the buffer is too small, 2023-12-14T06:30:53,697 it will fall back to a full dump. Creating full dumps can be slow, depending on the size of your dict. 2023-12-14T06:30:53,698 Whenever the buffer is full, a full dump will be created. A new shared memory is allocated just 2023-12-14T06:30:53,698 big enough for the full dump. Afterwards the streaming buffer is reset. All other users of the 2023-12-14T06:30:53,698 dict will automatically load the full dump and continue streaming updates. 2023-12-14T06:30:53,698 (Also see the section [Memory management](#memory-management) below!) 2023-12-14T06:30:53,698 `serializer`: Use a different serialized from the default pickle, e. g. marshal, dill, jsons. 2023-12-14T06:30:53,698 The module or object provided must support the methods *loads()* and *dumps()* 2023-12-14T06:30:53,699 `shared_lock`: When writing to the same dict at the same time from multiple, independent processes, 2023-12-14T06:30:53,699 they need a shared lock to synchronize and not overwrite each other's changes. Shared locks are slow. 2023-12-14T06:30:53,699 They rely on the [atomics](https://github.com/doodspav/atomics) package for atomic locks. By default, 2023-12-14T06:30:53,699 i18n_json will use a multiprocessing.RLock() instead which works well in fork context and is much faster. 2023-12-14T06:30:53,699 (Also see the section [Locking](#locking) below!) 2023-12-14T06:30:53,699 `full_dump_size`: If set, uses a static full dump memory instead of dynamically creating it. This 2023-12-14T06:30:53,699 might be necessary on Windows depending on your write behaviour. On Windows, the full dump memory goes 2023-12-14T06:30:53,699 away if the process goes away that had created the full dump. Thus you must plan ahead which processes might 2023-12-14T06:30:53,700 be writing to the dict and therefore creating full dumps. 2023-12-14T06:30:53,700 `auto_unlink`: If True, the creator of the shared memory will automatically unlink the handle at exit so 2023-12-14T06:30:53,700 it is not visible or accessible to new processes. All existing, still connected processes can continue to use the 2023-12-14T06:30:53,700 dict. 2023-12-14T06:30:53,700 `recurse`: If True, any nested dict objects will be automaticall wrapped in an `i18n_json` allowing transparent nested updates. 2023-12-14T06:30:53,700 `recurse_register`: Has to be either the `name` of an i18n_json or an i18n_json instance itself. Will be used internally to keep track of dynamically created, recursive jsondb_in_memorys for proper cleanup when using `recurse=True`. Usually does not have to be set by the user. 2023-12-14T06:30:53,701 ## Memory management 2023-12-14T06:30:53,701 `i18n_json` uses shared memory buffers and those usually live is RAM. `i18n_json` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them. 2023-12-14T06:30:53,701 By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `i18n_json`. The first process 2023-12-14T06:30:53,701 that is creating a certain `i18n_json` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`. 2023-12-14T06:30:53,701 When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers. 2023-12-14T06:30:53,702 A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `i18n_json` to keep 2023-12-14T06:30:53,702 track of recursively nested `i18n_json` instances. All child `jsondb_in_memorys` will write to this register the names of the shared memory buffers 2023-12-14T06:30:53,702 they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program. 2023-12-14T06:30:53,702 **Buffer sizes and read performance:** 2023-12-14T06:30:53,702 There are 3 cases that can occur when you read from an `i18n_json: 2023-12-14T06:30:53,702 1. No new updates: This is the fastes cases. `i18n_json` was optimized for this case to find out as quickly as possible if there are no updates on the stream and then just return the desired data. If you want even better read perforamance you can directly access the underlying `data` attribute of your `i18n_json`, though at the cost of not getting real time updates anymore. 2023-12-14T06:30:53,703 2. Streaming update: This is usually fast, depending on the size and amount of that data that was changed but not depending on the size of the whole `i18n_json`. Only the data that was actually changed has to be unserialized. 2023-12-14T06:30:53,703 3. Full dump load: This can be slow, depending on the total size of your data. If your `i18n_json` is big it might take long to unserialize it. 2023-12-14T06:30:53,703 Given the above 3 cases, you need to balance the size of your data and your write patterns with the streaming `buffer_size` of your i18n_json. If the streaming buffer is full, a full dump has to be created. Thus, if your full dumps are expensive due to their size, try to find a good `buffer_size` to avoid creating too many full dumps. 2023-12-14T06:30:53,703 On the other hand, if for example you only change back and forth the value of one single key in your `i18n_json`, it might be useless to process a stream of all these back and forth changes. It might be much more efficient to simply do one full dump which might be very small because it only contains one key. 2023-12-14T06:30:53,704 ## Locking 2023-12-14T06:30:53,704 Every i18n_json instance has a `lock` attribute which is either a [multiprocessing.RLock](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock) or an `i18n_json.SharedLock` if you set `shared_lock=True` when creating the i18n_json. 2023-12-14T06:30:53,704 RLock is the fastest locking method that is used by default but you can only use it if you fork your child processes. Forking is the default on Linux systems. 2023-12-14T06:30:53,704 In contrast, on Windows systems, forking is not available and Python will automatically use the spawn method when creating child processes. You should then use the parameter `shared_lock=True` when using i18n_json. This requires that the external [atomics](https://github.com/doodspav/atomics) package is installed. 2023-12-14T06:30:53,704 ### How to use the locking? 2023-12-14T06:30:53,704 ```python 2023-12-14T06:30:53,705 ultra = i18n_json(shared_lock=True) 2023-12-14T06:30:53,705 with ultra.lock: 2023-12-14T06:30:53,705 ultra['counter']++ 2023-12-14T06:30:53,705 # The same as above with all default parameters 2023-12-14T06:30:53,705 with ultra.lock(timeout=None, block=True, steal=False, sleep_time=0.000001): 2023-12-14T06:30:53,705 ultra['counter']++ 2023-12-14T06:30:53,706 # Busy wait, will result in 99 % CPU usage, fastest option 2023-12-14T06:30:53,706 # Ideally number of processes using the i18n_json should be < number of CPUs 2023-12-14T06:30:53,706 with ultra.lock(sleep_time=0): 2023-12-14T06:30:53,706 ultra['counter']++ 2023-12-14T06:30:53,706 try: 2023-12-14T06:30:53,706 result = ultra.lock.acquire(block=False) 2023-12-14T06:30:53,706 ultra.lock.release() 2023-12-14T06:30:53,706 except i18n_json.Exceptions.CannotAcquireLock as e: 2023-12-14T06:30:53,706 print(f'Process with PID {e.blocking_pid} is holding the lock') 2023-12-14T06:30:53,707 try: 2023-12-14T06:30:53,707 with ultra.lock(timeout=1.5): 2023-12-14T06:30:53,707 ultra['counter']++ 2023-12-14T06:30:53,707 except i18n_json.Exceptions.CannotAcquireLockTimeout: 2023-12-14T06:30:53,707 print('Stale lock?') 2023-12-14T06:30:53,707 with ultra.lock(timeout=1.5, steal_after_timeout=True): 2023-12-14T06:30:53,707 ultra['counter']++ 2023-12-14T06:30:53,708 ``` 2023-12-14T06:30:53,708 ## Explicit cleanup 2023-12-14T06:30:53,708 Sometimes, when your program crashes, no cleanup happens and you might have a corrupted shared memeory buffer that only goes away if you manually delete it. 2023-12-14T06:30:53,708 On Linux/Unix systems, those buffers usually live in a memory based filesystem in the folder `/dev/shm`. You can simply delete the files there. 2023-12-14T06:30:53,708 Another way to do this in code is like this: 2023-12-14T06:30:53,709 ```python 2023-12-14T06:30:53,709 # Unlink both shared memory buffers possibly used by i18n_json 2023-12-14T06:30:53,709 name = 'my-dict-name' 2023-12-14T06:30:53,709 i18n_json.unlink_by_name(name, ignore_errors=True) 2023-12-14T06:30:53,709 i18n_json.unlink_by_name(f'{name}_memory', ignore_errors=True) 2023-12-14T06:30:53,709 ``` 2023-12-14T06:30:53,709 ## Advanced usage 2023-12-14T06:30:53,710 See [examples](/examples) folder 2023-12-14T06:30:53,710 ```python 2023-12-14T06:30:53,710 >>> ultra = i18n_json({ 'init': 'some initial data' }, name='my-name', buffer_size=100_000) 2023-12-14T06:30:53,710 >>> # Let's use a value with 100k bytes length. 2023-12-14T06:30:53,710 >>> # This will not fit into our 100k bytes buffer due to the serialization overhead. 2023-12-14T06:30:53,710 >>> ultra[0] = ' ' * 100_000 2023-12-14T06:30:53,710 >>> ultra.print_status() 2023-12-14T06:30:53,710 {'buffer': SharedMemory('my-name_memory', size=100000), 2023-12-14T06:30:53,711 'buffer_size': 100000, 2023-12-14T06:30:53,711 'control': SharedMemory('my-name', size=1000), 2023-12-14T06:30:53,711 'full_dump_counter': 1, 2023-12-14T06:30:53,711 'full_dump_counter_remote': 1, 2023-12-14T06:30:53,711 'full_dump_memory': SharedMemory('psm_765691cd', size=100057), 2023-12-14T06:30:53,711 'full_dump_memory_name_remote': 'psm_765691cd', 2023-12-14T06:30:53,711 'full_dump_size': None, 2023-12-14T06:30:53,711 'full_dump_static_size_remote': , 2023-12-14T06:30:53,711 'lock': , 2023-12-14T06:30:53,711 'lock_pid_remote': 0, 2023-12-14T06:30:53,712 'lock_remote': 0, 2023-12-14T06:30:53,712 'name': 'my-name', 2023-12-14T06:30:53,712 'recurse': False, 2023-12-14T06:30:53,712 'recurse_remote': , 2023-12-14T06:30:53,712 'serializer': , 2023-12-14T06:30:53,712 'shared_lock_remote': , 2023-12-14T06:30:53,712 'update_stream_position': 0, 2023-12-14T06:30:53,712 'update_stream_position_remote': 0} 2023-12-14T06:30:53,712 ``` 2023-12-14T06:30:53,713 Note: All status keys ending with `_remote` are stored in the control shared memory space and shared across processes. 2023-12-14T06:30:53,713 Other things you can do: 2023-12-14T06:30:53,713 ```python 2023-12-14T06:30:53,713 >>> # Create a full dump 2023-12-14T06:30:53,713 >>> ultra.dump() 2023-12-14T06:30:53,713 >>> # Load latest full dump if one is available 2023-12-14T06:30:53,714 >>> ultra.load() 2023-12-14T06:30:53,714 >>> # Show statistics 2023-12-14T06:30:53,714 >>> ultra.print_status() 2023-12-14T06:30:53,714 >>> # Force load of latest full dump, even if we had already processed it. 2023-12-14T06:30:53,714 >>> # There might also be streaming updates available after loading the full dump. 2023-12-14T06:30:53,714 >>> ultra.load(force=True) 2023-12-14T06:30:53,715 >>> # Apply full dump and stream updates to 2023-12-14T06:30:53,715 >>> # underlying local dict, this is automatically 2023-12-14T06:30:53,715 >>> # called by accessing the i18n_json in any usual way, 2023-12-14T06:30:53,715 >>> # but can be useful to call after a forced load. 2023-12-14T06:30:53,715 >>> ultra.apply_update() 2023-12-14T06:30:53,715 >>> # Access underlying local dict directly for maximum performance 2023-12-14T06:30:53,715 >>> ultra.data 2023-12-14T06:30:53,715 >>> # Use any serializer you like, given it supports the loads() and dumps() methods 2023-12-14T06:30:53,716 >>> import jsons 2023-12-14T06:30:53,716 >>> ultra = i18n_json(serializer=jsons) 2023-12-14T06:30:53,716 >>> # Close connection to shared memory; will return the data as a dict 2023-12-14T06:30:53,716 >>> ultra.close() 2023-12-14T06:30:53,716 >>> # Unlink all shared memory, it will not be visible to new processes afterwards 2023-12-14T06:30:53,716 >>> ultra.unlink() 2023-12-14T06:30:53,717 ``` 2023-12-14T06:30:53,717 ## Contributing 2023-12-14T06:30:53,717 Contributions are always welcome! 2023-12-14T06:30:54,317 running bdist_wheel 2023-12-14T06:30:54,364 running build 2023-12-14T06:30:54,364 running build_py 2023-12-14T06:30:54,375 creating build 2023-12-14T06:30:54,375 creating build/lib.linux-armv7l-cpython-39 2023-12-14T06:30:54,376 creating build/lib.linux-armv7l-cpython-39/i18n_json 2023-12-14T06:30:54,377 copying ./setup.py -> build/lib.linux-armv7l-cpython-39/i18n_json 2023-12-14T06:30:54,379 copying ./__init__.py -> build/lib.linux-armv7l-cpython-39/i18n_json 2023-12-14T06:30:54,380 copying ./Exceptions.py -> build/lib.linux-armv7l-cpython-39/i18n_json 2023-12-14T06:30:54,382 copying ./i18n_json.py -> build/lib.linux-armv7l-cpython-39/i18n_json 2023-12-14T06:30:54,385 running egg_info 2023-12-14T06:30:54,391 writing i18n_json.egg-info/PKG-INFO 2023-12-14T06:30:54,395 writing dependency_links to i18n_json.egg-info/dependency_links.txt 2023-12-14T06:30:54,397 writing top-level names to i18n_json.egg-info/top_level.txt 2023-12-14T06:30:54,411 reading manifest file 'i18n_json.egg-info/SOURCES.txt' 2023-12-14T06:30:54,413 reading manifest template 'MANIFEST.in' 2023-12-14T06:30:54,414 adding license file 'LICENSE' 2023-12-14T06:30:54,416 writing manifest file 'i18n_json.egg-info/SOURCES.txt' 2023-12-14T06:30:54,418 running build_ext 2023-12-14T06:30:54,426 building 'i18n_json' extension 2023-12-14T06:30:54,426 creating build/temp.linux-armv7l-cpython-39 2023-12-14T06:30:54,427 arm-linux-gnueabihf-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.9 -c i18n_json.c -o build/temp.linux-armv7l-cpython-39/i18n_json.o 2023-12-14T06:32:05,360 arm-linux-gnueabihf-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-armv7l-cpython-39/i18n_json.o -L/usr/lib -o build/lib.linux-armv7l-cpython-39/i18n_json.cpython-39-arm-linux-gnueabihf.so 2023-12-14T06:32:05,591 installing to build/bdist.linux-armv7l/wheel 2023-12-14T06:32:05,592 running install 2023-12-14T06:32:05,617 running install_lib 2023-12-14T06:32:05,627 creating build/bdist.linux-armv7l 2023-12-14T06:32:05,627 creating build/bdist.linux-armv7l/wheel 2023-12-14T06:32:05,629 creating build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T06:32:05,629 copying build/lib.linux-armv7l-cpython-39/i18n_json/setup.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T06:32:05,631 copying build/lib.linux-armv7l-cpython-39/i18n_json/__init__.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T06:32:05,633 copying build/lib.linux-armv7l-cpython-39/i18n_json/Exceptions.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T06:32:05,634 copying build/lib.linux-armv7l-cpython-39/i18n_json/i18n_json.py -> build/bdist.linux-armv7l/wheel/i18n_json 2023-12-14T06:32:05,637 copying build/lib.linux-armv7l-cpython-39/i18n_json.cpython-39-arm-linux-gnueabihf.so -> build/bdist.linux-armv7l/wheel 2023-12-14T06:32:05,671 running install_egg_info 2023-12-14T06:32:05,685 Copying i18n_json.egg-info to build/bdist.linux-armv7l/wheel/i18n_json-0.0.9-py3.9.egg-info 2023-12-14T06:32:05,696 running install_scripts 2023-12-14T06:32:05,740 creating build/bdist.linux-armv7l/wheel/i18n_json-0.0.9.dist-info/WHEEL 2023-12-14T06:32:05,743 creating '/tmp/pip-wheel-vqfsb_9d/.tmp-8r8ytf_g/i18n_json-0.0.9-cp39-cp39-linux_armv7l.whl' and adding 'build/bdist.linux-armv7l/wheel' to it 2023-12-14T06:32:06,179 adding 'i18n_json.cpython-39-arm-linux-gnueabihf.so' 2023-12-14T06:32:06,193 adding 'i18n_json/Exceptions.py' 2023-12-14T06:32:06,194 adding 'i18n_json/__init__.py' 2023-12-14T06:32:06,199 adding 'i18n_json/i18n_json.py' 2023-12-14T06:32:06,201 adding 'i18n_json/setup.py' 2023-12-14T06:32:06,204 adding 'i18n_json-0.0.9.dist-info/LICENSE' 2023-12-14T06:32:06,207 adding 'i18n_json-0.0.9.dist-info/METADATA' 2023-12-14T06:32:06,208 adding 'i18n_json-0.0.9.dist-info/WHEEL' 2023-12-14T06:32:06,208 adding 'i18n_json-0.0.9.dist-info/top_level.txt' 2023-12-14T06:32:06,209 adding 'i18n_json-0.0.9.dist-info/RECORD' 2023-12-14T06:32:06,219 removing build/bdist.linux-armv7l/wheel 2023-12-14T06:32:06,345 Building wheel for i18n-json (pyproject.toml): finished with status 'done' 2023-12-14T06:32:06,365 Created wheel for i18n-json: filename=i18n_json-0.0.9-cp39-cp39-linux_armv7l.whl size=768513 sha256=cda4d5f6a603da85af1be996591d954dccf7f364577c6779200e84f4ca975e64 2023-12-14T06:32:06,366 Stored in directory: /tmp/pip-ephem-wheel-cache-et6pc6z5/wheels/61/da/d0/3cf1d1c47f8b501af39553d52c8062d39ba9dfb483b2dc59c9 2023-12-14T06:32:06,381 Successfully built i18n-json 2023-12-14T06:32:06,405 Removed build tracker: '/tmp/pip-build-tracker-wm5p1b0q'