2024-01-03T16:18:27,708 Created temporary directory: /tmp/pip-build-tracker-pk8v8s_l 2024-01-03T16:18:27,709 Initialized build tracking at /tmp/pip-build-tracker-pk8v8s_l 2024-01-03T16:18:27,709 Created build tracker: /tmp/pip-build-tracker-pk8v8s_l 2024-01-03T16:18:27,710 Entered build tracker: /tmp/pip-build-tracker-pk8v8s_l 2024-01-03T16:18:27,710 Created temporary directory: /tmp/pip-wheel-cum10hlh 2024-01-03T16:18:27,714 Created temporary directory: /tmp/pip-ephem-wheel-cache-jbj6cmjq 2024-01-03T16:18:27,738 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2024-01-03T16:18:27,741 2 location(s) to search for versions of lm-eval: 2024-01-03T16:18:27,741 * https://pypi.org/simple/lm-eval/ 2024-01-03T16:18:27,741 * https://www.piwheels.org/simple/lm-eval/ 2024-01-03T16:18:27,742 Fetching project page and analyzing links: https://pypi.org/simple/lm-eval/ 2024-01-03T16:18:27,743 Getting page https://pypi.org/simple/lm-eval/ 2024-01-03T16:18:27,744 Found index url https://pypi.org/simple/ 2024-01-03T16:18:27,952 Fetched page https://pypi.org/simple/lm-eval/ as application/vnd.pypi.simple.v1+json 2024-01-03T16:18:27,954 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/1b/5f/7841febb99c12ffb453d33a67b9841e89dba18c388b644bf22b81d137fc4/lm_eval-0.0.1-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:27,955 Found link https://files.pythonhosted.org/packages/21/5a/feb5ff3a1591ca963c54873d39116b0e6a4f80e493e961ac08569709c5d7/lm_eval-0.0.1.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.0.1 2024-01-03T16:18:27,955 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/f3/a7/63cbce8b51de25fabb1c49f3a3fd1704faaacadb5ed816401f800e4d2dbd/lm_eval-0.2.0-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:27,956 Found link https://files.pythonhosted.org/packages/c5/fd/edd21b0f258b4ec0260f99f5b2ac3864f7cddc8fb7c83bbb2379a6aab975/lm_eval-0.2.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.2.0 2024-01-03T16:18:27,957 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/61/c5/bff92e6b61fc2b0c1b7ac769633731910152e5176a404912ce7c07329ba0/lm_eval-0.3.0-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:27,958 Found link https://files.pythonhosted.org/packages/c4/f8/58abc65390a758c8c2e5f1d8bb9b58d7885d02535d5f48de27006453d07e/lm_eval-0.3.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.3.0 2024-01-03T16:18:27,959 Found link https://files.pythonhosted.org/packages/45/e0/05001c2e56e2f8f793189442176432ddb89a2f1f1ef00ea154d0bc00fe37/lm_eval-0.4.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8), version: 0.4.0 2024-01-03T16:18:27,960 Fetching project page and analyzing links: https://www.piwheels.org/simple/lm-eval/ 2024-01-03T16:18:27,961 Getting page https://www.piwheels.org/simple/lm-eval/ 2024-01-03T16:18:27,962 Found index url https://www.piwheels.org/simple/ 2024-01-03T16:18:28,131 Fetched page https://www.piwheels.org/simple/lm-eval/ as text/html 2024-01-03T16:18:28,133 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.3.0-py3-none-any.whl#sha256=498b8b8954c1f9c17f46e3ec096e9be6b9c96ee70560ee613a4eb9c7b9d31644 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:28,134 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.2.0-py3-none-any.whl#sha256=e06d3d7b6016be832e6889cbc9c4787b99156ba57f8feb31d3aeb27304c6558c (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:28,134 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.0.1-py3-none-any.whl#sha256=0afc289f69286f71017fb9811dfea6cda7c703bf693ad43106fbbff1f164cf14 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-01-03T16:18:28,135 Skipping link: not a file: https://www.piwheels.org/simple/lm-eval/ 2024-01-03T16:18:28,136 Skipping link: not a file: https://pypi.org/simple/lm-eval/ 2024-01-03T16:18:28,154 Given no hashes to check 1 links for project 'lm-eval': discarding no candidates 2024-01-03T16:18:28,171 Collecting lm-eval==0.4.0 2024-01-03T16:18:28,174 Created temporary directory: /tmp/pip-unpack-vbpag5mo 2024-01-03T16:18:28,307 Downloading lm_eval-0.4.0.tar.gz (457 kB) 2024-01-03T16:18:30,855 Added lm-eval==0.4.0 from https://files.pythonhosted.org/packages/45/e0/05001c2e56e2f8f793189442176432ddb89a2f1f1ef00ea154d0bc00fe37/lm_eval-0.4.0.tar.gz to build tracker '/tmp/pip-build-tracker-pk8v8s_l' 2024-01-03T16:18:30,861 Created temporary directory: /tmp/pip-build-env-tb9wzck6 2024-01-03T16:18:30,866 Installing build dependencies: started 2024-01-03T16:18:30,867 Running command pip subprocess to install build dependencies 2024-01-03T16:18:32,247 Using pip 23.3.1 from /usr/local/lib/python3.11/dist-packages/pip (python 3.11) 2024-01-03T16:18:32,735 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2024-01-03T16:18:34,173 Collecting setuptools>=40.8.0 2024-01-03T16:18:34,174 Obtaining dependency information for setuptools>=40.8.0 from https://files.pythonhosted.org/packages/55/3a/5121b58b578a598b269537e09a316ad2a94fdd561a2c6eb75cd68578cc6b/setuptools-69.0.3-py3-none-any.whl.metadata 2024-01-03T16:18:34,181 Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB) 2024-01-03T16:18:34,380 Collecting wheel 2024-01-03T16:18:34,394 Using cached https://www.piwheels.org/simple/wheel/wheel-0.42.0-py3-none-any.whl (65 kB) 2024-01-03T16:18:34,558 Using cached setuptools-69.0.3-py3-none-any.whl (819 kB) 2024-01-03T16:18:37,000 Installing collected packages: wheel, setuptools 2024-01-03T16:18:37,225 Creating /tmp/pip-build-env-tb9wzck6/overlay/local/bin 2024-01-03T16:18:37,228 changing mode of /tmp/pip-build-env-tb9wzck6/overlay/local/bin/wheel to 755 2024-01-03T16:18:39,519 Successfully installed setuptools-69.0.3 wheel-0.42.0 2024-01-03T16:18:39,773 [notice] A new release of pip is available: 23.3.1 -> 23.3.2 2024-01-03T16:18:39,774 [notice] To update, run: python3 -m pip install --upgrade pip 2024-01-03T16:18:40,040 Installing build dependencies: finished with status 'done' 2024-01-03T16:18:40,044 Getting requirements to build wheel: started 2024-01-03T16:18:40,045 Running command Getting requirements to build wheel 2024-01-03T16:18:40,919 running egg_info 2024-01-03T16:18:40,923 writing lm_eval.egg-info/PKG-INFO 2024-01-03T16:18:40,939 writing dependency_links to lm_eval.egg-info/dependency_links.txt 2024-01-03T16:18:40,941 writing entry points to lm_eval.egg-info/entry_points.txt 2024-01-03T16:18:40,949 writing requirements to lm_eval.egg-info/requires.txt 2024-01-03T16:18:40,950 writing top-level names to lm_eval.egg-info/top_level.txt 2024-01-03T16:18:41,339 reading manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:41,377 adding license file 'LICENSE.md' 2024-01-03T16:18:41,455 writing manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:41,575 Getting requirements to build wheel: finished with status 'done' 2024-01-03T16:18:41,585 Created temporary directory: /tmp/pip-modern-metadata-lsndfh8e 2024-01-03T16:18:41,587 Preparing metadata (pyproject.toml): started 2024-01-03T16:18:41,589 Running command Preparing metadata (pyproject.toml) 2024-01-03T16:18:42,380 running dist_info 2024-01-03T16:18:42,385 creating /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info 2024-01-03T16:18:42,389 writing /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/PKG-INFO 2024-01-03T16:18:42,405 writing dependency_links to /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/dependency_links.txt 2024-01-03T16:18:42,407 writing entry points to /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/entry_points.txt 2024-01-03T16:18:42,415 writing requirements to /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/requires.txt 2024-01-03T16:18:42,416 writing top-level names to /tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/top_level.txt 2024-01-03T16:18:42,418 writing manifest file '/tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:42,773 reading manifest file '/tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:42,776 adding license file 'LICENSE.md' 2024-01-03T16:18:42,830 writing manifest file '/tmp/pip-modern-metadata-lsndfh8e/lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:42,833 creating '/tmp/pip-modern-metadata-lsndfh8e/lm_eval-0.4.0.dist-info' 2024-01-03T16:18:42,997 Preparing metadata (pyproject.toml): finished with status 'done' 2024-01-03T16:18:43,003 Source in /tmp/pip-wheel-cum10hlh/lm-eval_a80c5fae40094c51aa88ec63e668f7bf has version 0.4.0, which satisfies requirement lm-eval==0.4.0 from https://files.pythonhosted.org/packages/45/e0/05001c2e56e2f8f793189442176432ddb89a2f1f1ef00ea154d0bc00fe37/lm_eval-0.4.0.tar.gz 2024-01-03T16:18:43,004 Removed lm-eval==0.4.0 from https://files.pythonhosted.org/packages/45/e0/05001c2e56e2f8f793189442176432ddb89a2f1f1ef00ea154d0bc00fe37/lm_eval-0.4.0.tar.gz from build tracker '/tmp/pip-build-tracker-pk8v8s_l' 2024-01-03T16:18:43,013 Created temporary directory: /tmp/pip-unpack-q9h0dfhq 2024-01-03T16:18:43,014 Created temporary directory: /tmp/pip-unpack-rh_30s7e 2024-01-03T16:18:43,131 Building wheels for collected packages: lm-eval 2024-01-03T16:18:43,136 Created temporary directory: /tmp/pip-wheel-bzbbg3lh 2024-01-03T16:18:43,136 Destination directory: /tmp/pip-wheel-bzbbg3lh 2024-01-03T16:18:43,138 Building wheel for lm-eval (pyproject.toml): started 2024-01-03T16:18:43,140 Running command Building wheel for lm-eval (pyproject.toml) 2024-01-03T16:18:43,939 running bdist_wheel 2024-01-03T16:18:43,955 running build 2024-01-03T16:18:43,955 running build_py 2024-01-03T16:18:43,960 creating build 2024-01-03T16:18:43,960 creating build/lib 2024-01-03T16:18:43,961 creating build/lib/lm_eval 2024-01-03T16:18:43,963 copying lm_eval/__main__.py -> build/lib/lm_eval 2024-01-03T16:18:43,967 copying lm_eval/evaluator.py -> build/lib/lm_eval 2024-01-03T16:18:43,970 copying lm_eval/__init__.py -> build/lib/lm_eval 2024-01-03T16:18:43,972 copying lm_eval/utils.py -> build/lib/lm_eval 2024-01-03T16:18:43,976 creating build/lib/lm_eval/prompts 2024-01-03T16:18:43,978 copying lm_eval/prompts/__init__.py -> build/lib/lm_eval/prompts 2024-01-03T16:18:43,985 creating build/lib/lm_eval/decontamination 2024-01-03T16:18:43,988 copying lm_eval/decontamination/decontaminate.py -> build/lib/lm_eval/decontamination 2024-01-03T16:18:43,995 copying lm_eval/decontamination/__init__.py -> build/lib/lm_eval/decontamination 2024-01-03T16:18:43,997 copying lm_eval/decontamination/archiver.py -> build/lib/lm_eval/decontamination 2024-01-03T16:18:44,001 copying lm_eval/decontamination/janitor.py -> build/lib/lm_eval/decontamination 2024-01-03T16:18:44,005 creating build/lib/lm_eval/models 2024-01-03T16:18:44,006 copying lm_eval/models/openai_completions.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,010 copying lm_eval/models/anthropic_llms.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,014 copying lm_eval/models/textsynth.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,017 copying lm_eval/models/__init__.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,019 copying lm_eval/models/gguf.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,021 copying lm_eval/models/huggingface.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,025 copying lm_eval/models/dummy.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,028 copying lm_eval/models/vllm_causallms.py -> build/lib/lm_eval/models 2024-01-03T16:18:44,032 creating build/lib/lm_eval/tasks 2024-01-03T16:18:44,034 copying lm_eval/tasks/__init__.py -> build/lib/lm_eval/tasks 2024-01-03T16:18:44,037 creating build/lib/lm_eval/api 2024-01-03T16:18:44,039 copying lm_eval/api/instance.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,042 copying lm_eval/api/metrics.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,045 copying lm_eval/api/model.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,048 copying lm_eval/api/registry.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,050 copying lm_eval/api/samplers.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,052 copying lm_eval/api/task.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,055 copying lm_eval/api/filter.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,057 copying lm_eval/api/__init__.py -> build/lib/lm_eval/api 2024-01-03T16:18:44,059 creating build/lib/lm_eval/filters 2024-01-03T16:18:44,060 copying lm_eval/filters/extraction.py -> build/lib/lm_eval/filters 2024-01-03T16:18:44,063 copying lm_eval/filters/transformation.py -> build/lib/lm_eval/filters 2024-01-03T16:18:44,065 copying lm_eval/filters/__init__.py -> build/lib/lm_eval/filters 2024-01-03T16:18:44,067 copying lm_eval/filters/selection.py -> build/lib/lm_eval/filters 2024-01-03T16:18:44,069 copying lm_eval/filters/decontamination.py -> build/lib/lm_eval/filters 2024-01-03T16:18:44,071 creating build/lib/lm_eval/tasks/toxigen 2024-01-03T16:18:44,072 copying lm_eval/tasks/toxigen/utils.py -> build/lib/lm_eval/tasks/toxigen 2024-01-03T16:18:44,074 creating build/lib/lm_eval/tasks/hellaswag 2024-01-03T16:18:44,076 copying lm_eval/tasks/hellaswag/utils.py -> build/lib/lm_eval/tasks/hellaswag 2024-01-03T16:18:44,078 creating build/lib/lm_eval/tasks/squadv2 2024-01-03T16:18:44,079 copying lm_eval/tasks/squadv2/task.py -> build/lib/lm_eval/tasks/squadv2 2024-01-03T16:18:44,083 creating build/lib/lm_eval/tasks/bbh 2024-01-03T16:18:44,085 copying lm_eval/tasks/bbh/_generate_configs.py -> build/lib/lm_eval/tasks/bbh 2024-01-03T16:18:44,089 creating build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:44,090 copying lm_eval/tasks/belebele/_generate_configs.py -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:44,093 creating build/lib/lm_eval/tasks/webqs 2024-01-03T16:18:44,094 copying lm_eval/tasks/webqs/utils.py -> build/lib/lm_eval/tasks/webqs 2024-01-03T16:18:44,096 creating build/lib/lm_eval/tasks/translation 2024-01-03T16:18:44,098 copying lm_eval/tasks/translation/utils.py -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:44,315 creating build/lib/lm_eval/tasks/race 2024-01-03T16:18:44,316 copying lm_eval/tasks/race/preprocess_race.py -> build/lib/lm_eval/tasks/race 2024-01-03T16:18:44,319 creating build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:44,320 copying lm_eval/tasks/crows_pairs/utils.py -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:44,324 creating build/lib/lm_eval/tasks/wmt2016 2024-01-03T16:18:44,325 copying lm_eval/tasks/wmt2016/metrics.py -> build/lib/lm_eval/tasks/wmt2016 2024-01-03T16:18:44,327 creating build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:44,328 copying lm_eval/tasks/ceval/_generate_configs.py -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:44,331 creating build/lib/lm_eval/tasks/scrolls 2024-01-03T16:18:44,332 copying lm_eval/tasks/scrolls/task.py -> build/lib/lm_eval/tasks/scrolls 2024-01-03T16:18:44,335 creating build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:44,336 copying lm_eval/tasks/blimp/generate_configs.py -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:44,339 creating build/lib/lm_eval/tasks/mutual 2024-01-03T16:18:44,340 copying lm_eval/tasks/mutual/utils.py -> build/lib/lm_eval/tasks/mutual 2024-01-03T16:18:44,343 creating build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:44,344 copying lm_eval/tasks/truthfulqa/utils.py -> build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:44,347 creating build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:44,349 copying lm_eval/tasks/xcopa/utils.py -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:44,351 creating build/lib/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:44,352 copying lm_eval/tasks/realtoxicityprompts/metric.py -> build/lib/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:44,355 creating build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:44,357 copying lm_eval/tasks/bigbench/push_bigbench_dataset.py -> build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:44,359 copying lm_eval/tasks/bigbench/generate_tasks.py -> build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:44,362 creating build/lib/lm_eval/tasks/drop 2024-01-03T16:18:44,363 copying lm_eval/tasks/drop/utils.py -> build/lib/lm_eval/tasks/drop 2024-01-03T16:18:44,366 creating build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:44,367 copying lm_eval/tasks/cmmlu/_generate_configs.py -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:44,370 creating build/lib/lm_eval/tasks/logiqa2 2024-01-03T16:18:44,371 copying lm_eval/tasks/logiqa2/utils_logiqa2.py -> build/lib/lm_eval/tasks/logiqa2 2024-01-03T16:18:44,374 creating build/lib/lm_eval/tasks/coqa 2024-01-03T16:18:44,375 copying lm_eval/tasks/coqa/utils.py -> build/lib/lm_eval/tasks/coqa 2024-01-03T16:18:44,378 creating build/lib/lm_eval/tasks/wikitext 2024-01-03T16:18:44,379 copying lm_eval/tasks/wikitext/preprocess_wikitext.py -> build/lib/lm_eval/tasks/wikitext 2024-01-03T16:18:44,382 creating build/lib/lm_eval/tasks/pubmedqa 2024-01-03T16:18:44,383 copying lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py -> build/lib/lm_eval/tasks/pubmedqa 2024-01-03T16:18:44,385 creating build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:44,387 copying lm_eval/tasks/paws-x/_generate_config.py -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:44,391 creating build/lib/lm_eval/tasks/mmlu 2024-01-03T16:18:44,392 copying lm_eval/tasks/mmlu/_generate_configs.py -> build/lib/lm_eval/tasks/mmlu 2024-01-03T16:18:44,395 creating build/lib/lm_eval/tasks/wsc273 2024-01-03T16:18:44,396 copying lm_eval/tasks/wsc273/utils.py -> build/lib/lm_eval/tasks/wsc273 2024-01-03T16:18:44,400 creating build/lib/lm_eval/tasks/mgsm 2024-01-03T16:18:44,401 copying lm_eval/tasks/mgsm/utils.py -> build/lib/lm_eval/tasks/mgsm 2024-01-03T16:18:44,405 creating build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:44,406 copying lm_eval/tasks/hendrycks_ethics/utils.py -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:44,409 creating build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:44,410 copying lm_eval/tasks/xwinograd/utils.py -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:44,413 creating build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:44,415 copying lm_eval/tasks/minerva_math/utils.py -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:44,417 creating build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:44,419 copying lm_eval/tasks/csatqa/_generate_configs.py -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:44,421 copying lm_eval/tasks/csatqa/utils.py -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:44,423 creating build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:44,424 copying lm_eval/tasks/xnli/utils.py -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:44,429 creating build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:44,430 copying lm_eval/tasks/qasper/metrics.py -> build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:44,433 copying lm_eval/tasks/qasper/utils.py -> build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:44,436 creating build/lib/lm_eval/tasks/logiqa 2024-01-03T16:18:44,437 copying lm_eval/tasks/logiqa/utils_logiqa.py -> build/lib/lm_eval/tasks/logiqa 2024-01-03T16:18:44,439 creating build/lib/lm_eval/tasks/mathqa 2024-01-03T16:18:44,441 copying lm_eval/tasks/mathqa/utils.py -> build/lib/lm_eval/tasks/mathqa 2024-01-03T16:18:44,443 creating build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:44,445 copying lm_eval/tasks/qa4mre/preprocess_qa4mre.py -> build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:44,447 creating build/lib/lm_eval/tasks/winogrande 2024-01-03T16:18:44,449 copying lm_eval/tasks/winogrande/preprocess_winogrande.py -> build/lib/lm_eval/tasks/winogrande 2024-01-03T16:18:44,453 creating build/lib/lm_eval/tasks/model_written_evals 2024-01-03T16:18:44,454 creating build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:44,456 copying lm_eval/tasks/model_written_evals/persona/_generate_configs.py -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:44,459 creating build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:44,460 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:44,463 creating build/lib/lm_eval/tasks/super_glue 2024-01-03T16:18:44,464 creating build/lib/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:44,465 copying lm_eval/tasks/super_glue/cb/aggregate.py -> build/lib/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:44,467 copying lm_eval/tasks/super_glue/cb/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:44,470 creating build/lib/lm_eval/tasks/super_glue/record 2024-01-03T16:18:44,471 copying lm_eval/tasks/super_glue/record/util.py -> build/lib/lm_eval/tasks/super_glue/record 2024-01-03T16:18:44,473 copying lm_eval/tasks/super_glue/record/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/record 2024-01-03T16:18:44,476 creating build/lib/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:44,477 copying lm_eval/tasks/super_glue/wsc/preprocess_wsc.py -> build/lib/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:44,479 copying lm_eval/tasks/super_glue/wsc/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:44,481 creating build/lib/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:44,482 copying lm_eval/tasks/super_glue/copa/utils.py -> build/lib/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:44,484 creating build/lib/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:44,485 copying lm_eval/tasks/super_glue/multirc/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:44,493 creating build/lib/lm_eval/tasks/code_x_glue 2024-01-03T16:18:44,494 creating build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:44,495 copying lm_eval/tasks/code_x_glue/code-text/bleu.py -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:44,498 copying lm_eval/tasks/code_x_glue/code-text/utils.py -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:44,501 creating build/lib/lm_eval/tasks/glue 2024-01-03T16:18:44,502 creating build/lib/lm_eval/tasks/glue/mnli 2024-01-03T16:18:44,503 copying lm_eval/tasks/glue/mnli/utils.py -> build/lib/lm_eval/tasks/glue/mnli 2024-01-03T16:18:44,507 running egg_info 2024-01-03T16:18:44,511 writing lm_eval.egg-info/PKG-INFO 2024-01-03T16:18:44,526 writing dependency_links to lm_eval.egg-info/dependency_links.txt 2024-01-03T16:18:44,528 writing entry points to lm_eval.egg-info/entry_points.txt 2024-01-03T16:18:44,536 writing requirements to lm_eval.egg-info/requires.txt 2024-01-03T16:18:44,537 writing top-level names to lm_eval.egg-info/top_level.txt 2024-01-03T16:18:44,919 reading manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:44,957 adding license file 'LICENSE.md' 2024-01-03T16:18:45,034 writing manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-01-03T16:18:45,392 copying lm_eval/tasks/toxigen/toxigen.yaml -> build/lib/lm_eval/tasks/toxigen 2024-01-03T16:18:45,394 copying lm_eval/tasks/hellaswag/hellaswag.yaml -> build/lib/lm_eval/tasks/hellaswag 2024-01-03T16:18:45,396 creating build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,397 copying lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,399 copying lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,401 copying lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,403 copying lm_eval/tasks/bbh/zeroshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,405 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,407 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,409 copying lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,411 copying lm_eval/tasks/bbh/zeroshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,413 copying lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,415 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,417 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,419 copying lm_eval/tasks/bbh/zeroshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,421 copying lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,423 copying lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,425 copying lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,427 copying lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,429 copying lm_eval/tasks/bbh/zeroshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,433 copying lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,435 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,437 copying lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,439 copying lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,441 copying lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,443 copying lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,445 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,447 copying lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,449 copying lm_eval/tasks/bbh/zeroshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,451 copying lm_eval/tasks/bbh/zeroshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:45,453 creating build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,454 copying lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,456 copying lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,458 copying lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,460 copying lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,462 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,464 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,466 copying lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,468 copying lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,470 copying lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,472 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,474 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,476 copying lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,478 copying lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,480 copying lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,482 copying lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,484 copying lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,486 copying lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,488 copying lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,490 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,492 copying lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,494 copying lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,496 copying lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,498 copying lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,500 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,502 copying lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,504 copying lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,506 copying lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:45,508 creating build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,509 copying lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,512 copying lm_eval/tasks/bbh/fewshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,515 copying lm_eval/tasks/bbh/fewshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,517 copying lm_eval/tasks/bbh/fewshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,519 copying lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,521 copying lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,523 copying lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,525 copying lm_eval/tasks/bbh/fewshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,527 copying lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,529 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,531 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,533 copying lm_eval/tasks/bbh/fewshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,535 copying lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,538 copying lm_eval/tasks/bbh/fewshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,540 copying lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,542 copying lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,543 copying lm_eval/tasks/bbh/fewshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,546 copying lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,548 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,550 copying lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,553 copying lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,555 copying lm_eval/tasks/bbh/fewshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,557 copying lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,559 copying lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,561 copying lm_eval/tasks/bbh/fewshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,563 copying lm_eval/tasks/bbh/fewshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,565 copying lm_eval/tasks/bbh/fewshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:45,567 creating build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,568 copying lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,570 copying lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,572 copying lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,574 copying lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,576 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,579 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,581 copying lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,583 copying lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,585 copying lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,587 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,590 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,592 copying lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,595 copying lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,597 copying lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,600 copying lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,603 copying lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,605 copying lm_eval/tasks/bbh/cot_fewshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,607 copying lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,609 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,611 copying lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,613 copying lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,615 copying lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,617 copying lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,619 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,621 copying lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,623 copying lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,625 copying lm_eval/tasks/bbh/cot_fewshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:45,627 creating build/lib/lm_eval/tasks/mc_taco 2024-01-03T16:18:45,628 copying lm_eval/tasks/mc_taco/default.yaml -> build/lib/lm_eval/tasks/mc_taco 2024-01-03T16:18:45,630 creating build/lib/lm_eval/tasks/asdiv 2024-01-03T16:18:45,631 copying lm_eval/tasks/asdiv/default.yaml -> build/lib/lm_eval/tasks/asdiv 2024-01-03T16:18:45,633 copying lm_eval/tasks/belebele/belebele_bam_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,635 copying lm_eval/tasks/belebele/belebele_bod_Tibt.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,637 copying lm_eval/tasks/belebele/belebele_gaz_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,639 copying lm_eval/tasks/belebele/belebele_ilo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,640 copying lm_eval/tasks/belebele/belebele_tel_Telu.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,642 copying lm_eval/tasks/belebele/belebele_nya_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,644 copying lm_eval/tasks/belebele/belebele_mar_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,646 copying lm_eval/tasks/belebele/belebele_khm_Khmr.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,648 copying lm_eval/tasks/belebele/belebele_yor_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,650 copying lm_eval/tasks/belebele/belebele_kan_Knda.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,652 copying lm_eval/tasks/belebele/belebele_apc_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,653 copying lm_eval/tasks/belebele/belebele_als_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,655 copying lm_eval/tasks/belebele/belebele_hat_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,657 copying lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,659 copying lm_eval/tasks/belebele/belebele_urd_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,661 copying lm_eval/tasks/belebele/belebele_cat_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,663 copying lm_eval/tasks/belebele/belebele_heb_Hebr.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,665 copying lm_eval/tasks/belebele/belebele_nld_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,667 copying lm_eval/tasks/belebele/belebele_som_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,669 copying lm_eval/tasks/belebele/belebele_mri_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,671 copying lm_eval/tasks/belebele/belebele_nso_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,673 copying lm_eval/tasks/belebele/belebele_sin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,675 copying lm_eval/tasks/belebele/belebele_hun_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,677 copying lm_eval/tasks/belebele/belebele_ben_Beng.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,679 copying lm_eval/tasks/belebele/belebele_kor_Hang.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,681 copying lm_eval/tasks/belebele/belebele_ibo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,682 copying lm_eval/tasks/belebele/belebele_tir_Ethi.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,684 copying lm_eval/tasks/belebele/belebele_zho_Hans.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,686 copying lm_eval/tasks/belebele/belebele_pan_Guru.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,688 copying lm_eval/tasks/belebele/belebele_tam_Taml.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,690 copying lm_eval/tasks/belebele/belebele_pol_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,692 copying lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,693 copying lm_eval/tasks/belebele/belebele_por_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,695 copying lm_eval/tasks/belebele/belebele_xho_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,697 copying lm_eval/tasks/belebele/belebele_spa_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,699 copying lm_eval/tasks/belebele/belebele_ces_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,701 copying lm_eval/tasks/belebele/belebele_fra_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,703 copying lm_eval/tasks/belebele/belebele_dan_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,705 copying lm_eval/tasks/belebele/belebele_shn_Mymr.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,707 copying lm_eval/tasks/belebele/belebele_asm_Beng.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,708 copying lm_eval/tasks/belebele/belebele_zul_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,710 copying lm_eval/tasks/belebele/belebele_ory_Orya.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,712 copying lm_eval/tasks/belebele/belebele_tgl_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,714 copying lm_eval/tasks/belebele/belebele_hau_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,716 copying lm_eval/tasks/belebele/belebele_wol_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,718 copying lm_eval/tasks/belebele/belebele_luo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,720 copying lm_eval/tasks/belebele/belebele_acm_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,722 copying lm_eval/tasks/belebele/belebele_swe_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,724 copying lm_eval/tasks/belebele/belebele_eng_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,726 copying lm_eval/tasks/belebele/belebele_uzn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,728 copying lm_eval/tasks/belebele/belebele_slk_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,729 copying lm_eval/tasks/belebele/belebele_jav_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,731 copying lm_eval/tasks/belebele/belebele_mya_Mymr.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,733 copying lm_eval/tasks/belebele/belebele_snd_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,735 copying lm_eval/tasks/belebele/belebele_plt_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,737 copying lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,739 copying lm_eval/tasks/belebele/belebele_lit_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,741 copying lm_eval/tasks/belebele/belebele_ars_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,743 copying lm_eval/tasks/belebele/belebele_hin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,744 copying lm_eval/tasks/belebele/belebele_kin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,746 copying lm_eval/tasks/belebele/belebele_tsn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,748 copying lm_eval/tasks/belebele/belebele_sot_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,750 copying lm_eval/tasks/belebele/belebele_arb_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,751 copying lm_eval/tasks/belebele/belebele_fin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,753 copying lm_eval/tasks/belebele/belebele_npi_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,755 copying lm_eval/tasks/belebele/belebele_lug_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,757 copying lm_eval/tasks/belebele/belebele_tha_Thai.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,758 copying lm_eval/tasks/belebele/belebele_slv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,760 copying lm_eval/tasks/belebele/belebele_isl_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,762 copying lm_eval/tasks/belebele/belebele_urd_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,764 copying lm_eval/tasks/belebele/belebele_npi_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,766 copying lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,768 copying lm_eval/tasks/belebele/belebele_azj_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,770 copying lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,771 copying lm_eval/tasks/belebele/belebele_vie_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,773 copying lm_eval/tasks/belebele/belebele_ceb_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,775 copying lm_eval/tasks/belebele/belebele_zho_Hant.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,777 copying lm_eval/tasks/belebele/belebele_pes_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,779 copying lm_eval/tasks/belebele/belebele_war_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,781 copying lm_eval/tasks/belebele/belebele_kat_Geor.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,783 copying lm_eval/tasks/belebele/belebele_sun_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,785 copying lm_eval/tasks/belebele/belebele_eus_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,786 copying lm_eval/tasks/belebele/belebele_hrv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,788 copying lm_eval/tasks/belebele/belebele_arz_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,790 copying lm_eval/tasks/belebele/belebele_pbt_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,792 copying lm_eval/tasks/belebele/belebele_ckb_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,794 copying lm_eval/tasks/belebele/belebele_hin_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,796 copying lm_eval/tasks/belebele/belebele_ell_Grek.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,798 copying lm_eval/tasks/belebele/belebele_lin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,800 copying lm_eval/tasks/belebele/belebele_hye_Armn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,802 copying lm_eval/tasks/belebele/belebele_sna_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,804 copying lm_eval/tasks/belebele/belebele_zsm_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,806 copying lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,808 copying lm_eval/tasks/belebele/belebele_tso_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,809 copying lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,811 copying lm_eval/tasks/belebele/belebele_grn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,813 copying lm_eval/tasks/belebele/belebele_mlt_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,815 copying lm_eval/tasks/belebele/belebele_ron_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,816 copying lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,818 copying lm_eval/tasks/belebele/belebele_ben_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,820 copying lm_eval/tasks/belebele/belebele_ita_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,822 copying lm_eval/tasks/belebele/belebele_tur_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,824 copying lm_eval/tasks/belebele/belebele_sin_Sinh.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,825 copying lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,827 copying lm_eval/tasks/belebele/belebele_est_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,829 copying lm_eval/tasks/belebele/belebele_guj_Gujr.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,831 copying lm_eval/tasks/belebele/belebele_kac_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,833 copying lm_eval/tasks/belebele/belebele_nob_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,835 copying lm_eval/tasks/belebele/belebele_kea_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,837 copying lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,839 copying lm_eval/tasks/belebele/belebele_fuv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,841 copying lm_eval/tasks/belebele/belebele_ary_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,843 copying lm_eval/tasks/belebele/belebele_ind_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,845 copying lm_eval/tasks/belebele/belebele_lao_Laoo.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,847 copying lm_eval/tasks/belebele/belebele_swh_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,849 copying lm_eval/tasks/belebele/belebele_ssw_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,851 copying lm_eval/tasks/belebele/belebele_deu_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,853 copying lm_eval/tasks/belebele/belebele_amh_Ethi.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,855 copying lm_eval/tasks/belebele/belebele_lvs_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,857 copying lm_eval/tasks/belebele/belebele_mal_Mlym.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,858 copying lm_eval/tasks/belebele/belebele_arb_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,860 copying lm_eval/tasks/belebele/belebele_afr_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:45,862 creating build/lib/lm_eval/tasks/triviaqa 2024-01-03T16:18:45,863 copying lm_eval/tasks/triviaqa/default.yaml -> build/lib/lm_eval/tasks/triviaqa 2024-01-03T16:18:45,865 copying lm_eval/tasks/webqs/webqs.yaml -> build/lib/lm_eval/tasks/webqs 2024-01-03T16:18:45,867 copying lm_eval/tasks/translation/wmt14_fr-en.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,870 copying lm_eval/tasks/translation/iwslt2017_en-ar.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,872 copying lm_eval/tasks/translation/iwslt2017_ar-en.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,874 copying lm_eval/tasks/translation/wmt16_ro-en.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,876 copying lm_eval/tasks/translation/wmt16_en-ro.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,877 copying lm_eval/tasks/translation/wmt16_en-de.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,879 copying lm_eval/tasks/translation/wmt14_en-fr.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,881 copying lm_eval/tasks/translation/wmt16_de-en.yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:45,883 copying lm_eval/tasks/race/race.yaml -> build/lib/lm_eval/tasks/race 2024-01-03T16:18:45,885 creating build/lib/lm_eval/tasks/nq_open 2024-01-03T16:18:45,886 copying lm_eval/tasks/nq_open/nq_open.yaml -> build/lib/lm_eval/tasks/nq_open 2024-01-03T16:18:45,888 copying lm_eval/tasks/crows_pairs/crows_pairs_french.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,889 copying lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,891 copying lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,893 copying lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,895 copying lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,897 copying lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,899 copying lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,901 copying lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,904 copying lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,906 copying lm_eval/tasks/crows_pairs/crows_pairs_english.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,908 copying lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,911 copying lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,912 copying lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,915 copying lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,917 copying lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,919 copying lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,921 copying lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,922 copying lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,924 copying lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,927 copying lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,929 copying lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,931 copying lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:45,933 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,935 copying lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,937 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,939 copying lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,941 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,943 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,946 copying lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,948 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,950 copying lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,951 copying lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,953 copying lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,955 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,957 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,959 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,961 copying lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,963 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,965 copying lm_eval/tasks/model_written_evals/persona/neuroticism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,967 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,969 copying lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,971 copying lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,973 copying lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,975 copying lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,978 copying lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,979 copying lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,981 copying lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,983 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,985 copying lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,987 copying lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,989 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,991 copying lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,993 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,995 copying lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,997 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:45,999 copying lm_eval/tasks/model_written_evals/persona/psychopathy.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,001 copying lm_eval/tasks/model_written_evals/persona/has-disability.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,003 copying lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,005 copying lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,008 copying lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,009 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,012 copying lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,014 copying lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,016 copying lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,018 copying lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,020 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,022 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,024 copying lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,026 copying lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,028 copying lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,030 copying lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,032 copying lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,035 copying lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,037 copying lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,038 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,040 copying lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,042 copying lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,044 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,046 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,049 copying lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,051 copying lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,053 copying lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,055 copying lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,058 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,060 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,062 copying lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,065 copying lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,068 copying lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,070 copying lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,072 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,075 copying lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,077 copying lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,080 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,082 copying lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,084 copying lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,087 copying lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,089 copying lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,092 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,094 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,096 copying lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,098 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,100 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,103 copying lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,105 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,107 copying lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,109 copying lm_eval/tasks/model_written_evals/persona/agreeableness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,111 copying lm_eval/tasks/model_written_evals/persona/narcissism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,113 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,116 copying lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,118 copying lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,120 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,122 copying lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,125 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,127 copying lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,129 copying lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,131 copying lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,133 copying lm_eval/tasks/model_written_evals/persona/self-replication.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,136 copying lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,138 copying lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,140 copying lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,143 copying lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,145 copying lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,147 copying lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,149 copying lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,151 copying lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,153 copying lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,156 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,158 copying lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,160 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,163 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,165 copying lm_eval/tasks/model_written_evals/persona/extraversion.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,167 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,169 copying lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,171 copying lm_eval/tasks/model_written_evals/persona/risk-averse.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,173 copying lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,176 copying lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,178 copying lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,180 copying lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,182 copying lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,184 copying lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,186 copying lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,189 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,191 copying lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,193 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,195 copying lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,197 copying lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,200 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,202 copying lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,204 copying lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,206 copying lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,208 copying lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,210 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,212 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,215 copying lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,217 copying lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,219 copying lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,221 copying lm_eval/tasks/model_written_evals/persona/openness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:46,223 creating build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:46,225 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:46,227 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:46,229 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:46,231 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,233 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,236 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,238 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,240 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,242 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,244 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,247 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,249 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,251 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,252 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,254 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,256 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,257 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,259 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,261 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,263 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,264 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,266 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,268 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,270 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,271 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,273 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,275 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,277 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,279 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,281 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,283 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,285 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,287 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,289 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,291 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,293 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,295 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,296 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,298 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,300 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,302 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,304 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,305 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,307 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,309 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,311 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,313 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,314 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,316 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,318 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,320 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,322 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:46,324 creating build/lib/lm_eval/tasks/gsm8k 2024-01-03T16:18:46,325 copying lm_eval/tasks/gsm8k/gsm8k-cot.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-01-03T16:18:46,327 copying lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-01-03T16:18:46,329 copying lm_eval/tasks/gsm8k/gsm8k.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-01-03T16:18:46,331 copying lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml -> build/lib/lm_eval/tasks/wmt2016 2024-01-03T16:18:46,333 copying lm_eval/tasks/ceval/ceval-valid_sports_science.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,335 copying lm_eval/tasks/ceval/ceval-valid_marxism.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,337 copying lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,338 copying lm_eval/tasks/ceval/ceval-valid_law.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,340 copying lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,342 copying lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,344 copying lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,346 copying lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,347 copying lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,349 copying lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,351 copying lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,353 copying lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,355 copying lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,357 copying lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,359 copying lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,360 copying lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,362 copying lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,364 copying lm_eval/tasks/ceval/ceval-valid_physician.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,366 copying lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,368 copying lm_eval/tasks/ceval/ceval-valid_computer_network.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,369 copying lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,371 copying lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,373 copying lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,375 copying lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,377 copying lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,379 copying lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,381 copying lm_eval/tasks/ceval/ceval-valid_logic.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,383 copying lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,385 copying lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,387 copying lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,388 copying lm_eval/tasks/ceval/ceval-valid_college_economics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,390 copying lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,392 copying lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,395 copying lm_eval/tasks/ceval/ceval-valid_accountant.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,396 copying lm_eval/tasks/ceval/ceval-valid_operating_system.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,398 copying lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,400 copying lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,401 copying lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,403 copying lm_eval/tasks/ceval/ceval-valid_college_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,405 copying lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,407 copying lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,408 copying lm_eval/tasks/ceval/ceval-valid_business_administration.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,410 copying lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,412 copying lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,413 copying lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,415 copying lm_eval/tasks/ceval/ceval-valid_art_studies.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,417 copying lm_eval/tasks/ceval/ceval-valid_college_programming.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,419 copying lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,421 copying lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,422 copying lm_eval/tasks/ceval/ceval-valid_education_science.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,424 copying lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,426 copying lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:46,428 copying lm_eval/tasks/scrolls/scrolls.yaml -> build/lib/lm_eval/tasks/scrolls 2024-01-03T16:18:46,430 copying lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,431 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,433 copying lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,435 copying lm_eval/tasks/blimp/complex_NP_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,437 copying lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,439 copying lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,441 copying lm_eval/tasks/blimp/only_npi_scope.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,443 copying lm_eval/tasks/blimp/passive_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,445 copying lm_eval/tasks/blimp/tough_vs_raising_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,447 copying lm_eval/tasks/blimp/wh_questions_subject_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,449 copying lm_eval/tasks/blimp/wh_questions_object_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,451 copying lm_eval/tasks/blimp/sentential_subject_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,453 copying lm_eval/tasks/blimp/transitive.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,455 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,457 copying lm_eval/tasks/blimp/wh_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,459 copying lm_eval/tasks/blimp/animate_subject_passive.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,460 copying lm_eval/tasks/blimp/anaphor_gender_agreement.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,462 copying lm_eval/tasks/blimp/principle_A_reconstruction.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,464 copying lm_eval/tasks/blimp/principle_A_c_command.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,466 copying lm_eval/tasks/blimp/principle_A_domain_3.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,468 copying lm_eval/tasks/blimp/intransitive.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,470 copying lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,471 copying lm_eval/tasks/blimp/left_branch_island_echo_question.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,473 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,475 copying lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,477 copying lm_eval/tasks/blimp/anaphor_number_agreement.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,479 copying lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,481 copying lm_eval/tasks/blimp/existential_there_object_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,482 copying lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,484 copying lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,486 copying lm_eval/tasks/blimp/principle_A_case_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,488 copying lm_eval/tasks/blimp/inchoative.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,489 copying lm_eval/tasks/blimp/superlative_quantifiers_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,491 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,493 copying lm_eval/tasks/blimp/passive_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,495 copying lm_eval/tasks/blimp/tough_vs_raising_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,496 copying lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,498 copying lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,500 copying lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,502 copying lm_eval/tasks/blimp/npi_present_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,504 copying lm_eval/tasks/blimp/left_branch_island_simple_question.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,506 copying lm_eval/tasks/blimp/causative.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,508 copying lm_eval/tasks/blimp/principle_A_domain_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,510 copying lm_eval/tasks/blimp/existential_there_subject_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,512 copying lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,514 copying lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,515 copying lm_eval/tasks/blimp/principle_A_domain_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,517 copying lm_eval/tasks/blimp/principle_A_case_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,519 copying lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,521 copying lm_eval/tasks/blimp/superlative_quantifiers_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,523 copying lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,525 copying lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,526 copying lm_eval/tasks/blimp/adjunct_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,528 copying lm_eval/tasks/blimp/only_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,530 copying lm_eval/tasks/blimp/animate_subject_trans.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,532 copying lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,534 copying lm_eval/tasks/blimp/expletive_it_object_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,536 copying lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,537 copying lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,539 copying lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,541 copying lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,546 copying lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,548 copying lm_eval/tasks/blimp/drop_argument.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,551 copying lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,553 copying lm_eval/tasks/blimp/npi_present_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,555 copying lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,557 copying lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:46,560 copying lm_eval/tasks/mutual/mutual.yaml -> build/lib/lm_eval/tasks/mutual 2024-01-03T16:18:46,562 copying lm_eval/tasks/mutual/multual_plus.yaml -> build/lib/lm_eval/tasks/mutual 2024-01-03T16:18:46,565 copying lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:46,567 copying lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:46,570 copying lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:46,572 copying lm_eval/tasks/xcopa/default_ta.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,575 copying lm_eval/tasks/xcopa/default_sw.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,577 copying lm_eval/tasks/xcopa/default_tr.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,579 copying lm_eval/tasks/xcopa/default_id.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,581 copying lm_eval/tasks/xcopa/default_et.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,584 copying lm_eval/tasks/xcopa/default_it.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,586 copying lm_eval/tasks/xcopa/default_vi.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,588 copying lm_eval/tasks/xcopa/default_qu.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,590 copying lm_eval/tasks/xcopa/default_zh.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,592 copying lm_eval/tasks/xcopa/default_ht.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,595 copying lm_eval/tasks/xcopa/default_th.yaml -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:46,597 copying lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml -> build/lib/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:46,599 creating build/lib/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:46,599 copying lm_eval/tasks/super_glue/wic/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:46,601 copying lm_eval/tasks/super_glue/wic/default.yaml -> build/lib/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:46,603 copying lm_eval/tasks/super_glue/cb/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:46,605 copying lm_eval/tasks/super_glue/cb/default.yaml -> build/lib/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:46,607 creating build/lib/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:46,608 copying lm_eval/tasks/super_glue/rte/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:46,610 copying lm_eval/tasks/super_glue/rte/default.yaml -> build/lib/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:46,612 creating build/lib/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:46,613 copying lm_eval/tasks/super_glue/boolq/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:46,615 copying lm_eval/tasks/super_glue/boolq/default.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:46,617 copying lm_eval/tasks/super_glue/boolq/seq2seq.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:46,619 copying lm_eval/tasks/super_glue/record/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/record 2024-01-03T16:18:46,621 copying lm_eval/tasks/super_glue/record/default.yaml -> build/lib/lm_eval/tasks/super_glue/record 2024-01-03T16:18:46,623 copying lm_eval/tasks/super_glue/wsc/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:46,625 copying lm_eval/tasks/super_glue/wsc/default.yaml -> build/lib/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:46,627 copying lm_eval/tasks/super_glue/copa/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:46,629 copying lm_eval/tasks/super_glue/copa/default.yaml -> build/lib/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:46,631 copying lm_eval/tasks/super_glue/multirc/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:46,632 copying lm_eval/tasks/super_glue/multirc/default.yaml -> build/lib/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:46,634 creating build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,635 copying lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,638 copying lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,640 copying lm_eval/tasks/bigbench/generate_until/codenames.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,642 copying lm_eval/tasks/bigbench/generate_until/multiemo.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,644 copying lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,646 copying lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,648 copying lm_eval/tasks/bigbench/generate_until/color.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,650 copying lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,652 copying lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,654 copying lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,656 copying lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,658 copying lm_eval/tasks/bigbench/generate_until/kannada.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,661 copying lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,663 copying lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,665 copying lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,667 copying lm_eval/tasks/bigbench/generate_until/gem.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,669 copying lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,671 copying lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,673 copying lm_eval/tasks/bigbench/generate_until/strange_stories.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,675 copying lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,677 copying lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,679 copying lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,681 copying lm_eval/tasks/bigbench/generate_until/word_sorting.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,683 copying lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,685 copying lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,688 copying lm_eval/tasks/bigbench/generate_until/logical_args.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,690 copying lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,692 copying lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,694 copying lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,696 copying lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,698 copying lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,700 copying lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,702 copying lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,704 copying lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,707 copying lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,709 copying lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,711 copying lm_eval/tasks/bigbench/generate_until/arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,713 copying lm_eval/tasks/bigbench/generate_until/fact_checker.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,716 copying lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,718 copying lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,721 copying lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,723 copying lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,725 copying lm_eval/tasks/bigbench/generate_until/language_identification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,727 copying lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,730 copying lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,732 copying lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,734 copying lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,737 copying lm_eval/tasks/bigbench/generate_until/physics.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,739 copying lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,741 copying lm_eval/tasks/bigbench/generate_until/rephrase.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,743 copying lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,746 copying lm_eval/tasks/bigbench/generate_until/date_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,748 copying lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,750 copying lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,753 copying lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,755 copying lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,758 copying lm_eval/tasks/bigbench/generate_until/physics_questions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,760 copying lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,762 copying lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,765 copying lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,767 copying lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,769 copying lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,772 copying lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,774 copying lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,776 copying lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,779 copying lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,781 copying lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,784 copying lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,786 copying lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,789 copying lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,791 copying lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,793 copying lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,795 copying lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,798 copying lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,800 copying lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,802 copying lm_eval/tasks/bigbench/generate_until/winowhy.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,804 copying lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,807 copying lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,809 copying lm_eval/tasks/bigbench/generate_until/strategyqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,811 copying lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,813 copying lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,815 copying lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,817 copying lm_eval/tasks/bigbench/generate_until/misconceptions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,820 copying lm_eval/tasks/bigbench/generate_until/implicatures.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,822 copying lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,824 copying lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,827 copying lm_eval/tasks/bigbench/generate_until/object_counting.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,829 copying lm_eval/tasks/bigbench/generate_until/social_support.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,831 copying lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,833 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,835 copying lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,838 copying lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,840 copying lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,842 copying lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,845 copying lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,847 copying lm_eval/tasks/bigbench/generate_until/anachronisms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,850 copying lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,852 copying lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,854 copying lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,856 copying lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,859 copying lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,861 copying lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,863 copying lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,865 copying lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,867 copying lm_eval/tasks/bigbench/generate_until/crass_ai.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,870 copying lm_eval/tasks/bigbench/generate_until/cryptonite.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,872 copying lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,875 copying lm_eval/tasks/bigbench/generate_until/social_iqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,877 copying lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,879 copying lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,881 copying lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,883 copying lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,886 copying lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,888 copying lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,890 copying lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,892 copying lm_eval/tasks/bigbench/generate_until/irony_identification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,895 copying lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,897 copying lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,899 copying lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,902 copying lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,904 copying lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,907 copying lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,909 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,912 copying lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,914 copying lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,916 copying lm_eval/tasks/bigbench/generate_until/timedial.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,918 copying lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,920 copying lm_eval/tasks/bigbench/generate_until/tense.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,923 copying lm_eval/tasks/bigbench/generate_until/navigate.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,925 copying lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,927 copying lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,930 copying lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,932 copying lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,934 copying lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,936 copying lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,939 copying lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,941 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,943 copying lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,945 copying lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,948 copying lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,950 copying lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,952 copying lm_eval/tasks/bigbench/generate_until/operators.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,954 copying lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,956 copying lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,959 copying lm_eval/tasks/bigbench/generate_until/list_functions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,964 copying lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,967 copying lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,970 copying lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,972 copying lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,975 copying lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,978 copying lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,981 copying lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,984 copying lm_eval/tasks/bigbench/generate_until/question_selection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,987 copying lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,990 copying lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,992 copying lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:46,995 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,003 copying lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,006 copying lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,008 copying lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,010 copying lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,012 copying lm_eval/tasks/bigbench/generate_until/ruin_names.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,014 copying lm_eval/tasks/bigbench/generate_until/code_line_description.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,017 copying lm_eval/tasks/bigbench/generate_until/language_games.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,019 copying lm_eval/tasks/bigbench/generate_until/topical_chat.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,021 copying lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,023 copying lm_eval/tasks/bigbench/generate_until/snarks.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:47,026 creating build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,027 copying lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,029 copying lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,032 copying lm_eval/tasks/bigbench/multiple_choice/codenames.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,034 copying lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,036 copying lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,038 copying lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,040 copying lm_eval/tasks/bigbench/multiple_choice/color.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,043 copying lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,045 copying lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,047 copying lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,050 copying lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,052 copying lm_eval/tasks/bigbench/multiple_choice/kannada.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,054 copying lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,056 copying lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,058 copying lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,061 copying lm_eval/tasks/bigbench/multiple_choice/gem.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,063 copying lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,065 copying lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,068 copying lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,070 copying lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,072 copying lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,075 copying lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,077 copying lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,080 copying lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,082 copying lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,084 copying lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,087 copying lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,089 copying lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,091 copying lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,093 copying lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,096 copying lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,098 copying lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,100 copying lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,102 copying lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,104 copying lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,107 copying lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,109 copying lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,111 copying lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,114 copying lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,116 copying lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,118 copying lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,120 copying lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,122 copying lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,124 copying lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,127 copying lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,129 copying lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,131 copying lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,133 copying lm_eval/tasks/bigbench/multiple_choice/physics.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,135 copying lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,138 copying lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,140 copying lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,143 copying lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,145 copying lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,147 copying lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,149 copying lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,152 copying lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,154 copying lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,156 copying lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,158 copying lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,160 copying lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,163 copying lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,165 copying lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,167 copying lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,169 copying lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,172 copying lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,174 copying lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,177 copying lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,179 copying lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,181 copying lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,184 copying lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,186 copying lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,189 copying lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,191 copying lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,194 copying lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,196 copying lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,198 copying lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,200 copying lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,202 copying lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,204 copying lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,206 copying lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,208 copying lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,210 copying lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,212 copying lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,215 copying lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,217 copying lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,219 copying lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,221 copying lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,223 copying lm_eval/tasks/bigbench/multiple_choice/social_support.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,225 copying lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,228 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,230 copying lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,232 copying lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,234 copying lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,236 copying lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,238 copying lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,240 copying lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,242 copying lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,244 copying lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,246 copying lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,248 copying lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,250 copying lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,252 copying lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,254 copying lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,256 copying lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,258 copying lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,260 copying lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,262 copying lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,264 copying lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,266 copying lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,268 copying lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,270 copying lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,272 copying lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,275 copying lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,277 copying lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,279 copying lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,281 copying lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,283 copying lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,285 copying lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,287 copying lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,289 copying lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,291 copying lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,293 copying lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,295 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,297 copying lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,299 copying lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,301 copying lm_eval/tasks/bigbench/multiple_choice/timedial.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,303 copying lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,305 copying lm_eval/tasks/bigbench/multiple_choice/tense.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,307 copying lm_eval/tasks/bigbench/multiple_choice/navigate.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,308 copying lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,310 copying lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,312 copying lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,314 copying lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,316 copying lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,318 copying lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,320 copying lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,322 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,325 copying lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,326 copying lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,329 copying lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,331 copying lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,333 copying lm_eval/tasks/bigbench/multiple_choice/operators.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,335 copying lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,337 copying lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,339 copying lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,341 copying lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,343 copying lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,345 copying lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,347 copying lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,349 copying lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,351 copying lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,353 copying lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,355 copying lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,357 copying lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,359 copying lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,360 copying lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,362 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,364 copying lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,366 copying lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,368 copying lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,370 copying lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,372 copying lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,374 copying lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,376 copying lm_eval/tasks/bigbench/multiple_choice/language_games.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,378 copying lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,380 copying lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,382 copying lm_eval/tasks/bigbench/multiple_choice/snarks.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:47,384 copying lm_eval/tasks/drop/default.yaml -> build/lib/lm_eval/tasks/drop 2024-01-03T16:18:47,386 copying lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,388 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,390 copying lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,392 copying lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,394 copying lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,396 copying lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,398 copying lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,401 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,403 copying lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,405 copying lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,406 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,408 copying lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,410 copying lm_eval/tasks/cmmlu/cmmlu_default_management.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,412 copying lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,414 copying lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,416 copying lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,418 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,420 copying lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,421 copying lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,423 copying lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,425 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,427 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,429 copying lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,431 copying lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,433 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,435 copying lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,437 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,439 copying lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,441 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,443 copying lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,445 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,447 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,449 copying lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,451 copying lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,453 copying lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,455 copying lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,457 copying lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,458 copying lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,461 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,463 copying lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,465 copying lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,467 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,469 copying lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,471 copying lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,473 copying lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,475 copying lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,477 copying lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,479 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,481 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,483 copying lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,485 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,487 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,489 copying lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,491 copying lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,494 copying lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,496 copying lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,498 copying lm_eval/tasks/cmmlu/cmmlu_default_education.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,500 copying lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,502 copying lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,504 copying lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,506 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,508 copying lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,510 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,512 copying lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,514 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,516 copying lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,518 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:47,520 copying lm_eval/tasks/logiqa2/logieval.yaml -> build/lib/lm_eval/tasks/logiqa2 2024-01-03T16:18:47,522 copying lm_eval/tasks/logiqa2/logiqa2.yaml -> build/lib/lm_eval/tasks/logiqa2 2024-01-03T16:18:47,524 creating build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,525 copying lm_eval/tasks/unscramble/reversed_words.yaml -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,527 copying lm_eval/tasks/unscramble/anagrams1.yaml -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,529 copying lm_eval/tasks/unscramble/cycle_letters.yaml -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,532 copying lm_eval/tasks/unscramble/random_insertion.yaml -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,534 copying lm_eval/tasks/unscramble/anagrams2.yaml -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:47,536 copying lm_eval/tasks/coqa/default.yaml -> build/lib/lm_eval/tasks/coqa 2024-01-03T16:18:47,538 creating build/lib/lm_eval/tasks/prost 2024-01-03T16:18:47,539 copying lm_eval/tasks/prost/corypaik_prost.yaml -> build/lib/lm_eval/tasks/prost 2024-01-03T16:18:47,541 copying lm_eval/tasks/wikitext/wikitext.yaml -> build/lib/lm_eval/tasks/wikitext 2024-01-03T16:18:47,543 copying lm_eval/tasks/pubmedqa/pubmedqa.yaml -> build/lib/lm_eval/tasks/pubmedqa 2024-01-03T16:18:47,545 copying lm_eval/tasks/paws-x/paws_en.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,547 copying lm_eval/tasks/paws-x/paws_de.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,549 copying lm_eval/tasks/paws-x/paws_zh.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,551 copying lm_eval/tasks/paws-x/paws_fr.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,553 copying lm_eval/tasks/paws-x/paws_es.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,555 copying lm_eval/tasks/paws-x/paws_ko.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,557 copying lm_eval/tasks/paws-x/paws_ja.yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:47,559 creating build/lib/lm_eval/tasks/babi 2024-01-03T16:18:47,559 copying lm_eval/tasks/babi/babi.yaml -> build/lib/lm_eval/tasks/babi 2024-01-03T16:18:47,561 creating build/lib/lm_eval/tasks/storycloze 2024-01-03T16:18:47,562 copying lm_eval/tasks/storycloze/storycloze_2018.yaml -> build/lib/lm_eval/tasks/storycloze 2024-01-03T16:18:47,564 copying lm_eval/tasks/storycloze/storycloze_2016.yaml -> build/lib/lm_eval/tasks/storycloze 2024-01-03T16:18:47,566 creating build/lib/lm_eval/tasks/lambada 2024-01-03T16:18:47,567 copying lm_eval/tasks/lambada/lambada_standard.yaml -> build/lib/lm_eval/tasks/lambada 2024-01-03T16:18:47,569 copying lm_eval/tasks/lambada/lambada_openai.yaml -> build/lib/lm_eval/tasks/lambada 2024-01-03T16:18:47,571 creating build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,572 copying lm_eval/tasks/mmlu/default/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,574 copying lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,576 copying lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,578 copying lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,580 copying lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,582 copying lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,584 copying lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,586 copying lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,588 copying lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,590 copying lm_eval/tasks/mmlu/default/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,593 copying lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,595 copying lm_eval/tasks/mmlu/default/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,597 copying lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,599 copying lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,601 copying lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,603 copying lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,605 copying lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,607 copying lm_eval/tasks/mmlu/default/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,609 copying lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,611 copying lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,612 copying lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,615 copying lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,617 copying lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,619 copying lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,620 copying lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,622 copying lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,624 copying lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,626 copying lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,628 copying lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,630 copying lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,632 copying lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,634 copying lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,636 copying lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,638 copying lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,640 copying lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,642 copying lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,644 copying lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,646 copying lm_eval/tasks/mmlu/default/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,648 copying lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,650 copying lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,652 copying lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,654 copying lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,656 copying lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,658 copying lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,660 copying lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,662 copying lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,665 copying lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,667 copying lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,669 copying lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,671 copying lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,673 copying lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,675 copying lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,677 copying lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,679 copying lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,681 copying lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,683 copying lm_eval/tasks/mmlu/default/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,685 copying lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,687 copying lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:47,689 creating build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,690 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,692 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,694 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,697 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,699 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,701 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,703 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,705 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,707 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,709 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,712 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,713 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,716 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,717 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,719 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,721 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,723 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,725 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,727 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,730 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,731 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,733 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,735 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,737 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,739 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,741 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,743 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,744 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,746 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,748 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,750 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,752 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,754 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,756 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,758 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,760 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,762 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,764 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,766 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,768 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,770 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,772 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,774 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,776 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,778 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,780 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,782 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,785 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,786 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,788 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,790 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,792 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,794 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,796 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,798 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,800 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,802 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,804 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:47,806 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot 2024-01-03T16:18:47,806 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,807 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,810 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,812 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,814 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,816 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,818 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,820 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,822 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,824 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,826 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,828 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,830 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,832 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,834 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,836 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,838 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,839 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,841 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,843 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,845 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,847 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,849 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,851 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,853 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,855 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,857 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,859 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,861 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,863 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,865 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,867 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,869 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,871 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,873 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,875 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,877 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,879 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,881 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,883 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,885 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,887 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,889 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,891 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,893 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,895 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,897 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,899 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,901 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,903 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,905 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,907 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,909 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,912 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,914 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,916 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,918 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,920 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,922 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:47,924 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,925 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,928 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,929 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,932 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,934 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,936 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,937 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,940 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,941 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,943 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,945 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,947 copying lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,949 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,951 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,953 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,956 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,958 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,960 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,962 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,964 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,966 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,968 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,970 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,972 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,974 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,976 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,978 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,980 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,982 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,984 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,986 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,988 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,990 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,992 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,995 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,996 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:47,999 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,001 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,003 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,005 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,007 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,009 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,011 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,013 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,015 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,017 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,019 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,021 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,023 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,025 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,027 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,029 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,031 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,033 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,035 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,037 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,039 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,041 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,043 creating build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,044 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,046 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,048 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,050 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,052 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,055 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,057 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,059 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,061 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,063 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,066 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,068 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,070 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,072 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,074 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,076 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,079 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,081 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,083 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,086 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,088 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,090 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,093 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,095 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,098 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,100 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,102 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,104 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,106 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,108 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,111 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,113 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,115 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,117 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,120 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,122 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,124 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,127 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,129 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,131 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,133 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,136 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,138 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,140 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,142 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,144 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,147 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,149 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,151 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,153 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,155 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,157 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,160 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,162 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,165 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,167 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,169 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,171 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,173 copying lm_eval/tasks/wsc273/default.yaml -> build/lib/lm_eval/tasks/wsc273 2024-01-03T16:18:48,175 creating build/lib/lm_eval/tasks/sciq 2024-01-03T16:18:48,176 copying lm_eval/tasks/sciq/sciq.yaml -> build/lib/lm_eval/tasks/sciq 2024-01-03T16:18:48,178 creating build/lib/lm_eval/tasks/arc 2024-01-03T16:18:48,179 copying lm_eval/tasks/arc/arc_challenge.yaml -> build/lib/lm_eval/tasks/arc 2024-01-03T16:18:48,181 copying lm_eval/tasks/arc/arc_easy.yaml -> build/lib/lm_eval/tasks/arc 2024-01-03T16:18:48,183 creating build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,184 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_zh.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,186 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ja.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,188 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_sw.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,189 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_es.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,191 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_de.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,193 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_en.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,195 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_th.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,197 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_te.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,199 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_bn.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,202 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_fr.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,205 copying lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ru.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,208 creating build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,209 copying lm_eval/tasks/mgsm/en_cot/mgsm_th_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,211 copying lm_eval/tasks/mgsm/en_cot/mgsm_ja_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,213 copying lm_eval/tasks/mgsm/en_cot/mgsm_fr_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,215 copying lm_eval/tasks/mgsm/en_cot/mgsm_zh_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,217 copying lm_eval/tasks/mgsm/en_cot/mgsm_de_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,219 copying lm_eval/tasks/mgsm/en_cot/mgsm_es_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,221 copying lm_eval/tasks/mgsm/en_cot/mgsm_sw_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,223 copying lm_eval/tasks/mgsm/en_cot/mgsm_te_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,225 copying lm_eval/tasks/mgsm/en_cot/mgsm_ru_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,227 copying lm_eval/tasks/mgsm/en_cot/mgsm_bn_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,229 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_en-cot.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,231 creating build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,232 copying lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,235 copying lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,237 copying lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,239 copying lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,241 copying lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,243 copying lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,245 copying lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,247 copying lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,248 copying lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,250 copying lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,252 copying lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,254 creating build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,255 copying lm_eval/tasks/arithmetic/arithmetic_3da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,257 copying lm_eval/tasks/arithmetic/arithmetic_2da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,259 copying lm_eval/tasks/arithmetic/arithmetic_3ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,261 copying lm_eval/tasks/arithmetic/arithmetic_4da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,263 copying lm_eval/tasks/arithmetic/arithmetic_5ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,265 copying lm_eval/tasks/arithmetic/arithmetic_4ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,267 copying lm_eval/tasks/arithmetic/arithmetic_2dm.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,269 copying lm_eval/tasks/arithmetic/arithmetic_1dc.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,271 copying lm_eval/tasks/arithmetic/arithmetic_5da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,273 copying lm_eval/tasks/arithmetic/arithmetic_2ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,275 copying lm_eval/tasks/code_x_glue/code-text/go.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,277 copying lm_eval/tasks/code_x_glue/code-text/ruby.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,279 copying lm_eval/tasks/code_x_glue/code-text/php.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,281 copying lm_eval/tasks/code_x_glue/code-text/python.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,283 copying lm_eval/tasks/code_x_glue/code-text/java.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,285 copying lm_eval/tasks/code_x_glue/code-text/javascript.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:48,287 copying lm_eval/tasks/hendrycks_ethics/justice.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,289 copying lm_eval/tasks/hendrycks_ethics/commonsense.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,291 copying lm_eval/tasks/hendrycks_ethics/deontology.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,293 copying lm_eval/tasks/hendrycks_ethics/virtue.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,295 copying lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,297 creating build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,298 copying lm_eval/tasks/pile/pile_gutenberg.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,300 copying lm_eval/tasks/pile/pile_youtubesubtitles.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,302 copying lm_eval/tasks/pile/pile_europarl.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,304 copying lm_eval/tasks/pile/pile_wikipedia.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,305 copying lm_eval/tasks/pile/pile_books3.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,307 copying lm_eval/tasks/pile/pile_github.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,309 copying lm_eval/tasks/pile/pile_bookcorpus2.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,311 copying lm_eval/tasks/pile/pile_enron.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,313 copying lm_eval/tasks/pile/pile_dm-mathematics.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,315 copying lm_eval/tasks/pile/pile_nih-exporter.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,317 copying lm_eval/tasks/pile/pile_arxiv.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,319 copying lm_eval/tasks/pile/pile_pubmed-central.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,321 copying lm_eval/tasks/pile/pile_ubuntu-irc.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,323 copying lm_eval/tasks/pile/pile_stackexchange.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,325 copying lm_eval/tasks/pile/pile_pile-cc.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,327 copying lm_eval/tasks/pile/pile_pubmed-abstracts.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,329 copying lm_eval/tasks/pile/pile_opensubtitles.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,331 copying lm_eval/tasks/pile/pile_openwebtext2.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,333 copying lm_eval/tasks/pile/pile_uspto.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,335 copying lm_eval/tasks/pile/pile_hackernews.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,337 copying lm_eval/tasks/pile/pile_freelaw.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,338 copying lm_eval/tasks/pile/pile_philpapers.yaml -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,340 creating build/lib/lm_eval/tasks/openbookqa 2024-01-03T16:18:48,341 copying lm_eval/tasks/openbookqa/openbookqa.yaml -> build/lib/lm_eval/tasks/openbookqa 2024-01-03T16:18:48,343 creating build/lib/lm_eval/tasks/anli 2024-01-03T16:18:48,344 copying lm_eval/tasks/anli/anli_r3.yaml -> build/lib/lm_eval/tasks/anli 2024-01-03T16:18:48,346 copying lm_eval/tasks/anli/anli_r2.yaml -> build/lib/lm_eval/tasks/anli 2024-01-03T16:18:48,348 copying lm_eval/tasks/anli/anli_r1.yaml -> build/lib/lm_eval/tasks/anli 2024-01-03T16:18:48,350 copying lm_eval/tasks/xwinograd/xwinograd_zh.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,352 copying lm_eval/tasks/xwinograd/xwinograd_fr.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,354 copying lm_eval/tasks/xwinograd/xwinograd_pt.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,356 copying lm_eval/tasks/xwinograd/xwinograd_en.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,358 copying lm_eval/tasks/xwinograd/xwinograd_ru.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,360 copying lm_eval/tasks/xwinograd/xwinograd_jp.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,362 creating build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,363 copying lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,365 copying lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,367 copying lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,369 copying lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,372 copying lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,374 creating build/lib/lm_eval/tasks/polemo2 2024-01-03T16:18:48,375 copying lm_eval/tasks/polemo2/polemo2_out.yaml -> build/lib/lm_eval/tasks/polemo2 2024-01-03T16:18:48,377 copying lm_eval/tasks/polemo2/polemo2_in.yaml -> build/lib/lm_eval/tasks/polemo2 2024-01-03T16:18:48,379 copying lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,381 copying lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,383 copying lm_eval/tasks/minerva_math/minerva_math_algebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,385 copying lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,387 copying lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,389 copying lm_eval/tasks/minerva_math/minerva_math_geometry.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,391 copying lm_eval/tasks/minerva_math/minerva_math_precalc.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,393 copying lm_eval/tasks/csatqa/csatqa_rcss.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,395 copying lm_eval/tasks/csatqa/csatqa_gr.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,397 copying lm_eval/tasks/csatqa/csatqa_li.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,399 copying lm_eval/tasks/csatqa/csatqa_rcs.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,401 copying lm_eval/tasks/csatqa/csatqa_rch.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,403 copying lm_eval/tasks/csatqa/csatqa_wr.yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,405 copying lm_eval/tasks/xnli/xnli_zh.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,407 copying lm_eval/tasks/xnli/xnli_ar.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,409 copying lm_eval/tasks/xnli/xnli_tr.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,411 copying lm_eval/tasks/xnli/xnli_fr.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,413 copying lm_eval/tasks/xnli/xnli_en.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,415 copying lm_eval/tasks/xnli/xnli_el.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,417 copying lm_eval/tasks/xnli/xnli_ur.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,419 copying lm_eval/tasks/xnli/xnli_de.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,421 copying lm_eval/tasks/xnli/xnli_vi.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,423 copying lm_eval/tasks/xnli/xnli_es.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,425 copying lm_eval/tasks/xnli/xnli_bg.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,427 copying lm_eval/tasks/xnli/xnli_ru.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,429 copying lm_eval/tasks/xnli/xnli_hi.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,431 copying lm_eval/tasks/xnli/xnli_sw.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,433 copying lm_eval/tasks/xnli/xnli_th.yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,436 creating build/lib/lm_eval/tasks/piqa 2024-01-03T16:18:48,437 copying lm_eval/tasks/piqa/piqa.yaml -> build/lib/lm_eval/tasks/piqa 2024-01-03T16:18:48,439 creating build/lib/lm_eval/tasks/swag 2024-01-03T16:18:48,440 copying lm_eval/tasks/swag/swag.yaml -> build/lib/lm_eval/tasks/swag 2024-01-03T16:18:48,441 creating build/lib/lm_eval/tasks/benchmarks 2024-01-03T16:18:48,443 copying lm_eval/tasks/benchmarks/pythia.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-01-03T16:18:48,445 copying lm_eval/tasks/benchmarks/t0_eval.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-01-03T16:18:48,447 copying lm_eval/tasks/benchmarks/minerva_math.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-01-03T16:18:48,448 creating build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,449 copying lm_eval/tasks/benchmarks/flan/flan_arc.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,451 copying lm_eval/tasks/benchmarks/flan/flan_cot.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,453 copying lm_eval/tasks/benchmarks/flan/flan_held_in.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,455 copying lm_eval/tasks/benchmarks/flan/flan_held_out.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,457 copying lm_eval/tasks/benchmarks/flan/flan_anli.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,459 copying lm_eval/tasks/benchmarks/flan/flan_boolq.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,461 copying lm_eval/tasks/benchmarks/flan/flan_rte.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,463 creating build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:48,465 copying lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml -> build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:48,467 copying lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml -> build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:48,469 copying lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml -> build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:48,471 copying lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml -> build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:48,473 creating build/lib/lm_eval/tasks/headqa 2024-01-03T16:18:48,474 copying lm_eval/tasks/headqa/headqa_es.yaml -> build/lib/lm_eval/tasks/headqa 2024-01-03T16:18:48,476 copying lm_eval/tasks/headqa/headqa_en.yaml -> build/lib/lm_eval/tasks/headqa 2024-01-03T16:18:48,478 creating build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,479 copying lm_eval/tasks/xstorycloze/default_sw.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,481 copying lm_eval/tasks/xstorycloze/default_hi.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,483 copying lm_eval/tasks/xstorycloze/default_id.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,485 copying lm_eval/tasks/xstorycloze/default_eu.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,486 copying lm_eval/tasks/xstorycloze/default_en.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,488 copying lm_eval/tasks/xstorycloze/default_my.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,490 copying lm_eval/tasks/xstorycloze/default_ar.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,492 copying lm_eval/tasks/xstorycloze/default_es.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,494 copying lm_eval/tasks/xstorycloze/default_te.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,496 copying lm_eval/tasks/xstorycloze/default_zh.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,498 copying lm_eval/tasks/xstorycloze/default_ru.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,500 copying lm_eval/tasks/qasper/bool.yaml -> build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:48,502 copying lm_eval/tasks/qasper/freeform.yaml -> build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:48,504 creating build/lib/lm_eval/tasks/glue/sst 2024-01-03T16:18:48,505 copying lm_eval/tasks/glue/sst/default.yaml -> build/lib/lm_eval/tasks/glue/sst 2024-01-03T16:18:48,507 copying lm_eval/tasks/glue/mnli/default.yaml -> build/lib/lm_eval/tasks/glue/mnli 2024-01-03T16:18:48,509 copying lm_eval/tasks/glue/mnli/mismatch.yaml -> build/lib/lm_eval/tasks/glue/mnli 2024-01-03T16:18:48,511 creating build/lib/lm_eval/tasks/glue/cola 2024-01-03T16:18:48,512 copying lm_eval/tasks/glue/cola/default.yaml -> build/lib/lm_eval/tasks/glue/cola 2024-01-03T16:18:48,514 creating build/lib/lm_eval/tasks/glue/rte 2024-01-03T16:18:48,515 copying lm_eval/tasks/glue/rte/default.yaml -> build/lib/lm_eval/tasks/glue/rte 2024-01-03T16:18:48,517 creating build/lib/lm_eval/tasks/glue/qnli 2024-01-03T16:18:48,518 copying lm_eval/tasks/glue/qnli/default.yaml -> build/lib/lm_eval/tasks/glue/qnli 2024-01-03T16:18:48,520 creating build/lib/lm_eval/tasks/glue/wnli 2024-01-03T16:18:48,521 copying lm_eval/tasks/glue/wnli/default.yaml -> build/lib/lm_eval/tasks/glue/wnli 2024-01-03T16:18:48,523 creating build/lib/lm_eval/tasks/glue/qqp 2024-01-03T16:18:48,523 copying lm_eval/tasks/glue/qqp/default.yaml -> build/lib/lm_eval/tasks/glue/qqp 2024-01-03T16:18:48,525 creating build/lib/lm_eval/tasks/glue/mrpc 2024-01-03T16:18:48,526 copying lm_eval/tasks/glue/mrpc/default.yaml -> build/lib/lm_eval/tasks/glue/mrpc 2024-01-03T16:18:48,528 copying lm_eval/tasks/logiqa/logiqa.yaml -> build/lib/lm_eval/tasks/logiqa 2024-01-03T16:18:48,530 copying lm_eval/tasks/mathqa/mathqa.yaml -> build/lib/lm_eval/tasks/mathqa 2024-01-03T16:18:48,532 creating build/lib/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:48,533 copying lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml -> build/lib/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:48,535 copying lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml -> build/lib/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:48,537 copying lm_eval/tasks/qa4mre/qa4mre_2013.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:48,539 copying lm_eval/tasks/qa4mre/qa4mre_2012.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:48,541 copying lm_eval/tasks/qa4mre/qa4mre_2011.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:48,543 copying lm_eval/tasks/winogrande/default.yaml -> build/lib/lm_eval/tasks/winogrande 2024-01-03T16:18:48,545 copying lm_eval/tasks/README.md -> build/lib/lm_eval/tasks 2024-01-03T16:18:48,547 copying lm_eval/tasks/toxigen/README.md -> build/lib/lm_eval/tasks/toxigen 2024-01-03T16:18:48,550 copying lm_eval/tasks/hellaswag/README.md -> build/lib/lm_eval/tasks/hellaswag 2024-01-03T16:18:48,552 copying lm_eval/tasks/squadv2/README.md -> build/lib/lm_eval/tasks/squadv2 2024-01-03T16:18:48,555 copying lm_eval/tasks/bbh/README.md -> build/lib/lm_eval/tasks/bbh 2024-01-03T16:18:48,557 copying lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:48,559 copying lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:48,561 copying lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:48,563 copying lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:48,565 copying lm_eval/tasks/mc_taco/README.md -> build/lib/lm_eval/tasks/mc_taco 2024-01-03T16:18:48,567 copying lm_eval/tasks/asdiv/README.md -> build/lib/lm_eval/tasks/asdiv 2024-01-03T16:18:48,570 copying lm_eval/tasks/belebele/README.md -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:48,572 copying lm_eval/tasks/belebele/_default_template_yaml -> build/lib/lm_eval/tasks/belebele 2024-01-03T16:18:48,574 copying lm_eval/tasks/triviaqa/README.md -> build/lib/lm_eval/tasks/triviaqa 2024-01-03T16:18:48,576 copying lm_eval/tasks/webqs/README.md -> build/lib/lm_eval/tasks/webqs 2024-01-03T16:18:48,579 copying lm_eval/tasks/translation/wmt_common_yaml -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:48,581 copying lm_eval/tasks/translation/README.md -> build/lib/lm_eval/tasks/translation 2024-01-03T16:18:48,584 copying lm_eval/tasks/race/README.md -> build/lib/lm_eval/tasks/race 2024-01-03T16:18:48,586 copying lm_eval/tasks/nq_open/README.md -> build/lib/lm_eval/tasks/nq_open 2024-01-03T16:18:48,588 copying lm_eval/tasks/crows_pairs/README.md -> build/lib/lm_eval/tasks/crows_pairs 2024-01-03T16:18:48,591 copying lm_eval/tasks/model_written_evals/persona/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:48,593 creating build/lib/lm_eval/tasks/model_written_evals/winogenerated 2024-01-03T16:18:48,594 copying lm_eval/tasks/model_written_evals/winogenerated/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/winogenerated 2024-01-03T16:18:48,596 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:48,598 copying lm_eval/tasks/gsm8k/README.md -> build/lib/lm_eval/tasks/gsm8k 2024-01-03T16:18:48,600 copying lm_eval/tasks/wmt2016/README.md -> build/lib/lm_eval/tasks/wmt2016 2024-01-03T16:18:48,602 copying lm_eval/tasks/ceval/_default_ceval_yaml -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:48,605 copying lm_eval/tasks/ceval/README.md -> build/lib/lm_eval/tasks/ceval 2024-01-03T16:18:48,608 copying lm_eval/tasks/scrolls/README.md -> build/lib/lm_eval/tasks/scrolls 2024-01-03T16:18:48,610 copying lm_eval/tasks/blimp/README.md -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:48,612 copying lm_eval/tasks/blimp/_template_yaml -> build/lib/lm_eval/tasks/blimp 2024-01-03T16:18:48,615 copying lm_eval/tasks/mutual/README.md -> build/lib/lm_eval/tasks/mutual 2024-01-03T16:18:48,617 copying lm_eval/tasks/truthfulqa/README.md -> build/lib/lm_eval/tasks/truthfulqa 2024-01-03T16:18:48,620 copying lm_eval/tasks/xcopa/README.md -> build/lib/lm_eval/tasks/xcopa 2024-01-03T16:18:48,623 copying lm_eval/tasks/super_glue/README.md -> build/lib/lm_eval/tasks/super_glue 2024-01-03T16:18:48,628 copying lm_eval/tasks/bigbench/generate_until_template_yaml -> build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:48,631 copying lm_eval/tasks/bigbench/README.md -> build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:48,633 copying lm_eval/tasks/bigbench/multiple_choice_template_yaml -> build/lib/lm_eval/tasks/bigbench 2024-01-03T16:18:48,635 copying lm_eval/tasks/drop/README.md -> build/lib/lm_eval/tasks/drop 2024-01-03T16:18:48,638 copying lm_eval/tasks/cmmlu/README.md -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:48,640 copying lm_eval/tasks/cmmlu/_default_template_yaml -> build/lib/lm_eval/tasks/cmmlu 2024-01-03T16:18:48,643 copying lm_eval/tasks/logiqa2/README.md -> build/lib/lm_eval/tasks/logiqa2 2024-01-03T16:18:48,645 copying lm_eval/tasks/unscramble/README.md -> build/lib/lm_eval/tasks/unscramble 2024-01-03T16:18:48,647 copying lm_eval/tasks/coqa/README.md -> build/lib/lm_eval/tasks/coqa 2024-01-03T16:18:48,650 copying lm_eval/tasks/prost/README.md -> build/lib/lm_eval/tasks/prost 2024-01-03T16:18:48,652 copying lm_eval/tasks/wikitext/README.md -> build/lib/lm_eval/tasks/wikitext 2024-01-03T16:18:48,655 copying lm_eval/tasks/pubmedqa/README.md -> build/lib/lm_eval/tasks/pubmedqa 2024-01-03T16:18:48,657 copying lm_eval/tasks/paws-x/pawsx_template_yaml -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:48,659 copying lm_eval/tasks/paws-x/README.md -> build/lib/lm_eval/tasks/paws-x 2024-01-03T16:18:48,661 copying lm_eval/tasks/babi/README.md -> build/lib/lm_eval/tasks/babi 2024-01-03T16:18:48,663 copying lm_eval/tasks/storycloze/README.md -> build/lib/lm_eval/tasks/storycloze 2024-01-03T16:18:48,665 copying lm_eval/tasks/lambada/README.md -> build/lib/lm_eval/tasks/lambada 2024-01-03T16:18:48,668 copying lm_eval/tasks/mmlu/default/_default_template_yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-01-03T16:18:48,670 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:48,672 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:48,674 copying lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:48,676 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,678 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:48,684 copying lm_eval/tasks/wsc273/README.md -> build/lib/lm_eval/tasks/wsc273 2024-01-03T16:18:48,686 creating build/lib/lm_eval/tasks/siqa 2024-01-03T16:18:48,687 copying lm_eval/tasks/siqa/README.md -> build/lib/lm_eval/tasks/siqa 2024-01-03T16:18:48,689 copying lm_eval/tasks/siqa/default.yml -> build/lib/lm_eval/tasks/siqa 2024-01-03T16:18:48,691 copying lm_eval/tasks/sciq/README.md -> build/lib/lm_eval/tasks/sciq 2024-01-03T16:18:48,693 copying lm_eval/tasks/arc/README.md -> build/lib/lm_eval/tasks/arc 2024-01-03T16:18:48,695 copying lm_eval/tasks/mgsm/README.md -> build/lib/lm_eval/tasks/mgsm 2024-01-03T16:18:48,698 copying lm_eval/tasks/mgsm/native_cot/cot_yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:48,700 copying lm_eval/tasks/mgsm/en_cot/cot_yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:48,702 copying lm_eval/tasks/mgsm/direct/direct_yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:48,704 copying lm_eval/tasks/arithmetic/README.md -> build/lib/lm_eval/tasks/arithmetic 2024-01-03T16:18:48,707 copying lm_eval/tasks/hendrycks_ethics/README.md -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,709 copying lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:48,711 copying lm_eval/tasks/pile/README.md -> build/lib/lm_eval/tasks/pile 2024-01-03T16:18:48,713 copying lm_eval/tasks/openbookqa/README.md -> build/lib/lm_eval/tasks/openbookqa 2024-01-03T16:18:48,716 copying lm_eval/tasks/anli/README.md -> build/lib/lm_eval/tasks/anli 2024-01-03T16:18:48,718 copying lm_eval/tasks/xwinograd/README.md -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,720 copying lm_eval/tasks/xwinograd/xwinograd_common_yaml -> build/lib/lm_eval/tasks/xwinograd 2024-01-03T16:18:48,723 copying lm_eval/tasks/lambada_multilingual/README.md -> build/lib/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:48,725 copying lm_eval/tasks/polemo2/README.md -> build/lib/lm_eval/tasks/polemo2 2024-01-03T16:18:48,727 copying lm_eval/tasks/minerva_math/README.md -> build/lib/lm_eval/tasks/minerva_math 2024-01-03T16:18:48,730 copying lm_eval/tasks/csatqa/_default_csatqa_yaml -> build/lib/lm_eval/tasks/csatqa 2024-01-03T16:18:48,732 copying lm_eval/tasks/xnli/xnli_common_yaml -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,734 copying lm_eval/tasks/xnli/README.md -> build/lib/lm_eval/tasks/xnli 2024-01-03T16:18:48,737 copying lm_eval/tasks/piqa/README.md -> build/lib/lm_eval/tasks/piqa 2024-01-03T16:18:48,739 copying lm_eval/tasks/swag/README.md -> build/lib/lm_eval/tasks/swag 2024-01-03T16:18:48,741 copying lm_eval/tasks/benchmarks/flan/flan_held_in_yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:48,743 creating build/lib/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:48,744 copying lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml -> build/lib/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:48,746 copying lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml -> build/lib/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:48,748 copying lm_eval/tasks/headqa/README.md -> build/lib/lm_eval/tasks/headqa 2024-01-03T16:18:48,750 copying lm_eval/tasks/xstorycloze/README.md -> build/lib/lm_eval/tasks/xstorycloze 2024-01-03T16:18:48,752 copying lm_eval/tasks/qasper/README.md -> build/lib/lm_eval/tasks/qasper 2024-01-03T16:18:48,755 copying lm_eval/tasks/glue/README.md -> build/lib/lm_eval/tasks/glue 2024-01-03T16:18:48,758 copying lm_eval/tasks/logiqa/README.md -> build/lib/lm_eval/tasks/logiqa 2024-01-03T16:18:48,760 copying lm_eval/tasks/mathqa/README.md -> build/lib/lm_eval/tasks/mathqa 2024-01-03T16:18:48,762 copying lm_eval/tasks/lambada_cloze/README.md -> build/lib/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:48,765 copying lm_eval/tasks/qa4mre/README.md -> build/lib/lm_eval/tasks/qa4mre 2024-01-03T16:18:48,767 copying lm_eval/tasks/winogrande/README.md -> build/lib/lm_eval/tasks/winogrande 2024-01-03T16:18:49,316 installing to build/bdist.linux-armv7l/wheel 2024-01-03T16:18:49,317 running install 2024-01-03T16:18:49,340 running install_lib 2024-01-03T16:18:49,344 creating build/bdist.linux-armv7l 2024-01-03T16:18:49,345 creating build/bdist.linux-armv7l/wheel 2024-01-03T16:18:49,346 creating build/bdist.linux-armv7l/wheel/lm_eval 2024-01-03T16:18:49,348 creating build/bdist.linux-armv7l/wheel/lm_eval/prompts 2024-01-03T16:18:49,349 copying build/lib/lm_eval/prompts/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/prompts 2024-01-03T16:18:49,351 copying build/lib/lm_eval/__main__.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-01-03T16:18:49,353 copying build/lib/lm_eval/evaluator.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-01-03T16:18:49,356 creating build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-01-03T16:18:49,356 copying build/lib/lm_eval/decontamination/decontaminate.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-01-03T16:18:49,359 copying build/lib/lm_eval/decontamination/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-01-03T16:18:49,360 copying build/lib/lm_eval/decontamination/archiver.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-01-03T16:18:49,362 copying build/lib/lm_eval/decontamination/janitor.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-01-03T16:18:49,365 creating build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,366 copying build/lib/lm_eval/models/openai_completions.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,369 copying build/lib/lm_eval/models/anthropic_llms.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,371 copying build/lib/lm_eval/models/textsynth.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,373 copying build/lib/lm_eval/models/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,375 copying build/lib/lm_eval/models/gguf.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,377 copying build/lib/lm_eval/models/huggingface.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,380 copying build/lib/lm_eval/models/dummy.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,382 copying build/lib/lm_eval/models/vllm_causallms.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-01-03T16:18:49,384 copying build/lib/lm_eval/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-01-03T16:18:49,388 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks 2024-01-03T16:18:49,389 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-01-03T16:18:49,390 copying build/lib/lm_eval/tasks/toxigen/toxigen.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-01-03T16:18:49,392 copying build/lib/lm_eval/tasks/toxigen/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-01-03T16:18:49,394 copying build/lib/lm_eval/tasks/toxigen/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-01-03T16:18:49,396 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-01-03T16:18:49,397 copying build/lib/lm_eval/tasks/hellaswag/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-01-03T16:18:49,399 copying build/lib/lm_eval/tasks/hellaswag/hellaswag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-01-03T16:18:49,401 copying build/lib/lm_eval/tasks/hellaswag/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-01-03T16:18:49,403 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-01-03T16:18:49,404 copying build/lib/lm_eval/tasks/squadv2/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-01-03T16:18:49,407 copying build/lib/lm_eval/tasks/squadv2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-01-03T16:18:49,409 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-01-03T16:18:49,410 copying build/lib/lm_eval/tasks/bbh/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-01-03T16:18:49,413 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,413 copying build/lib/lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,415 copying build/lib/lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,417 copying build/lib/lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,418 copying build/lib/lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,420 copying build/lib/lm_eval/tasks/bbh/zeroshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,422 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,424 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,425 copying build/lib/lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,427 copying build/lib/lm_eval/tasks/bbh/zeroshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,429 copying build/lib/lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,431 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,432 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,434 copying build/lib/lm_eval/tasks/bbh/zeroshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,436 copying build/lib/lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,438 copying build/lib/lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,439 copying build/lib/lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,441 copying build/lib/lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,443 copying build/lib/lm_eval/tasks/bbh/zeroshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,445 copying build/lib/lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,447 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,448 copying build/lib/lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,450 copying build/lib/lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,452 copying build/lib/lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,454 copying build/lib/lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,455 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,457 copying build/lib/lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,459 copying build/lib/lm_eval/tasks/bbh/zeroshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,461 copying build/lib/lm_eval/tasks/bbh/zeroshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-01-03T16:18:49,685 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,685 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,687 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,689 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,690 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,692 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,693 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,695 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,697 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,699 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,700 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,702 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,704 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,705 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,707 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,708 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,710 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,712 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,713 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,715 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,717 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,719 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,720 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,722 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,724 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,725 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,727 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,729 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,730 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-01-03T16:18:49,733 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,733 copying build/lib/lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,735 copying build/lib/lm_eval/tasks/bbh/fewshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,737 copying build/lib/lm_eval/tasks/bbh/fewshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,739 copying build/lib/lm_eval/tasks/bbh/fewshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,741 copying build/lib/lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,742 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,744 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,746 copying build/lib/lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,748 copying build/lib/lm_eval/tasks/bbh/fewshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,749 copying build/lib/lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,751 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,753 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,755 copying build/lib/lm_eval/tasks/bbh/fewshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,757 copying build/lib/lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,759 copying build/lib/lm_eval/tasks/bbh/fewshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,760 copying build/lib/lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,762 copying build/lib/lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,764 copying build/lib/lm_eval/tasks/bbh/fewshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,765 copying build/lib/lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,767 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,769 copying build/lib/lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,770 copying build/lib/lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,772 copying build/lib/lm_eval/tasks/bbh/fewshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,773 copying build/lib/lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,775 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,777 copying build/lib/lm_eval/tasks/bbh/fewshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,779 copying build/lib/lm_eval/tasks/bbh/fewshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,780 copying build/lib/lm_eval/tasks/bbh/fewshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-01-03T16:18:49,782 copying build/lib/lm_eval/tasks/bbh/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-01-03T16:18:49,785 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,786 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,788 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,789 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,791 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,793 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,795 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,797 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,799 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,801 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,803 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,805 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,807 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,809 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,811 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,813 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,814 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,816 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,818 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,820 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,821 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,823 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,825 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,827 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,829 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,830 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,832 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,834 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,836 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-01-03T16:18:49,838 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-01-03T16:18:49,839 copying build/lib/lm_eval/tasks/mc_taco/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-01-03T16:18:49,841 copying build/lib/lm_eval/tasks/mc_taco/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-01-03T16:18:49,843 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-01-03T16:18:49,844 copying build/lib/lm_eval/tasks/asdiv/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-01-03T16:18:49,846 copying build/lib/lm_eval/tasks/asdiv/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-01-03T16:18:49,850 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,851 copying build/lib/lm_eval/tasks/belebele/belebele_bam_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,853 copying build/lib/lm_eval/tasks/belebele/belebele_bod_Tibt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,855 copying build/lib/lm_eval/tasks/belebele/belebele_gaz_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,857 copying build/lib/lm_eval/tasks/belebele/belebele_ilo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,858 copying build/lib/lm_eval/tasks/belebele/belebele_tel_Telu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,860 copying build/lib/lm_eval/tasks/belebele/belebele_nya_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,862 copying build/lib/lm_eval/tasks/belebele/belebele_mar_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,863 copying build/lib/lm_eval/tasks/belebele/belebele_khm_Khmr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,865 copying build/lib/lm_eval/tasks/belebele/belebele_yor_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,867 copying build/lib/lm_eval/tasks/belebele/belebele_kan_Knda.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,869 copying build/lib/lm_eval/tasks/belebele/belebele_apc_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,871 copying build/lib/lm_eval/tasks/belebele/belebele_als_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,874 copying build/lib/lm_eval/tasks/belebele/belebele_hat_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,876 copying build/lib/lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,878 copying build/lib/lm_eval/tasks/belebele/belebele_urd_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,880 copying build/lib/lm_eval/tasks/belebele/belebele_cat_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,882 copying build/lib/lm_eval/tasks/belebele/belebele_heb_Hebr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,884 copying build/lib/lm_eval/tasks/belebele/belebele_nld_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,886 copying build/lib/lm_eval/tasks/belebele/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,888 copying build/lib/lm_eval/tasks/belebele/belebele_som_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,890 copying build/lib/lm_eval/tasks/belebele/belebele_mri_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,892 copying build/lib/lm_eval/tasks/belebele/belebele_nso_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,895 copying build/lib/lm_eval/tasks/belebele/belebele_sin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,897 copying build/lib/lm_eval/tasks/belebele/belebele_hun_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,899 copying build/lib/lm_eval/tasks/belebele/belebele_ben_Beng.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,901 copying build/lib/lm_eval/tasks/belebele/belebele_kor_Hang.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,903 copying build/lib/lm_eval/tasks/belebele/belebele_ibo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,905 copying build/lib/lm_eval/tasks/belebele/belebele_tir_Ethi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,907 copying build/lib/lm_eval/tasks/belebele/belebele_zho_Hans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,910 copying build/lib/lm_eval/tasks/belebele/belebele_pan_Guru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,912 copying build/lib/lm_eval/tasks/belebele/belebele_tam_Taml.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,914 copying build/lib/lm_eval/tasks/belebele/belebele_pol_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,916 copying build/lib/lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,919 copying build/lib/lm_eval/tasks/belebele/belebele_por_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,921 copying build/lib/lm_eval/tasks/belebele/belebele_xho_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,923 copying build/lib/lm_eval/tasks/belebele/belebele_spa_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,925 copying build/lib/lm_eval/tasks/belebele/belebele_ces_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,927 copying build/lib/lm_eval/tasks/belebele/belebele_fra_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,929 copying build/lib/lm_eval/tasks/belebele/belebele_dan_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,931 copying build/lib/lm_eval/tasks/belebele/belebele_shn_Mymr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,933 copying build/lib/lm_eval/tasks/belebele/belebele_asm_Beng.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,935 copying build/lib/lm_eval/tasks/belebele/belebele_zul_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,937 copying build/lib/lm_eval/tasks/belebele/belebele_ory_Orya.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,939 copying build/lib/lm_eval/tasks/belebele/belebele_tgl_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,941 copying build/lib/lm_eval/tasks/belebele/belebele_hau_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,943 copying build/lib/lm_eval/tasks/belebele/belebele_wol_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,946 copying build/lib/lm_eval/tasks/belebele/belebele_luo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,948 copying build/lib/lm_eval/tasks/belebele/belebele_acm_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,950 copying build/lib/lm_eval/tasks/belebele/belebele_swe_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,952 copying build/lib/lm_eval/tasks/belebele/belebele_eng_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,954 copying build/lib/lm_eval/tasks/belebele/belebele_uzn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,956 copying build/lib/lm_eval/tasks/belebele/belebele_slk_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,958 copying build/lib/lm_eval/tasks/belebele/belebele_jav_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,960 copying build/lib/lm_eval/tasks/belebele/belebele_mya_Mymr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,963 copying build/lib/lm_eval/tasks/belebele/belebele_snd_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,964 copying build/lib/lm_eval/tasks/belebele/belebele_plt_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,966 copying build/lib/lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,968 copying build/lib/lm_eval/tasks/belebele/belebele_lit_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,970 copying build/lib/lm_eval/tasks/belebele/belebele_ars_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,972 copying build/lib/lm_eval/tasks/belebele/belebele_hin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,973 copying build/lib/lm_eval/tasks/belebele/belebele_kin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,975 copying build/lib/lm_eval/tasks/belebele/belebele_tsn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,977 copying build/lib/lm_eval/tasks/belebele/belebele_sot_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,978 copying build/lib/lm_eval/tasks/belebele/belebele_arb_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,980 copying build/lib/lm_eval/tasks/belebele/belebele_fin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,982 copying build/lib/lm_eval/tasks/belebele/belebele_npi_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,984 copying build/lib/lm_eval/tasks/belebele/belebele_lug_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,985 copying build/lib/lm_eval/tasks/belebele/belebele_tha_Thai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,987 copying build/lib/lm_eval/tasks/belebele/belebele_slv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,989 copying build/lib/lm_eval/tasks/belebele/belebele_isl_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,991 copying build/lib/lm_eval/tasks/belebele/belebele_urd_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:49,997 copying build/lib/lm_eval/tasks/belebele/belebele_npi_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,000 copying build/lib/lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,004 copying build/lib/lm_eval/tasks/belebele/belebele_azj_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,007 copying build/lib/lm_eval/tasks/belebele/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,009 copying build/lib/lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,011 copying build/lib/lm_eval/tasks/belebele/belebele_vie_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,014 copying build/lib/lm_eval/tasks/belebele/belebele_ceb_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,018 copying build/lib/lm_eval/tasks/belebele/belebele_zho_Hant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,021 copying build/lib/lm_eval/tasks/belebele/belebele_pes_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,023 copying build/lib/lm_eval/tasks/belebele/belebele_war_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,025 copying build/lib/lm_eval/tasks/belebele/belebele_kat_Geor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,026 copying build/lib/lm_eval/tasks/belebele/belebele_sun_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,028 copying build/lib/lm_eval/tasks/belebele/belebele_eus_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,030 copying build/lib/lm_eval/tasks/belebele/belebele_hrv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,032 copying build/lib/lm_eval/tasks/belebele/belebele_arz_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,033 copying build/lib/lm_eval/tasks/belebele/belebele_pbt_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,035 copying build/lib/lm_eval/tasks/belebele/belebele_ckb_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,037 copying build/lib/lm_eval/tasks/belebele/belebele_hin_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,039 copying build/lib/lm_eval/tasks/belebele/belebele_ell_Grek.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,040 copying build/lib/lm_eval/tasks/belebele/belebele_lin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,042 copying build/lib/lm_eval/tasks/belebele/belebele_hye_Armn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,043 copying build/lib/lm_eval/tasks/belebele/belebele_sna_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,045 copying build/lib/lm_eval/tasks/belebele/belebele_zsm_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,047 copying build/lib/lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,048 copying build/lib/lm_eval/tasks/belebele/belebele_tso_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,050 copying build/lib/lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,052 copying build/lib/lm_eval/tasks/belebele/belebele_grn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,053 copying build/lib/lm_eval/tasks/belebele/belebele_mlt_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,055 copying build/lib/lm_eval/tasks/belebele/belebele_ron_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,057 copying build/lib/lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,059 copying build/lib/lm_eval/tasks/belebele/belebele_ben_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,060 copying build/lib/lm_eval/tasks/belebele/belebele_ita_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,062 copying build/lib/lm_eval/tasks/belebele/belebele_tur_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,063 copying build/lib/lm_eval/tasks/belebele/belebele_sin_Sinh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,065 copying build/lib/lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,067 copying build/lib/lm_eval/tasks/belebele/belebele_est_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,069 copying build/lib/lm_eval/tasks/belebele/belebele_guj_Gujr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,071 copying build/lib/lm_eval/tasks/belebele/belebele_kac_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,073 copying build/lib/lm_eval/tasks/belebele/belebele_nob_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,074 copying build/lib/lm_eval/tasks/belebele/belebele_kea_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,076 copying build/lib/lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,078 copying build/lib/lm_eval/tasks/belebele/belebele_fuv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,080 copying build/lib/lm_eval/tasks/belebele/belebele_ary_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,081 copying build/lib/lm_eval/tasks/belebele/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,083 copying build/lib/lm_eval/tasks/belebele/belebele_ind_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,085 copying build/lib/lm_eval/tasks/belebele/belebele_lao_Laoo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,087 copying build/lib/lm_eval/tasks/belebele/belebele_swh_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,089 copying build/lib/lm_eval/tasks/belebele/belebele_ssw_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,090 copying build/lib/lm_eval/tasks/belebele/belebele_deu_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,092 copying build/lib/lm_eval/tasks/belebele/belebele_amh_Ethi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,093 copying build/lib/lm_eval/tasks/belebele/belebele_lvs_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,095 copying build/lib/lm_eval/tasks/belebele/belebele_mal_Mlym.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,096 copying build/lib/lm_eval/tasks/belebele/belebele_arb_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,098 copying build/lib/lm_eval/tasks/belebele/belebele_afr_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-01-03T16:18:50,100 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-01-03T16:18:50,101 copying build/lib/lm_eval/tasks/triviaqa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-01-03T16:18:50,103 copying build/lib/lm_eval/tasks/triviaqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-01-03T16:18:50,105 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-01-03T16:18:50,106 copying build/lib/lm_eval/tasks/webqs/webqs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-01-03T16:18:50,108 copying build/lib/lm_eval/tasks/webqs/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-01-03T16:18:50,110 copying build/lib/lm_eval/tasks/webqs/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-01-03T16:18:50,112 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,113 copying build/lib/lm_eval/tasks/translation/wmt14_fr-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,114 copying build/lib/lm_eval/tasks/translation/iwslt2017_en-ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,116 copying build/lib/lm_eval/tasks/translation/iwslt2017_ar-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,118 copying build/lib/lm_eval/tasks/translation/wmt16_ro-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,120 copying build/lib/lm_eval/tasks/translation/wmt16_en-ro.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,121 copying build/lib/lm_eval/tasks/translation/wmt_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,123 copying build/lib/lm_eval/tasks/translation/wmt16_en-de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,125 copying build/lib/lm_eval/tasks/translation/wmt14_en-fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,126 copying build/lib/lm_eval/tasks/translation/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,128 copying build/lib/lm_eval/tasks/translation/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,130 copying build/lib/lm_eval/tasks/translation/wmt16_de-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-01-03T16:18:50,132 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-01-03T16:18:50,133 copying build/lib/lm_eval/tasks/race/race.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-01-03T16:18:50,135 copying build/lib/lm_eval/tasks/race/preprocess_race.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-01-03T16:18:50,137 copying build/lib/lm_eval/tasks/race/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-01-03T16:18:50,139 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-01-03T16:18:50,140 copying build/lib/lm_eval/tasks/nq_open/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-01-03T16:18:50,142 copying build/lib/lm_eval/tasks/nq_open/nq_open.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-01-03T16:18:50,144 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,145 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,146 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,148 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,149 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,151 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,153 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,154 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,156 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,157 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,159 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,161 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,163 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,164 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,166 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,168 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,169 copying build/lib/lm_eval/tasks/crows_pairs/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,172 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,173 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,175 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,177 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,179 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,181 copying build/lib/lm_eval/tasks/crows_pairs/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,183 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,184 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-01-03T16:18:50,186 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals 2024-01-03T16:18:50,189 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,190 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,192 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,194 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,195 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,197 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,199 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,200 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,202 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,204 copying build/lib/lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,206 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,207 copying build/lib/lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,209 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,211 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,213 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,215 copying build/lib/lm_eval/tasks/model_written_evals/persona/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,216 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,218 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,220 copying build/lib/lm_eval/tasks/model_written_evals/persona/neuroticism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,221 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,223 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,224 copying build/lib/lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,226 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,228 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,229 copying build/lib/lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,231 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,232 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,234 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,236 copying build/lib/lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,237 copying build/lib/lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,239 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,241 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,242 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,244 copying build/lib/lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,246 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,248 copying build/lib/lm_eval/tasks/model_written_evals/persona/psychopathy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,249 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,251 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,253 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,255 copying build/lib/lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,257 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,258 copying build/lib/lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,260 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,261 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,263 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,265 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,266 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,268 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,269 copying build/lib/lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,271 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,273 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,275 copying build/lib/lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,276 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,278 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,280 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,281 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,283 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,285 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,287 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,288 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,290 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,291 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,293 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,294 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,296 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,297 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,299 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,301 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,302 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,304 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,306 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,308 copying build/lib/lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,309 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,311 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,313 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,315 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,316 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,318 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,320 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,322 copying build/lib/lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,323 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,325 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,326 copying build/lib/lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,328 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,329 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,331 copying build/lib/lm_eval/tasks/model_written_evals/persona/agreeableness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,333 copying build/lib/lm_eval/tasks/model_written_evals/persona/narcissism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,335 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,336 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,338 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,339 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,341 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,343 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,344 copying build/lib/lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,346 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,348 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,349 copying build/lib/lm_eval/tasks/model_written_evals/persona/self-replication.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,351 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,353 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,355 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,357 copying build/lib/lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,358 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,360 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,361 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,364 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,365 copying build/lib/lm_eval/tasks/model_written_evals/persona/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,367 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,368 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,370 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,372 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,374 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,375 copying build/lib/lm_eval/tasks/model_written_evals/persona/extraversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,377 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,379 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,380 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-averse.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,382 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,384 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,386 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,388 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,389 copying build/lib/lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,391 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,393 copying build/lib/lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,394 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,396 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,398 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,399 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,401 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,403 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,405 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,406 copying build/lib/lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,408 copying build/lib/lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,410 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,411 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,413 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,415 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,416 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,418 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,419 copying build/lib/lm_eval/tasks/model_written_evals/persona/openness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-01-03T16:18:50,421 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/winogenerated 2024-01-03T16:18:50,422 copying build/lib/lm_eval/tasks/model_written_evals/winogenerated/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/winogenerated 2024-01-03T16:18:50,425 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:50,426 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:50,428 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:50,429 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-01-03T16:18:50,432 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,432 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,434 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,436 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,437 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,439 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,440 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,442 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,444 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,445 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,447 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,449 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,451 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,452 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,454 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,456 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,457 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,459 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,461 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,462 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,464 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,465 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,467 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,469 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,470 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,472 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,474 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,475 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,477 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,479 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,481 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,483 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,484 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,486 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,488 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,490 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,492 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,493 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,495 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,497 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,498 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,500 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,501 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,503 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,505 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,507 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,508 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,510 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,511 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,513 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,515 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,517 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-01-03T16:18:50,519 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-01-03T16:18:50,520 copying build/lib/lm_eval/tasks/gsm8k/gsm8k-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-01-03T16:18:50,522 copying build/lib/lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-01-03T16:18:50,524 copying build/lib/lm_eval/tasks/gsm8k/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-01-03T16:18:50,526 copying build/lib/lm_eval/tasks/gsm8k/gsm8k.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-01-03T16:18:50,528 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-01-03T16:18:50,529 copying build/lib/lm_eval/tasks/wmt2016/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-01-03T16:18:50,530 copying build/lib/lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-01-03T16:18:50,532 copying build/lib/lm_eval/tasks/wmt2016/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-01-03T16:18:50,535 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,535 copying build/lib/lm_eval/tasks/ceval/ceval-valid_sports_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,537 copying build/lib/lm_eval/tasks/ceval/ceval-valid_marxism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,539 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,540 copying build/lib/lm_eval/tasks/ceval/ceval-valid_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,542 copying build/lib/lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,543 copying build/lib/lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,545 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,546 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,548 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,550 copying build/lib/lm_eval/tasks/ceval/_default_ceval_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,551 copying build/lib/lm_eval/tasks/ceval/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,553 copying build/lib/lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,555 copying build/lib/lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,556 copying build/lib/lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,558 copying build/lib/lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,560 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,561 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,563 copying build/lib/lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,565 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,566 copying build/lib/lm_eval/tasks/ceval/ceval-valid_physician.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,568 copying build/lib/lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,570 copying build/lib/lm_eval/tasks/ceval/ceval-valid_computer_network.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,572 copying build/lib/lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,573 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,575 copying build/lib/lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,577 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,578 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,580 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,581 copying build/lib/lm_eval/tasks/ceval/ceval-valid_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,583 copying build/lib/lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,584 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,586 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,587 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,589 copying build/lib/lm_eval/tasks/ceval/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,591 copying build/lib/lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,592 copying build/lib/lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,594 copying build/lib/lm_eval/tasks/ceval/ceval-valid_accountant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,596 copying build/lib/lm_eval/tasks/ceval/ceval-valid_operating_system.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,597 copying build/lib/lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,599 copying build/lib/lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,601 copying build/lib/lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,602 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,604 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,606 copying build/lib/lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,608 copying build/lib/lm_eval/tasks/ceval/ceval-valid_business_administration.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,609 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,611 copying build/lib/lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,613 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,614 copying build/lib/lm_eval/tasks/ceval/ceval-valid_art_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,616 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_programming.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,618 copying build/lib/lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,620 copying build/lib/lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,621 copying build/lib/lm_eval/tasks/ceval/ceval-valid_education_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,623 copying build/lib/lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,624 copying build/lib/lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-01-03T16:18:50,626 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-01-03T16:18:50,627 copying build/lib/lm_eval/tasks/scrolls/scrolls.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-01-03T16:18:50,629 copying build/lib/lm_eval/tasks/scrolls/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-01-03T16:18:50,631 copying build/lib/lm_eval/tasks/scrolls/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-01-03T16:18:50,634 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,635 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,637 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,638 copying build/lib/lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,640 copying build/lib/lm_eval/tasks/blimp/complex_NP_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,642 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,643 copying build/lib/lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,646 copying build/lib/lm_eval/tasks/blimp/only_npi_scope.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,647 copying build/lib/lm_eval/tasks/blimp/passive_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,649 copying build/lib/lm_eval/tasks/blimp/tough_vs_raising_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,651 copying build/lib/lm_eval/tasks/blimp/wh_questions_subject_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,652 copying build/lib/lm_eval/tasks/blimp/wh_questions_object_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,654 copying build/lib/lm_eval/tasks/blimp/sentential_subject_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,656 copying build/lib/lm_eval/tasks/blimp/transitive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,657 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,659 copying build/lib/lm_eval/tasks/blimp/wh_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,661 copying build/lib/lm_eval/tasks/blimp/animate_subject_passive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,662 copying build/lib/lm_eval/tasks/blimp/anaphor_gender_agreement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,664 copying build/lib/lm_eval/tasks/blimp/principle_A_reconstruction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,666 copying build/lib/lm_eval/tasks/blimp/principle_A_c_command.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,668 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,669 copying build/lib/lm_eval/tasks/blimp/intransitive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,671 copying build/lib/lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,672 copying build/lib/lm_eval/tasks/blimp/left_branch_island_echo_question.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,674 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,676 copying build/lib/lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,677 copying build/lib/lm_eval/tasks/blimp/anaphor_number_agreement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,679 copying build/lib/lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,681 copying build/lib/lm_eval/tasks/blimp/existential_there_object_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,682 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,684 copying build/lib/lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,686 copying build/lib/lm_eval/tasks/blimp/principle_A_case_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,687 copying build/lib/lm_eval/tasks/blimp/inchoative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,689 copying build/lib/lm_eval/tasks/blimp/superlative_quantifiers_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,691 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,693 copying build/lib/lm_eval/tasks/blimp/passive_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,695 copying build/lib/lm_eval/tasks/blimp/tough_vs_raising_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,696 copying build/lib/lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,698 copying build/lib/lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,699 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,701 copying build/lib/lm_eval/tasks/blimp/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,703 copying build/lib/lm_eval/tasks/blimp/npi_present_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,704 copying build/lib/lm_eval/tasks/blimp/left_branch_island_simple_question.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,706 copying build/lib/lm_eval/tasks/blimp/causative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,708 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,710 copying build/lib/lm_eval/tasks/blimp/existential_there_subject_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,711 copying build/lib/lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,713 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,715 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,716 copying build/lib/lm_eval/tasks/blimp/principle_A_case_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,718 copying build/lib/lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,719 copying build/lib/lm_eval/tasks/blimp/superlative_quantifiers_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,721 copying build/lib/lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,722 copying build/lib/lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,724 copying build/lib/lm_eval/tasks/blimp/adjunct_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,726 copying build/lib/lm_eval/tasks/blimp/only_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,727 copying build/lib/lm_eval/tasks/blimp/animate_subject_trans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,729 copying build/lib/lm_eval/tasks/blimp/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,731 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,732 copying build/lib/lm_eval/tasks/blimp/expletive_it_object_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,734 copying build/lib/lm_eval/tasks/blimp/generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,736 copying build/lib/lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,738 copying build/lib/lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,739 copying build/lib/lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,741 copying build/lib/lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,743 copying build/lib/lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,744 copying build/lib/lm_eval/tasks/blimp/drop_argument.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,746 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,748 copying build/lib/lm_eval/tasks/blimp/npi_present_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,749 copying build/lib/lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,751 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-01-03T16:18:50,753 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-01-03T16:18:50,754 copying build/lib/lm_eval/tasks/mutual/mutual.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-01-03T16:18:50,756 copying build/lib/lm_eval/tasks/mutual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-01-03T16:18:50,758 copying build/lib/lm_eval/tasks/mutual/multual_plus.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-01-03T16:18:50,759 copying build/lib/lm_eval/tasks/mutual/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-01-03T16:18:50,761 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,762 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,764 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,765 copying build/lib/lm_eval/tasks/truthfulqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,767 copying build/lib/lm_eval/tasks/truthfulqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,769 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-01-03T16:18:50,771 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,772 copying build/lib/lm_eval/tasks/xcopa/default_ta.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,773 copying build/lib/lm_eval/tasks/xcopa/default_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,775 copying build/lib/lm_eval/tasks/xcopa/default_tr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,777 copying build/lib/lm_eval/tasks/xcopa/default_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,778 copying build/lib/lm_eval/tasks/xcopa/default_et.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,780 copying build/lib/lm_eval/tasks/xcopa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,782 copying build/lib/lm_eval/tasks/xcopa/default_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,784 copying build/lib/lm_eval/tasks/xcopa/default_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,785 copying build/lib/lm_eval/tasks/xcopa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,787 copying build/lib/lm_eval/tasks/xcopa/default_qu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,789 copying build/lib/lm_eval/tasks/xcopa/default_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,791 copying build/lib/lm_eval/tasks/xcopa/default_ht.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,792 copying build/lib/lm_eval/tasks/xcopa/default_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-01-03T16:18:50,795 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:50,795 copying build/lib/lm_eval/tasks/realtoxicityprompts/metric.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:50,797 copying build/lib/lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-01-03T16:18:50,800 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue 2024-01-03T16:18:50,801 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:50,802 copying build/lib/lm_eval/tasks/super_glue/wic/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:50,804 copying build/lib/lm_eval/tasks/super_glue/wic/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-01-03T16:18:50,806 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:50,807 copying build/lib/lm_eval/tasks/super_glue/cb/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:50,809 copying build/lib/lm_eval/tasks/super_glue/cb/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:50,811 copying build/lib/lm_eval/tasks/super_glue/cb/aggregate.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:50,812 copying build/lib/lm_eval/tasks/super_glue/cb/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-01-03T16:18:50,815 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:50,815 copying build/lib/lm_eval/tasks/super_glue/rte/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:50,817 copying build/lib/lm_eval/tasks/super_glue/rte/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-01-03T16:18:50,819 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:50,820 copying build/lib/lm_eval/tasks/super_glue/boolq/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:50,821 copying build/lib/lm_eval/tasks/super_glue/boolq/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:50,823 copying build/lib/lm_eval/tasks/super_glue/boolq/seq2seq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-01-03T16:18:50,825 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-01-03T16:18:50,826 copying build/lib/lm_eval/tasks/super_glue/record/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-01-03T16:18:50,828 copying build/lib/lm_eval/tasks/super_glue/record/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-01-03T16:18:50,829 copying build/lib/lm_eval/tasks/super_glue/record/util.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-01-03T16:18:50,831 copying build/lib/lm_eval/tasks/super_glue/record/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-01-03T16:18:50,833 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:50,834 copying build/lib/lm_eval/tasks/super_glue/wsc/preprocess_wsc.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:50,836 copying build/lib/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:50,838 copying build/lib/lm_eval/tasks/super_glue/wsc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:50,839 copying build/lib/lm_eval/tasks/super_glue/wsc/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-01-03T16:18:50,841 copying build/lib/lm_eval/tasks/super_glue/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue 2024-01-03T16:18:50,843 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:50,844 copying build/lib/lm_eval/tasks/super_glue/copa/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:50,846 copying build/lib/lm_eval/tasks/super_glue/copa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:50,848 copying build/lib/lm_eval/tasks/super_glue/copa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-01-03T16:18:50,850 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:50,851 copying build/lib/lm_eval/tasks/super_glue/multirc/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:50,853 copying build/lib/lm_eval/tasks/super_glue/multirc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:50,854 copying build/lib/lm_eval/tasks/super_glue/multirc/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-01-03T16:18:50,856 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:50,857 copying build/lib/lm_eval/tasks/bigbench/generate_until_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:50,859 copying build/lib/lm_eval/tasks/bigbench/push_bigbench_dataset.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:50,864 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,865 copying build/lib/lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,866 copying build/lib/lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,868 copying build/lib/lm_eval/tasks/bigbench/generate_until/codenames.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,869 copying build/lib/lm_eval/tasks/bigbench/generate_until/multiemo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,871 copying build/lib/lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,873 copying build/lib/lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,874 copying build/lib/lm_eval/tasks/bigbench/generate_until/color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,876 copying build/lib/lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,878 copying build/lib/lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,880 copying build/lib/lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,881 copying build/lib/lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,883 copying build/lib/lm_eval/tasks/bigbench/generate_until/kannada.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,884 copying build/lib/lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,886 copying build/lib/lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,888 copying build/lib/lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,889 copying build/lib/lm_eval/tasks/bigbench/generate_until/gem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,891 copying build/lib/lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,893 copying build/lib/lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,894 copying build/lib/lm_eval/tasks/bigbench/generate_until/strange_stories.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,896 copying build/lib/lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,898 copying build/lib/lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,900 copying build/lib/lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,901 copying build/lib/lm_eval/tasks/bigbench/generate_until/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,903 copying build/lib/lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,905 copying build/lib/lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,906 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_args.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,908 copying build/lib/lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,910 copying build/lib/lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,911 copying build/lib/lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,913 copying build/lib/lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,914 copying build/lib/lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,916 copying build/lib/lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,918 copying build/lib/lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,919 copying build/lib/lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,921 copying build/lib/lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,923 copying build/lib/lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,924 copying build/lib/lm_eval/tasks/bigbench/generate_until/arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,926 copying build/lib/lm_eval/tasks/bigbench/generate_until/fact_checker.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,928 copying build/lib/lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,929 copying build/lib/lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,931 copying build/lib/lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,933 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,935 copying build/lib/lm_eval/tasks/bigbench/generate_until/language_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,937 copying build/lib/lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,938 copying build/lib/lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,940 copying build/lib/lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,942 copying build/lib/lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,943 copying build/lib/lm_eval/tasks/bigbench/generate_until/physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,945 copying build/lib/lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,947 copying build/lib/lm_eval/tasks/bigbench/generate_until/rephrase.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,948 copying build/lib/lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,950 copying build/lib/lm_eval/tasks/bigbench/generate_until/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,951 copying build/lib/lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,953 copying build/lib/lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,955 copying build/lib/lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,956 copying build/lib/lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,958 copying build/lib/lm_eval/tasks/bigbench/generate_until/physics_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,960 copying build/lib/lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,963 copying build/lib/lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,964 copying build/lib/lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,966 copying build/lib/lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,968 copying build/lib/lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,969 copying build/lib/lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,971 copying build/lib/lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,973 copying build/lib/lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,974 copying build/lib/lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,976 copying build/lib/lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,978 copying build/lib/lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,980 copying build/lib/lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,981 copying build/lib/lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,983 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,984 copying build/lib/lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,986 copying build/lib/lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,987 copying build/lib/lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,989 copying build/lib/lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,990 copying build/lib/lm_eval/tasks/bigbench/generate_until/winowhy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,992 copying build/lib/lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,994 copying build/lib/lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,995 copying build/lib/lm_eval/tasks/bigbench/generate_until/strategyqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,997 copying build/lib/lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:50,999 copying build/lib/lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,000 copying build/lib/lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,002 copying build/lib/lm_eval/tasks/bigbench/generate_until/misconceptions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,004 copying build/lib/lm_eval/tasks/bigbench/generate_until/implicatures.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,005 copying build/lib/lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,007 copying build/lib/lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,009 copying build/lib/lm_eval/tasks/bigbench/generate_until/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,010 copying build/lib/lm_eval/tasks/bigbench/generate_until/social_support.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,012 copying build/lib/lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,014 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,016 copying build/lib/lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,017 copying build/lib/lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,019 copying build/lib/lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,020 copying build/lib/lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,022 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,023 copying build/lib/lm_eval/tasks/bigbench/generate_until/anachronisms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,025 copying build/lib/lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,027 copying build/lib/lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,028 copying build/lib/lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,030 copying build/lib/lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,031 copying build/lib/lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,033 copying build/lib/lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,035 copying build/lib/lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,036 copying build/lib/lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,038 copying build/lib/lm_eval/tasks/bigbench/generate_until/crass_ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,040 copying build/lib/lm_eval/tasks/bigbench/generate_until/cryptonite.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,042 copying build/lib/lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,043 copying build/lib/lm_eval/tasks/bigbench/generate_until/social_iqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,045 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,047 copying build/lib/lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,049 copying build/lib/lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,050 copying build/lib/lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,052 copying build/lib/lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,054 copying build/lib/lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,056 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,058 copying build/lib/lm_eval/tasks/bigbench/generate_until/irony_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,059 copying build/lib/lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,061 copying build/lib/lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,062 copying build/lib/lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,064 copying build/lib/lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,065 copying build/lib/lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,067 copying build/lib/lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,068 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,070 copying build/lib/lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,072 copying build/lib/lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,074 copying build/lib/lm_eval/tasks/bigbench/generate_until/timedial.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,076 copying build/lib/lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,078 copying build/lib/lm_eval/tasks/bigbench/generate_until/tense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,079 copying build/lib/lm_eval/tasks/bigbench/generate_until/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,081 copying build/lib/lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,083 copying build/lib/lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,084 copying build/lib/lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,086 copying build/lib/lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,088 copying build/lib/lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,090 copying build/lib/lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,091 copying build/lib/lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,093 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,095 copying build/lib/lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,096 copying build/lib/lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,098 copying build/lib/lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,100 copying build/lib/lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,101 copying build/lib/lm_eval/tasks/bigbench/generate_until/operators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,103 copying build/lib/lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,105 copying build/lib/lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,107 copying build/lib/lm_eval/tasks/bigbench/generate_until/list_functions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,108 copying build/lib/lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,110 copying build/lib/lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,112 copying build/lib/lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,113 copying build/lib/lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,115 copying build/lib/lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,117 copying build/lib/lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,119 copying build/lib/lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,120 copying build/lib/lm_eval/tasks/bigbench/generate_until/question_selection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,122 copying build/lib/lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,124 copying build/lib/lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,126 copying build/lib/lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,127 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,129 copying build/lib/lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,131 copying build/lib/lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,132 copying build/lib/lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,134 copying build/lib/lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,136 copying build/lib/lm_eval/tasks/bigbench/generate_until/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,137 copying build/lib/lm_eval/tasks/bigbench/generate_until/code_line_description.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,139 copying build/lib/lm_eval/tasks/bigbench/generate_until/language_games.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,140 copying build/lib/lm_eval/tasks/bigbench/generate_until/topical_chat.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,142 copying build/lib/lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,143 copying build/lib/lm_eval/tasks/bigbench/generate_until/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-01-03T16:18:51,145 copying build/lib/lm_eval/tasks/bigbench/generate_tasks.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:51,147 copying build/lib/lm_eval/tasks/bigbench/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:51,152 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,153 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,155 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,156 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/codenames.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,158 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,160 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,162 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,163 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,165 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,167 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,169 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,170 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,172 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/kannada.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,174 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,176 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,177 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,179 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,180 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,182 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,184 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,185 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,187 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,188 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,190 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,192 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,193 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,195 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,197 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,199 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,200 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,202 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,203 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,205 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,207 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,208 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,210 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,212 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,214 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,215 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,217 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,218 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,220 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,221 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,223 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,224 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,226 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,228 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,229 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,231 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,233 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,235 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,236 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,238 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,240 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,241 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,243 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,245 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,247 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,248 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,250 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,252 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,253 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,255 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,256 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,258 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,259 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,261 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,263 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,264 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,266 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,268 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,269 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,271 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,272 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,274 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,276 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,278 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,279 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,281 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,283 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,284 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,286 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,288 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,289 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,291 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,292 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,294 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,296 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,297 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/social_support.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,299 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,300 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,302 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,303 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,305 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,307 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,308 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,310 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,312 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,313 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,315 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,317 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,318 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,320 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,322 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,324 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,325 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,327 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,328 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,330 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,331 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,333 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,334 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,336 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,338 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,339 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,341 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,342 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,344 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,346 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,348 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,349 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,351 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,353 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,355 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,356 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,358 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,360 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/timedial.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,361 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,363 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/tense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,365 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,366 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,368 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,369 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,371 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,372 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,374 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,376 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,377 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,379 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,381 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,382 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,384 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,386 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/operators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,388 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,389 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,391 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,393 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,394 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,396 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,398 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,400 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,401 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,403 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,404 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,406 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,408 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,409 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,411 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,412 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,414 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,416 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,417 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,419 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,421 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,422 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/language_games.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,424 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,426 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,428 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-01-03T16:18:51,429 copying build/lib/lm_eval/tasks/bigbench/multiple_choice_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-01-03T16:18:51,431 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-01-03T16:18:51,432 copying build/lib/lm_eval/tasks/drop/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-01-03T16:18:51,434 copying build/lib/lm_eval/tasks/drop/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-01-03T16:18:51,436 copying build/lib/lm_eval/tasks/drop/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-01-03T16:18:51,439 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,440 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,442 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,444 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,445 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,447 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,448 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,449 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,451 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,453 copying build/lib/lm_eval/tasks/cmmlu/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,455 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,456 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,458 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,459 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,461 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,462 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,464 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,466 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,468 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,469 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,471 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,473 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,475 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,476 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,478 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,480 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,481 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,483 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,485 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,486 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,488 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,489 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,491 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,492 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,494 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,496 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,497 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,499 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,500 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,502 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,503 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,505 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,507 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,509 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,511 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,512 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,514 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,516 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,517 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,519 copying build/lib/lm_eval/tasks/cmmlu/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,521 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,523 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,524 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,526 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,528 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,530 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,531 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,533 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,534 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,536 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,537 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,539 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,541 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,542 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,544 copying build/lib/lm_eval/tasks/cmmlu/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,545 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,547 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,549 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,550 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,552 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,554 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-01-03T16:18:51,556 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-01-03T16:18:51,557 copying build/lib/lm_eval/tasks/logiqa2/logieval.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-01-03T16:18:51,559 copying build/lib/lm_eval/tasks/logiqa2/utils_logiqa2.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-01-03T16:18:51,560 copying build/lib/lm_eval/tasks/logiqa2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-01-03T16:18:51,562 copying build/lib/lm_eval/tasks/logiqa2/logiqa2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-01-03T16:18:51,564 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,565 copying build/lib/lm_eval/tasks/unscramble/reversed_words.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,567 copying build/lib/lm_eval/tasks/unscramble/anagrams1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,569 copying build/lib/lm_eval/tasks/unscramble/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,571 copying build/lib/lm_eval/tasks/unscramble/cycle_letters.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,572 copying build/lib/lm_eval/tasks/unscramble/random_insertion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,574 copying build/lib/lm_eval/tasks/unscramble/anagrams2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-01-03T16:18:51,576 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-01-03T16:18:51,577 copying build/lib/lm_eval/tasks/coqa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-01-03T16:18:51,579 copying build/lib/lm_eval/tasks/coqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-01-03T16:18:51,580 copying build/lib/lm_eval/tasks/coqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-01-03T16:18:51,582 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-01-03T16:18:51,583 copying build/lib/lm_eval/tasks/prost/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-01-03T16:18:51,585 copying build/lib/lm_eval/tasks/prost/corypaik_prost.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-01-03T16:18:51,587 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-01-03T16:18:51,588 copying build/lib/lm_eval/tasks/wikitext/wikitext.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-01-03T16:18:51,589 copying build/lib/lm_eval/tasks/wikitext/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-01-03T16:18:51,591 copying build/lib/lm_eval/tasks/wikitext/preprocess_wikitext.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-01-03T16:18:51,593 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-01-03T16:18:51,594 copying build/lib/lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-01-03T16:18:51,596 copying build/lib/lm_eval/tasks/pubmedqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-01-03T16:18:51,598 copying build/lib/lm_eval/tasks/pubmedqa/pubmedqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-01-03T16:18:51,600 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,601 copying build/lib/lm_eval/tasks/paws-x/paws_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,603 copying build/lib/lm_eval/tasks/paws-x/paws_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,604 copying build/lib/lm_eval/tasks/paws-x/paws_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,606 copying build/lib/lm_eval/tasks/paws-x/pawsx_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,608 copying build/lib/lm_eval/tasks/paws-x/paws_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,609 copying build/lib/lm_eval/tasks/paws-x/paws_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,611 copying build/lib/lm_eval/tasks/paws-x/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,613 copying build/lib/lm_eval/tasks/paws-x/_generate_config.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,615 copying build/lib/lm_eval/tasks/paws-x/paws_ko.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,617 copying build/lib/lm_eval/tasks/paws-x/paws_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-01-03T16:18:51,619 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-01-03T16:18:51,620 copying build/lib/lm_eval/tasks/babi/babi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-01-03T16:18:51,622 copying build/lib/lm_eval/tasks/babi/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-01-03T16:18:51,624 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-01-03T16:18:51,625 copying build/lib/lm_eval/tasks/storycloze/storycloze_2018.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-01-03T16:18:51,627 copying build/lib/lm_eval/tasks/storycloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-01-03T16:18:51,629 copying build/lib/lm_eval/tasks/storycloze/storycloze_2016.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-01-03T16:18:51,631 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-01-03T16:18:51,632 copying build/lib/lm_eval/tasks/lambada/lambada_standard.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-01-03T16:18:51,633 copying build/lib/lm_eval/tasks/lambada/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-01-03T16:18:51,635 copying build/lib/lm_eval/tasks/lambada/lambada_openai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-01-03T16:18:51,637 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu 2024-01-03T16:18:51,638 copying build/lib/lm_eval/tasks/mmlu/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu 2024-01-03T16:18:51,641 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,641 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,643 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,645 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,646 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,648 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,649 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,651 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,653 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,655 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,656 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,658 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,659 copying build/lib/lm_eval/tasks/mmlu/default/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,661 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,663 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,664 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,666 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,668 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,669 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,671 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,673 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,675 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,677 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,678 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,680 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,681 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,683 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,684 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,686 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,687 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,689 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,691 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,692 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,694 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,696 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,697 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,699 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,701 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,702 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,704 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,706 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,707 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,709 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,711 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,712 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,714 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,716 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,718 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,719 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,721 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,723 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,724 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,726 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,727 copying build/lib/lm_eval/tasks/mmlu/default/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,729 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,731 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,732 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,734 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,735 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,737 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-01-03T16:18:51,740 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,741 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,743 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,745 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,746 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,748 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,749 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,751 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,753 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,755 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,756 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,758 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,760 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,762 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,764 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,765 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,767 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,768 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,770 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,771 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,773 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,775 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,776 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,778 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,780 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,781 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,783 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,785 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,786 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,788 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,790 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,791 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,793 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,795 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,796 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,798 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,800 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,802 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,803 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,805 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,806 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,808 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,809 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,811 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,813 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,814 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,816 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,817 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,819 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,821 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,823 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,824 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,826 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,828 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,829 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,831 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,833 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,835 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,837 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,839 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-01-03T16:18:51,841 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot 2024-01-03T16:18:51,843 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,844 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,845 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,847 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,849 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,851 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,852 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,854 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,856 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,858 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,860 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,861 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,863 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,865 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,867 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,869 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,871 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,872 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,875 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,876 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,878 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,879 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,881 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,883 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,884 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,886 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,887 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,889 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,891 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,892 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,894 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,896 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,898 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,899 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,901 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,903 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,904 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,906 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,908 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,909 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,911 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,913 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,915 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,916 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,918 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,920 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,921 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,923 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,925 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,927 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,928 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,930 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,932 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,934 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,935 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,937 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,939 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,941 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,942 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,944 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-01-03T16:18:51,947 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,948 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,950 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,952 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,953 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,955 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,957 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,959 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,960 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,962 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,963 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,965 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,967 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,968 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,970 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,972 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,973 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,975 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,977 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,979 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,980 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,982 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,984 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,986 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,987 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,989 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,991 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,993 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,994 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,996 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:51,998 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,000 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,001 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,003 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,005 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,007 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,009 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,010 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,012 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,014 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,015 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,017 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,019 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,021 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,023 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,024 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,026 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,027 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,029 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,031 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,033 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,035 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,036 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,038 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,040 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,041 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,043 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,045 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,047 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,048 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-01-03T16:18:52,051 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,052 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,054 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,056 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,058 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,060 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,062 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,064 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,066 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,068 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,070 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,071 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,073 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,075 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,077 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,079 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,080 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,083 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,085 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,086 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,088 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,090 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,092 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,094 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,096 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,098 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,100 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,102 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,104 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,106 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,108 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,109 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,111 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,113 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,115 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,117 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,119 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,120 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,123 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,125 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,127 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,129 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,130 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,133 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,134 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,136 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,138 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,140 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,142 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,144 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,146 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,151 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,154 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,156 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,157 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,160 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,161 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,163 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,165 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,167 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,169 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-01-03T16:18:52,172 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-01-03T16:18:52,172 copying build/lib/lm_eval/tasks/wsc273/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-01-03T16:18:52,174 copying build/lib/lm_eval/tasks/wsc273/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-01-03T16:18:52,176 copying build/lib/lm_eval/tasks/wsc273/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-01-03T16:18:52,178 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-01-03T16:18:52,179 copying build/lib/lm_eval/tasks/siqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-01-03T16:18:52,181 copying build/lib/lm_eval/tasks/siqa/default.yml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-01-03T16:18:52,183 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-01-03T16:18:52,184 copying build/lib/lm_eval/tasks/sciq/sciq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-01-03T16:18:52,186 copying build/lib/lm_eval/tasks/sciq/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-01-03T16:18:52,188 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-01-03T16:18:52,189 copying build/lib/lm_eval/tasks/arc/arc_challenge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-01-03T16:18:52,190 copying build/lib/lm_eval/tasks/arc/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-01-03T16:18:52,192 copying build/lib/lm_eval/tasks/arc/arc_easy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-01-03T16:18:52,194 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-01-03T16:18:52,196 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,197 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,199 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,200 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,202 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,204 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,205 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,207 copying build/lib/lm_eval/tasks/mgsm/native_cot/cot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,209 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,210 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,212 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,213 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,215 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-01-03T16:18:52,217 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,218 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_th_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,220 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_ja_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,221 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_fr_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,223 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_zh_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,225 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_de_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,227 copying build/lib/lm_eval/tasks/mgsm/en_cot/cot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,228 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_es_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,230 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_sw_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,231 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_te_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,233 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_ru_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,235 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_bn_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,237 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_en-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-01-03T16:18:52,238 copying build/lib/lm_eval/tasks/mgsm/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-01-03T16:18:52,240 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,241 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,244 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,245 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,247 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,249 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,250 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,252 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,253 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,255 copying build/lib/lm_eval/tasks/mgsm/direct/direct_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,257 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,258 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,260 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-01-03T16:18:52,261 copying build/lib/lm_eval/tasks/mgsm/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-01-03T16:18:52,263 copying build/lib/lm_eval/tasks/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks 2024-01-03T16:18:52,266 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,266 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_3da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,268 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,270 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_3ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,271 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_4da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,273 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_5ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,275 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_4ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,277 copying build/lib/lm_eval/tasks/arithmetic/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,279 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2dm.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,280 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_1dc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,282 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_5da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,284 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-01-03T16:18:52,286 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue 2024-01-03T16:18:52,287 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,288 copying build/lib/lm_eval/tasks/code_x_glue/code-text/go.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,290 copying build/lib/lm_eval/tasks/code_x_glue/code-text/ruby.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,292 copying build/lib/lm_eval/tasks/code_x_glue/code-text/php.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,293 copying build/lib/lm_eval/tasks/code_x_glue/code-text/python.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,295 copying build/lib/lm_eval/tasks/code_x_glue/code-text/bleu.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,297 copying build/lib/lm_eval/tasks/code_x_glue/code-text/java.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,299 copying build/lib/lm_eval/tasks/code_x_glue/code-text/javascript.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,301 copying build/lib/lm_eval/tasks/code_x_glue/code-text/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-01-03T16:18:52,303 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,303 copying build/lib/lm_eval/tasks/hendrycks_ethics/justice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,305 copying build/lib/lm_eval/tasks/hendrycks_ethics/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,307 copying build/lib/lm_eval/tasks/hendrycks_ethics/commonsense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,309 copying build/lib/lm_eval/tasks/hendrycks_ethics/deontology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,310 copying build/lib/lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,312 copying build/lib/lm_eval/tasks/hendrycks_ethics/virtue.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,314 copying build/lib/lm_eval/tasks/hendrycks_ethics/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,315 copying build/lib/lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-01-03T16:18:52,318 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,319 copying build/lib/lm_eval/tasks/pile/pile_gutenberg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,320 copying build/lib/lm_eval/tasks/pile/pile_youtubesubtitles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,322 copying build/lib/lm_eval/tasks/pile/pile_europarl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,324 copying build/lib/lm_eval/tasks/pile/pile_wikipedia.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,325 copying build/lib/lm_eval/tasks/pile/pile_books3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,327 copying build/lib/lm_eval/tasks/pile/pile_github.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,328 copying build/lib/lm_eval/tasks/pile/pile_bookcorpus2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,330 copying build/lib/lm_eval/tasks/pile/pile_enron.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,332 copying build/lib/lm_eval/tasks/pile/pile_dm-mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,333 copying build/lib/lm_eval/tasks/pile/pile_nih-exporter.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,335 copying build/lib/lm_eval/tasks/pile/pile_arxiv.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,337 copying build/lib/lm_eval/tasks/pile/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,339 copying build/lib/lm_eval/tasks/pile/pile_pubmed-central.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,340 copying build/lib/lm_eval/tasks/pile/pile_ubuntu-irc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,342 copying build/lib/lm_eval/tasks/pile/pile_stackexchange.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,343 copying build/lib/lm_eval/tasks/pile/pile_pile-cc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,345 copying build/lib/lm_eval/tasks/pile/pile_pubmed-abstracts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,347 copying build/lib/lm_eval/tasks/pile/pile_opensubtitles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,348 copying build/lib/lm_eval/tasks/pile/pile_openwebtext2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,350 copying build/lib/lm_eval/tasks/pile/pile_uspto.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,351 copying build/lib/lm_eval/tasks/pile/pile_hackernews.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,353 copying build/lib/lm_eval/tasks/pile/pile_freelaw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,354 copying build/lib/lm_eval/tasks/pile/pile_philpapers.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-01-03T16:18:52,356 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-01-03T16:18:52,357 copying build/lib/lm_eval/tasks/openbookqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-01-03T16:18:52,359 copying build/lib/lm_eval/tasks/openbookqa/openbookqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-01-03T16:18:52,361 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-01-03T16:18:52,361 copying build/lib/lm_eval/tasks/anli/anli_r3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-01-03T16:18:52,363 copying build/lib/lm_eval/tasks/anli/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-01-03T16:18:52,365 copying build/lib/lm_eval/tasks/anli/anli_r2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-01-03T16:18:52,367 copying build/lib/lm_eval/tasks/anli/anli_r1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-01-03T16:18:52,368 copying build/lib/lm_eval/tasks/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks 2024-01-03T16:18:52,371 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,372 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,373 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,375 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_pt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,377 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,378 copying build/lib/lm_eval/tasks/xwinograd/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,380 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,381 copying build/lib/lm_eval/tasks/xwinograd/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,383 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,385 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_jp.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-01-03T16:18:52,387 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,388 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,390 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,392 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,394 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,396 copying build/lib/lm_eval/tasks/lambada_multilingual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,398 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-01-03T16:18:52,399 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-01-03T16:18:52,400 copying build/lib/lm_eval/tasks/polemo2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-01-03T16:18:52,402 copying build/lib/lm_eval/tasks/polemo2/polemo2_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-01-03T16:18:52,404 copying build/lib/lm_eval/tasks/polemo2/polemo2_in.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-01-03T16:18:52,406 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,406 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,408 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,410 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,411 copying build/lib/lm_eval/tasks/minerva_math/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,413 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,415 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,416 copying build/lib/lm_eval/tasks/minerva_math/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,418 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,420 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_precalc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-01-03T16:18:52,422 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,423 copying build/lib/lm_eval/tasks/csatqa/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,425 copying build/lib/lm_eval/tasks/csatqa/csatqa_rcss.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,427 copying build/lib/lm_eval/tasks/csatqa/csatqa_gr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,429 copying build/lib/lm_eval/tasks/csatqa/csatqa_li.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,430 copying build/lib/lm_eval/tasks/csatqa/csatqa_rcs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,432 copying build/lib/lm_eval/tasks/csatqa/_default_csatqa_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,433 copying build/lib/lm_eval/tasks/csatqa/csatqa_rch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,435 copying build/lib/lm_eval/tasks/csatqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,437 copying build/lib/lm_eval/tasks/csatqa/csatqa_wr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-01-03T16:18:52,439 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,440 copying build/lib/lm_eval/tasks/xnli/xnli_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,442 copying build/lib/lm_eval/tasks/xnli/xnli_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,444 copying build/lib/lm_eval/tasks/xnli/xnli_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,446 copying build/lib/lm_eval/tasks/xnli/xnli_tr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,447 copying build/lib/lm_eval/tasks/xnli/xnli_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,449 copying build/lib/lm_eval/tasks/xnli/xnli_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,450 copying build/lib/lm_eval/tasks/xnli/xnli_el.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,452 copying build/lib/lm_eval/tasks/xnli/xnli_ur.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,453 copying build/lib/lm_eval/tasks/xnli/xnli_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,455 copying build/lib/lm_eval/tasks/xnli/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,456 copying build/lib/lm_eval/tasks/xnli/xnli_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,458 copying build/lib/lm_eval/tasks/xnli/xnli_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,460 copying build/lib/lm_eval/tasks/xnli/xnli_bg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,461 copying build/lib/lm_eval/tasks/xnli/xnli_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,463 copying build/lib/lm_eval/tasks/xnli/xnli_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,464 copying build/lib/lm_eval/tasks/xnli/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,467 copying build/lib/lm_eval/tasks/xnli/xnli_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,468 copying build/lib/lm_eval/tasks/xnli/xnli_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-01-03T16:18:52,470 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-01-03T16:18:52,471 copying build/lib/lm_eval/tasks/piqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-01-03T16:18:52,473 copying build/lib/lm_eval/tasks/piqa/piqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-01-03T16:18:52,475 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-01-03T16:18:52,476 copying build/lib/lm_eval/tasks/swag/swag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-01-03T16:18:52,478 copying build/lib/lm_eval/tasks/swag/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-01-03T16:18:52,480 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-01-03T16:18:52,481 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,482 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_arc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,485 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:52,485 copying build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:52,488 copying build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:52,489 copying build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:52,491 copying build/lib/lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/prompt_templates 2024-01-03T16:18:52,493 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,495 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:52,496 copying build/lib/lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:52,498 copying build/lib/lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan/yaml_templates 2024-01-03T16:18:52,499 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,501 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_held_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,502 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_held_in_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,504 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_anli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,506 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_boolq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,507 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_rte.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-01-03T16:18:52,509 copying build/lib/lm_eval/tasks/benchmarks/pythia.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-01-03T16:18:52,510 copying build/lib/lm_eval/tasks/benchmarks/t0_eval.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-01-03T16:18:52,512 copying build/lib/lm_eval/tasks/benchmarks/minerva_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-01-03T16:18:52,514 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-01-03T16:18:52,515 copying build/lib/lm_eval/tasks/headqa/headqa_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-01-03T16:18:52,516 copying build/lib/lm_eval/tasks/headqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-01-03T16:18:52,518 copying build/lib/lm_eval/tasks/headqa/headqa_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-01-03T16:18:52,520 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,521 copying build/lib/lm_eval/tasks/xstorycloze/default_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,523 copying build/lib/lm_eval/tasks/xstorycloze/default_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,524 copying build/lib/lm_eval/tasks/xstorycloze/default_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,526 copying build/lib/lm_eval/tasks/xstorycloze/default_eu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,528 copying build/lib/lm_eval/tasks/xstorycloze/default_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,529 copying build/lib/lm_eval/tasks/xstorycloze/default_my.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,531 copying build/lib/lm_eval/tasks/xstorycloze/default_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,533 copying build/lib/lm_eval/tasks/xstorycloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,535 copying build/lib/lm_eval/tasks/xstorycloze/default_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,536 copying build/lib/lm_eval/tasks/xstorycloze/default_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,538 copying build/lib/lm_eval/tasks/xstorycloze/default_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,540 copying build/lib/lm_eval/tasks/xstorycloze/default_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-01-03T16:18:52,542 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,543 copying build/lib/lm_eval/tasks/qasper/bool.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,545 copying build/lib/lm_eval/tasks/qasper/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,547 copying build/lib/lm_eval/tasks/qasper/freeform.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,548 copying build/lib/lm_eval/tasks/qasper/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,550 copying build/lib/lm_eval/tasks/qasper/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-01-03T16:18:52,552 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue 2024-01-03T16:18:52,553 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/sst 2024-01-03T16:18:52,554 copying build/lib/lm_eval/tasks/glue/sst/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/sst 2024-01-03T16:18:52,556 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-01-03T16:18:52,557 copying build/lib/lm_eval/tasks/glue/mnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-01-03T16:18:52,559 copying build/lib/lm_eval/tasks/glue/mnli/mismatch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-01-03T16:18:52,561 copying build/lib/lm_eval/tasks/glue/mnli/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-01-03T16:18:52,563 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/cola 2024-01-03T16:18:52,563 copying build/lib/lm_eval/tasks/glue/cola/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/cola 2024-01-03T16:18:52,566 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/rte 2024-01-03T16:18:52,566 copying build/lib/lm_eval/tasks/glue/rte/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/rte 2024-01-03T16:18:52,568 copying build/lib/lm_eval/tasks/glue/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue 2024-01-03T16:18:52,570 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qnli 2024-01-03T16:18:52,571 copying build/lib/lm_eval/tasks/glue/qnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qnli 2024-01-03T16:18:52,573 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/wnli 2024-01-03T16:18:52,574 copying build/lib/lm_eval/tasks/glue/wnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/wnli 2024-01-03T16:18:52,576 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qqp 2024-01-03T16:18:52,577 copying build/lib/lm_eval/tasks/glue/qqp/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qqp 2024-01-03T16:18:52,580 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mrpc 2024-01-03T16:18:52,581 copying build/lib/lm_eval/tasks/glue/mrpc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mrpc 2024-01-03T16:18:52,583 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-01-03T16:18:52,584 copying build/lib/lm_eval/tasks/logiqa/utils_logiqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-01-03T16:18:52,585 copying build/lib/lm_eval/tasks/logiqa/logiqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-01-03T16:18:52,587 copying build/lib/lm_eval/tasks/logiqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-01-03T16:18:52,589 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-01-03T16:18:52,590 copying build/lib/lm_eval/tasks/mathqa/mathqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-01-03T16:18:52,592 copying build/lib/lm_eval/tasks/mathqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-01-03T16:18:52,594 copying build/lib/lm_eval/tasks/mathqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-01-03T16:18:52,596 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:52,597 copying build/lib/lm_eval/tasks/lambada_cloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:52,599 copying build/lib/lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:52,600 copying build/lib/lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-01-03T16:18:52,602 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,603 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2013.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,605 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2012.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,606 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2011.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,608 copying build/lib/lm_eval/tasks/qa4mre/preprocess_qa4mre.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,609 copying build/lib/lm_eval/tasks/qa4mre/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-01-03T16:18:52,612 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-01-03T16:18:52,612 copying build/lib/lm_eval/tasks/winogrande/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-01-03T16:18:52,614 copying build/lib/lm_eval/tasks/winogrande/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-01-03T16:18:52,616 copying build/lib/lm_eval/tasks/winogrande/preprocess_winogrande.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-01-03T16:18:52,618 creating build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,619 copying build/lib/lm_eval/api/instance.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,621 copying build/lib/lm_eval/api/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,623 copying build/lib/lm_eval/api/model.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,625 copying build/lib/lm_eval/api/registry.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,627 copying build/lib/lm_eval/api/samplers.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,629 copying build/lib/lm_eval/api/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,632 copying build/lib/lm_eval/api/filter.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,635 copying build/lib/lm_eval/api/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-01-03T16:18:52,636 copying build/lib/lm_eval/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-01-03T16:18:52,639 creating build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,640 copying build/lib/lm_eval/filters/extraction.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,642 copying build/lib/lm_eval/filters/transformation.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,643 copying build/lib/lm_eval/filters/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,645 copying build/lib/lm_eval/filters/selection.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,647 copying build/lib/lm_eval/filters/decontamination.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-01-03T16:18:52,649 running install_egg_info 2024-01-03T16:18:52,652 Copying lm_eval.egg-info to build/bdist.linux-armv7l/wheel/lm_eval-0.4.0-py3.11.egg-info 2024-01-03T16:18:52,666 running install_scripts 2024-01-03T16:18:52,695 creating build/bdist.linux-armv7l/wheel/lm_eval-0.4.0.dist-info/WHEEL 2024-01-03T16:18:52,697 creating '/tmp/pip-wheel-bzbbg3lh/.tmp-tw8pjy7f/lm_eval-0.4.0-py3-none-any.whl' and adding 'build/bdist.linux-armv7l/wheel' to it 2024-01-03T16:18:52,700 adding 'lm_eval/__init__.py' 2024-01-03T16:18:52,702 adding 'lm_eval/__main__.py' 2024-01-03T16:18:52,705 adding 'lm_eval/evaluator.py' 2024-01-03T16:18:52,708 adding 'lm_eval/utils.py' 2024-01-03T16:18:52,710 adding 'lm_eval/api/__init__.py' 2024-01-03T16:18:52,711 adding 'lm_eval/api/filter.py' 2024-01-03T16:18:52,712 adding 'lm_eval/api/instance.py' 2024-01-03T16:18:52,714 adding 'lm_eval/api/metrics.py' 2024-01-03T16:18:52,716 adding 'lm_eval/api/model.py' 2024-01-03T16:18:52,717 adding 'lm_eval/api/registry.py' 2024-01-03T16:18:52,718 adding 'lm_eval/api/samplers.py' 2024-01-03T16:18:52,723 adding 'lm_eval/api/task.py' 2024-01-03T16:18:52,725 adding 'lm_eval/decontamination/__init__.py' 2024-01-03T16:18:52,726 adding 'lm_eval/decontamination/archiver.py' 2024-01-03T16:18:52,728 adding 'lm_eval/decontamination/decontaminate.py' 2024-01-03T16:18:52,730 adding 'lm_eval/decontamination/janitor.py' 2024-01-03T16:18:52,732 adding 'lm_eval/filters/__init__.py' 2024-01-03T16:18:52,733 adding 'lm_eval/filters/decontamination.py' 2024-01-03T16:18:52,734 adding 'lm_eval/filters/extraction.py' 2024-01-03T16:18:52,735 adding 'lm_eval/filters/selection.py' 2024-01-03T16:18:52,737 adding 'lm_eval/filters/transformation.py' 2024-01-03T16:18:52,738 adding 'lm_eval/models/__init__.py' 2024-01-03T16:18:52,740 adding 'lm_eval/models/anthropic_llms.py' 2024-01-03T16:18:52,741 adding 'lm_eval/models/dummy.py' 2024-01-03T16:18:52,742 adding 'lm_eval/models/gguf.py' 2024-01-03T16:18:52,747 adding 'lm_eval/models/huggingface.py' 2024-01-03T16:18:52,749 adding 'lm_eval/models/openai_completions.py' 2024-01-03T16:18:52,751 adding 'lm_eval/models/textsynth.py' 2024-01-03T16:18:52,753 adding 'lm_eval/models/vllm_causallms.py' 2024-01-03T16:18:52,755 adding 'lm_eval/prompts/__init__.py' 2024-01-03T16:18:52,758 adding 'lm_eval/tasks/README.md' 2024-01-03T16:18:52,759 adding 'lm_eval/tasks/__init__.py' 2024-01-03T16:18:52,761 adding 'lm_eval/tasks/anli/README.md' 2024-01-03T16:18:52,762 adding 'lm_eval/tasks/anli/anli_r1.yaml' 2024-01-03T16:18:52,764 adding 'lm_eval/tasks/anli/anli_r2.yaml' 2024-01-03T16:18:52,765 adding 'lm_eval/tasks/anli/anli_r3.yaml' 2024-01-03T16:18:52,766 adding 'lm_eval/tasks/arc/README.md' 2024-01-03T16:18:52,767 adding 'lm_eval/tasks/arc/arc_challenge.yaml' 2024-01-03T16:18:52,768 adding 'lm_eval/tasks/arc/arc_easy.yaml' 2024-01-03T16:18:52,770 adding 'lm_eval/tasks/arithmetic/README.md' 2024-01-03T16:18:52,771 adding 'lm_eval/tasks/arithmetic/arithmetic_1dc.yaml' 2024-01-03T16:18:52,773 adding 'lm_eval/tasks/arithmetic/arithmetic_2da.yaml' 2024-01-03T16:18:52,774 adding 'lm_eval/tasks/arithmetic/arithmetic_2dm.yaml' 2024-01-03T16:18:52,775 adding 'lm_eval/tasks/arithmetic/arithmetic_2ds.yaml' 2024-01-03T16:18:52,776 adding 'lm_eval/tasks/arithmetic/arithmetic_3da.yaml' 2024-01-03T16:18:52,777 adding 'lm_eval/tasks/arithmetic/arithmetic_3ds.yaml' 2024-01-03T16:18:52,778 adding 'lm_eval/tasks/arithmetic/arithmetic_4da.yaml' 2024-01-03T16:18:52,779 adding 'lm_eval/tasks/arithmetic/arithmetic_4ds.yaml' 2024-01-03T16:18:52,780 adding 'lm_eval/tasks/arithmetic/arithmetic_5da.yaml' 2024-01-03T16:18:52,782 adding 'lm_eval/tasks/arithmetic/arithmetic_5ds.yaml' 2024-01-03T16:18:52,783 adding 'lm_eval/tasks/asdiv/README.md' 2024-01-03T16:18:52,785 adding 'lm_eval/tasks/asdiv/default.yaml' 2024-01-03T16:18:52,786 adding 'lm_eval/tasks/babi/README.md' 2024-01-03T16:18:52,788 adding 'lm_eval/tasks/babi/babi.yaml' 2024-01-03T16:18:52,789 adding 'lm_eval/tasks/bbh/README.md' 2024-01-03T16:18:52,791 adding 'lm_eval/tasks/bbh/_generate_configs.py' 2024-01-03T16:18:52,793 adding 'lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml' 2024-01-03T16:18:52,794 adding 'lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml' 2024-01-03T16:18:52,796 adding 'lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml' 2024-01-03T16:18:52,797 adding 'lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml' 2024-01-03T16:18:52,799 adding 'lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml' 2024-01-03T16:18:52,800 adding 'lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml' 2024-01-03T16:18:52,801 adding 'lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml' 2024-01-03T16:18:52,803 adding 'lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml' 2024-01-03T16:18:52,804 adding 'lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml' 2024-01-03T16:18:52,805 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml' 2024-01-03T16:18:52,806 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml' 2024-01-03T16:18:52,808 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml' 2024-01-03T16:18:52,809 adding 'lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml' 2024-01-03T16:18:52,810 adding 'lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml' 2024-01-03T16:18:52,811 adding 'lm_eval/tasks/bbh/cot_fewshot/navigate.yaml' 2024-01-03T16:18:52,812 adding 'lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml' 2024-01-03T16:18:52,814 adding 'lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml' 2024-01-03T16:18:52,815 adding 'lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:52,816 adding 'lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml' 2024-01-03T16:18:52,818 adding 'lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml' 2024-01-03T16:18:52,819 adding 'lm_eval/tasks/bbh/cot_fewshot/snarks.yaml' 2024-01-03T16:18:52,820 adding 'lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml' 2024-01-03T16:18:52,822 adding 'lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml' 2024-01-03T16:18:52,823 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml' 2024-01-03T16:18:52,824 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml' 2024-01-03T16:18:52,825 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml' 2024-01-03T16:18:52,827 adding 'lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml' 2024-01-03T16:18:52,828 adding 'lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml' 2024-01-03T16:18:52,830 adding 'lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml' 2024-01-03T16:18:52,831 adding 'lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml' 2024-01-03T16:18:52,832 adding 'lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml' 2024-01-03T16:18:52,834 adding 'lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml' 2024-01-03T16:18:52,835 adding 'lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml' 2024-01-03T16:18:52,836 adding 'lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml' 2024-01-03T16:18:52,837 adding 'lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml' 2024-01-03T16:18:52,838 adding 'lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml' 2024-01-03T16:18:52,839 adding 'lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml' 2024-01-03T16:18:52,840 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml' 2024-01-03T16:18:52,841 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml' 2024-01-03T16:18:52,843 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml' 2024-01-03T16:18:52,844 adding 'lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml' 2024-01-03T16:18:52,845 adding 'lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml' 2024-01-03T16:18:52,846 adding 'lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml' 2024-01-03T16:18:52,847 adding 'lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml' 2024-01-03T16:18:52,848 adding 'lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml' 2024-01-03T16:18:52,850 adding 'lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:52,851 adding 'lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml' 2024-01-03T16:18:52,852 adding 'lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml' 2024-01-03T16:18:52,853 adding 'lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml' 2024-01-03T16:18:52,854 adding 'lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml' 2024-01-03T16:18:52,856 adding 'lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml' 2024-01-03T16:18:52,857 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml' 2024-01-03T16:18:52,858 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml' 2024-01-03T16:18:52,859 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml' 2024-01-03T16:18:52,860 adding 'lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml' 2024-01-03T16:18:52,861 adding 'lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml' 2024-01-03T16:18:52,863 adding 'lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml' 2024-01-03T16:18:52,864 adding 'lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml' 2024-01-03T16:18:52,866 adding 'lm_eval/tasks/bbh/fewshot/causal_judgement.yaml' 2024-01-03T16:18:52,867 adding 'lm_eval/tasks/bbh/fewshot/date_understanding.yaml' 2024-01-03T16:18:52,868 adding 'lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml' 2024-01-03T16:18:52,869 adding 'lm_eval/tasks/bbh/fewshot/dyck_languages.yaml' 2024-01-03T16:18:52,870 adding 'lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml' 2024-01-03T16:18:52,872 adding 'lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml' 2024-01-03T16:18:52,873 adding 'lm_eval/tasks/bbh/fewshot/hyperbaton.yaml' 2024-01-03T16:18:52,874 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml' 2024-01-03T16:18:52,875 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml' 2024-01-03T16:18:52,876 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml' 2024-01-03T16:18:52,877 adding 'lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml' 2024-01-03T16:18:52,878 adding 'lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml' 2024-01-03T16:18:52,880 adding 'lm_eval/tasks/bbh/fewshot/navigate.yaml' 2024-01-03T16:18:52,881 adding 'lm_eval/tasks/bbh/fewshot/object_counting.yaml' 2024-01-03T16:18:52,882 adding 'lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml' 2024-01-03T16:18:52,883 adding 'lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:52,884 adding 'lm_eval/tasks/bbh/fewshot/ruin_names.yaml' 2024-01-03T16:18:52,886 adding 'lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml' 2024-01-03T16:18:52,887 adding 'lm_eval/tasks/bbh/fewshot/snarks.yaml' 2024-01-03T16:18:52,888 adding 'lm_eval/tasks/bbh/fewshot/sports_understanding.yaml' 2024-01-03T16:18:52,889 adding 'lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml' 2024-01-03T16:18:52,891 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml' 2024-01-03T16:18:52,892 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml' 2024-01-03T16:18:52,893 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml' 2024-01-03T16:18:52,894 adding 'lm_eval/tasks/bbh/fewshot/web_of_lies.yaml' 2024-01-03T16:18:52,896 adding 'lm_eval/tasks/bbh/fewshot/word_sorting.yaml' 2024-01-03T16:18:52,898 adding 'lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml' 2024-01-03T16:18:52,899 adding 'lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml' 2024-01-03T16:18:52,900 adding 'lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml' 2024-01-03T16:18:52,901 adding 'lm_eval/tasks/bbh/zeroshot/date_understanding.yaml' 2024-01-03T16:18:52,902 adding 'lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml' 2024-01-03T16:18:52,903 adding 'lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml' 2024-01-03T16:18:52,905 adding 'lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml' 2024-01-03T16:18:52,906 adding 'lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml' 2024-01-03T16:18:52,907 adding 'lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml' 2024-01-03T16:18:52,908 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml' 2024-01-03T16:18:52,909 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml' 2024-01-03T16:18:52,910 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml' 2024-01-03T16:18:52,912 adding 'lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml' 2024-01-03T16:18:52,913 adding 'lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml' 2024-01-03T16:18:52,914 adding 'lm_eval/tasks/bbh/zeroshot/navigate.yaml' 2024-01-03T16:18:52,915 adding 'lm_eval/tasks/bbh/zeroshot/object_counting.yaml' 2024-01-03T16:18:52,916 adding 'lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml' 2024-01-03T16:18:52,917 adding 'lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:52,918 adding 'lm_eval/tasks/bbh/zeroshot/ruin_names.yaml' 2024-01-03T16:18:52,919 adding 'lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml' 2024-01-03T16:18:52,920 adding 'lm_eval/tasks/bbh/zeroshot/snarks.yaml' 2024-01-03T16:18:52,921 adding 'lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml' 2024-01-03T16:18:52,922 adding 'lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml' 2024-01-03T16:18:52,923 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml' 2024-01-03T16:18:52,924 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml' 2024-01-03T16:18:52,926 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml' 2024-01-03T16:18:52,927 adding 'lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml' 2024-01-03T16:18:52,928 adding 'lm_eval/tasks/bbh/zeroshot/word_sorting.yaml' 2024-01-03T16:18:52,931 adding 'lm_eval/tasks/belebele/README.md' 2024-01-03T16:18:52,933 adding 'lm_eval/tasks/belebele/_default_template_yaml' 2024-01-03T16:18:52,934 adding 'lm_eval/tasks/belebele/_generate_configs.py' 2024-01-03T16:18:52,935 adding 'lm_eval/tasks/belebele/belebele_acm_Arab.yaml' 2024-01-03T16:18:52,936 adding 'lm_eval/tasks/belebele/belebele_afr_Latn.yaml' 2024-01-03T16:18:52,937 adding 'lm_eval/tasks/belebele/belebele_als_Latn.yaml' 2024-01-03T16:18:52,938 adding 'lm_eval/tasks/belebele/belebele_amh_Ethi.yaml' 2024-01-03T16:18:52,939 adding 'lm_eval/tasks/belebele/belebele_apc_Arab.yaml' 2024-01-03T16:18:52,940 adding 'lm_eval/tasks/belebele/belebele_arb_Arab.yaml' 2024-01-03T16:18:52,942 adding 'lm_eval/tasks/belebele/belebele_arb_Latn.yaml' 2024-01-03T16:18:52,943 adding 'lm_eval/tasks/belebele/belebele_ars_Arab.yaml' 2024-01-03T16:18:52,945 adding 'lm_eval/tasks/belebele/belebele_ary_Arab.yaml' 2024-01-03T16:18:52,946 adding 'lm_eval/tasks/belebele/belebele_arz_Arab.yaml' 2024-01-03T16:18:52,947 adding 'lm_eval/tasks/belebele/belebele_asm_Beng.yaml' 2024-01-03T16:18:52,948 adding 'lm_eval/tasks/belebele/belebele_azj_Latn.yaml' 2024-01-03T16:18:52,949 adding 'lm_eval/tasks/belebele/belebele_bam_Latn.yaml' 2024-01-03T16:18:52,950 adding 'lm_eval/tasks/belebele/belebele_ben_Beng.yaml' 2024-01-03T16:18:52,951 adding 'lm_eval/tasks/belebele/belebele_ben_Latn.yaml' 2024-01-03T16:18:52,952 adding 'lm_eval/tasks/belebele/belebele_bod_Tibt.yaml' 2024-01-03T16:18:52,953 adding 'lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml' 2024-01-03T16:18:52,955 adding 'lm_eval/tasks/belebele/belebele_cat_Latn.yaml' 2024-01-03T16:18:52,956 adding 'lm_eval/tasks/belebele/belebele_ceb_Latn.yaml' 2024-01-03T16:18:52,957 adding 'lm_eval/tasks/belebele/belebele_ces_Latn.yaml' 2024-01-03T16:18:52,958 adding 'lm_eval/tasks/belebele/belebele_ckb_Arab.yaml' 2024-01-03T16:18:52,959 adding 'lm_eval/tasks/belebele/belebele_dan_Latn.yaml' 2024-01-03T16:18:52,960 adding 'lm_eval/tasks/belebele/belebele_deu_Latn.yaml' 2024-01-03T16:18:52,961 adding 'lm_eval/tasks/belebele/belebele_ell_Grek.yaml' 2024-01-03T16:18:52,962 adding 'lm_eval/tasks/belebele/belebele_eng_Latn.yaml' 2024-01-03T16:18:52,963 adding 'lm_eval/tasks/belebele/belebele_est_Latn.yaml' 2024-01-03T16:18:52,965 adding 'lm_eval/tasks/belebele/belebele_eus_Latn.yaml' 2024-01-03T16:18:52,966 adding 'lm_eval/tasks/belebele/belebele_fin_Latn.yaml' 2024-01-03T16:18:52,967 adding 'lm_eval/tasks/belebele/belebele_fra_Latn.yaml' 2024-01-03T16:18:52,968 adding 'lm_eval/tasks/belebele/belebele_fuv_Latn.yaml' 2024-01-03T16:18:52,969 adding 'lm_eval/tasks/belebele/belebele_gaz_Latn.yaml' 2024-01-03T16:18:52,970 adding 'lm_eval/tasks/belebele/belebele_grn_Latn.yaml' 2024-01-03T16:18:52,971 adding 'lm_eval/tasks/belebele/belebele_guj_Gujr.yaml' 2024-01-03T16:18:52,972 adding 'lm_eval/tasks/belebele/belebele_hat_Latn.yaml' 2024-01-03T16:18:52,973 adding 'lm_eval/tasks/belebele/belebele_hau_Latn.yaml' 2024-01-03T16:18:52,974 adding 'lm_eval/tasks/belebele/belebele_heb_Hebr.yaml' 2024-01-03T16:18:52,978 adding 'lm_eval/tasks/belebele/belebele_hin_Deva.yaml' 2024-01-03T16:18:52,979 adding 'lm_eval/tasks/belebele/belebele_hin_Latn.yaml' 2024-01-03T16:18:52,980 adding 'lm_eval/tasks/belebele/belebele_hrv_Latn.yaml' 2024-01-03T16:18:52,981 adding 'lm_eval/tasks/belebele/belebele_hun_Latn.yaml' 2024-01-03T16:18:52,982 adding 'lm_eval/tasks/belebele/belebele_hye_Armn.yaml' 2024-01-03T16:18:52,983 adding 'lm_eval/tasks/belebele/belebele_ibo_Latn.yaml' 2024-01-03T16:18:52,984 adding 'lm_eval/tasks/belebele/belebele_ilo_Latn.yaml' 2024-01-03T16:18:52,986 adding 'lm_eval/tasks/belebele/belebele_ind_Latn.yaml' 2024-01-03T16:18:52,987 adding 'lm_eval/tasks/belebele/belebele_isl_Latn.yaml' 2024-01-03T16:18:52,988 adding 'lm_eval/tasks/belebele/belebele_ita_Latn.yaml' 2024-01-03T16:18:52,989 adding 'lm_eval/tasks/belebele/belebele_jav_Latn.yaml' 2024-01-03T16:18:52,990 adding 'lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml' 2024-01-03T16:18:52,991 adding 'lm_eval/tasks/belebele/belebele_kac_Latn.yaml' 2024-01-03T16:18:52,992 adding 'lm_eval/tasks/belebele/belebele_kan_Knda.yaml' 2024-01-03T16:18:52,993 adding 'lm_eval/tasks/belebele/belebele_kat_Geor.yaml' 2024-01-03T16:18:52,994 adding 'lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml' 2024-01-03T16:18:52,995 adding 'lm_eval/tasks/belebele/belebele_kea_Latn.yaml' 2024-01-03T16:18:52,996 adding 'lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml' 2024-01-03T16:18:52,997 adding 'lm_eval/tasks/belebele/belebele_khm_Khmr.yaml' 2024-01-03T16:18:52,999 adding 'lm_eval/tasks/belebele/belebele_kin_Latn.yaml' 2024-01-03T16:18:53,000 adding 'lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml' 2024-01-03T16:18:53,001 adding 'lm_eval/tasks/belebele/belebele_kor_Hang.yaml' 2024-01-03T16:18:53,002 adding 'lm_eval/tasks/belebele/belebele_lao_Laoo.yaml' 2024-01-03T16:18:53,003 adding 'lm_eval/tasks/belebele/belebele_lin_Latn.yaml' 2024-01-03T16:18:53,004 adding 'lm_eval/tasks/belebele/belebele_lit_Latn.yaml' 2024-01-03T16:18:53,005 adding 'lm_eval/tasks/belebele/belebele_lug_Latn.yaml' 2024-01-03T16:18:53,006 adding 'lm_eval/tasks/belebele/belebele_luo_Latn.yaml' 2024-01-03T16:18:53,007 adding 'lm_eval/tasks/belebele/belebele_lvs_Latn.yaml' 2024-01-03T16:18:53,008 adding 'lm_eval/tasks/belebele/belebele_mal_Mlym.yaml' 2024-01-03T16:18:53,010 adding 'lm_eval/tasks/belebele/belebele_mar_Deva.yaml' 2024-01-03T16:18:53,011 adding 'lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml' 2024-01-03T16:18:53,012 adding 'lm_eval/tasks/belebele/belebele_mlt_Latn.yaml' 2024-01-03T16:18:53,013 adding 'lm_eval/tasks/belebele/belebele_mri_Latn.yaml' 2024-01-03T16:18:53,014 adding 'lm_eval/tasks/belebele/belebele_mya_Mymr.yaml' 2024-01-03T16:18:53,015 adding 'lm_eval/tasks/belebele/belebele_nld_Latn.yaml' 2024-01-03T16:18:53,016 adding 'lm_eval/tasks/belebele/belebele_nob_Latn.yaml' 2024-01-03T16:18:53,017 adding 'lm_eval/tasks/belebele/belebele_npi_Deva.yaml' 2024-01-03T16:18:53,019 adding 'lm_eval/tasks/belebele/belebele_npi_Latn.yaml' 2024-01-03T16:18:53,020 adding 'lm_eval/tasks/belebele/belebele_nso_Latn.yaml' 2024-01-03T16:18:53,021 adding 'lm_eval/tasks/belebele/belebele_nya_Latn.yaml' 2024-01-03T16:18:53,022 adding 'lm_eval/tasks/belebele/belebele_ory_Orya.yaml' 2024-01-03T16:18:53,023 adding 'lm_eval/tasks/belebele/belebele_pan_Guru.yaml' 2024-01-03T16:18:53,024 adding 'lm_eval/tasks/belebele/belebele_pbt_Arab.yaml' 2024-01-03T16:18:53,025 adding 'lm_eval/tasks/belebele/belebele_pes_Arab.yaml' 2024-01-03T16:18:53,026 adding 'lm_eval/tasks/belebele/belebele_plt_Latn.yaml' 2024-01-03T16:18:53,027 adding 'lm_eval/tasks/belebele/belebele_pol_Latn.yaml' 2024-01-03T16:18:53,028 adding 'lm_eval/tasks/belebele/belebele_por_Latn.yaml' 2024-01-03T16:18:53,029 adding 'lm_eval/tasks/belebele/belebele_ron_Latn.yaml' 2024-01-03T16:18:53,031 adding 'lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml' 2024-01-03T16:18:53,032 adding 'lm_eval/tasks/belebele/belebele_shn_Mymr.yaml' 2024-01-03T16:18:53,033 adding 'lm_eval/tasks/belebele/belebele_sin_Latn.yaml' 2024-01-03T16:18:53,034 adding 'lm_eval/tasks/belebele/belebele_sin_Sinh.yaml' 2024-01-03T16:18:53,035 adding 'lm_eval/tasks/belebele/belebele_slk_Latn.yaml' 2024-01-03T16:18:53,036 adding 'lm_eval/tasks/belebele/belebele_slv_Latn.yaml' 2024-01-03T16:18:53,037 adding 'lm_eval/tasks/belebele/belebele_sna_Latn.yaml' 2024-01-03T16:18:53,038 adding 'lm_eval/tasks/belebele/belebele_snd_Arab.yaml' 2024-01-03T16:18:53,039 adding 'lm_eval/tasks/belebele/belebele_som_Latn.yaml' 2024-01-03T16:18:53,040 adding 'lm_eval/tasks/belebele/belebele_sot_Latn.yaml' 2024-01-03T16:18:53,041 adding 'lm_eval/tasks/belebele/belebele_spa_Latn.yaml' 2024-01-03T16:18:53,042 adding 'lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml' 2024-01-03T16:18:53,043 adding 'lm_eval/tasks/belebele/belebele_ssw_Latn.yaml' 2024-01-03T16:18:53,044 adding 'lm_eval/tasks/belebele/belebele_sun_Latn.yaml' 2024-01-03T16:18:53,045 adding 'lm_eval/tasks/belebele/belebele_swe_Latn.yaml' 2024-01-03T16:18:53,046 adding 'lm_eval/tasks/belebele/belebele_swh_Latn.yaml' 2024-01-03T16:18:53,047 adding 'lm_eval/tasks/belebele/belebele_tam_Taml.yaml' 2024-01-03T16:18:53,048 adding 'lm_eval/tasks/belebele/belebele_tel_Telu.yaml' 2024-01-03T16:18:53,049 adding 'lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml' 2024-01-03T16:18:53,050 adding 'lm_eval/tasks/belebele/belebele_tgl_Latn.yaml' 2024-01-03T16:18:53,052 adding 'lm_eval/tasks/belebele/belebele_tha_Thai.yaml' 2024-01-03T16:18:53,053 adding 'lm_eval/tasks/belebele/belebele_tir_Ethi.yaml' 2024-01-03T16:18:53,054 adding 'lm_eval/tasks/belebele/belebele_tsn_Latn.yaml' 2024-01-03T16:18:53,055 adding 'lm_eval/tasks/belebele/belebele_tso_Latn.yaml' 2024-01-03T16:18:53,056 adding 'lm_eval/tasks/belebele/belebele_tur_Latn.yaml' 2024-01-03T16:18:53,057 adding 'lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml' 2024-01-03T16:18:53,058 adding 'lm_eval/tasks/belebele/belebele_urd_Arab.yaml' 2024-01-03T16:18:53,059 adding 'lm_eval/tasks/belebele/belebele_urd_Latn.yaml' 2024-01-03T16:18:53,060 adding 'lm_eval/tasks/belebele/belebele_uzn_Latn.yaml' 2024-01-03T16:18:53,061 adding 'lm_eval/tasks/belebele/belebele_vie_Latn.yaml' 2024-01-03T16:18:53,063 adding 'lm_eval/tasks/belebele/belebele_war_Latn.yaml' 2024-01-03T16:18:53,064 adding 'lm_eval/tasks/belebele/belebele_wol_Latn.yaml' 2024-01-03T16:18:53,065 adding 'lm_eval/tasks/belebele/belebele_xho_Latn.yaml' 2024-01-03T16:18:53,066 adding 'lm_eval/tasks/belebele/belebele_yor_Latn.yaml' 2024-01-03T16:18:53,067 adding 'lm_eval/tasks/belebele/belebele_zho_Hans.yaml' 2024-01-03T16:18:53,068 adding 'lm_eval/tasks/belebele/belebele_zho_Hant.yaml' 2024-01-03T16:18:53,069 adding 'lm_eval/tasks/belebele/belebele_zsm_Latn.yaml' 2024-01-03T16:18:53,070 adding 'lm_eval/tasks/belebele/belebele_zul_Latn.yaml' 2024-01-03T16:18:53,072 adding 'lm_eval/tasks/benchmarks/minerva_math.yaml' 2024-01-03T16:18:53,073 adding 'lm_eval/tasks/benchmarks/pythia.yaml' 2024-01-03T16:18:53,075 adding 'lm_eval/tasks/benchmarks/t0_eval.yaml' 2024-01-03T16:18:53,076 adding 'lm_eval/tasks/benchmarks/flan/flan_anli.yaml' 2024-01-03T16:18:53,078 adding 'lm_eval/tasks/benchmarks/flan/flan_arc.yaml' 2024-01-03T16:18:53,079 adding 'lm_eval/tasks/benchmarks/flan/flan_boolq.yaml' 2024-01-03T16:18:53,080 adding 'lm_eval/tasks/benchmarks/flan/flan_cot.yaml' 2024-01-03T16:18:53,081 adding 'lm_eval/tasks/benchmarks/flan/flan_held_in.yaml' 2024-01-03T16:18:53,082 adding 'lm_eval/tasks/benchmarks/flan/flan_held_in_yaml' 2024-01-03T16:18:53,084 adding 'lm_eval/tasks/benchmarks/flan/flan_held_out.yaml' 2024-01-03T16:18:53,085 adding 'lm_eval/tasks/benchmarks/flan/flan_rte.yaml' 2024-01-03T16:18:53,086 adding 'lm_eval/tasks/benchmarks/flan/prompt_templates/anli.yaml' 2024-01-03T16:18:53,087 adding 'lm_eval/tasks/benchmarks/flan/prompt_templates/arc.yaml' 2024-01-03T16:18:53,088 adding 'lm_eval/tasks/benchmarks/flan/prompt_templates/boolq.yaml' 2024-01-03T16:18:53,089 adding 'lm_eval/tasks/benchmarks/flan/prompt_templates/rte.yaml' 2024-01-03T16:18:53,091 adding 'lm_eval/tasks/benchmarks/flan/yaml_templates/cot_template_yaml' 2024-01-03T16:18:53,092 adding 'lm_eval/tasks/benchmarks/flan/yaml_templates/held_in_template_yaml' 2024-01-03T16:18:53,095 adding 'lm_eval/tasks/bigbench/README.md' 2024-01-03T16:18:53,096 adding 'lm_eval/tasks/bigbench/generate_tasks.py' 2024-01-03T16:18:53,098 adding 'lm_eval/tasks/bigbench/generate_until_template_yaml' 2024-01-03T16:18:53,099 adding 'lm_eval/tasks/bigbench/multiple_choice_template_yaml' 2024-01-03T16:18:53,100 adding 'lm_eval/tasks/bigbench/push_bigbench_dataset.py' 2024-01-03T16:18:53,105 adding 'lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml' 2024-01-03T16:18:53,106 adding 'lm_eval/tasks/bigbench/generate_until/anachronisms.yaml' 2024-01-03T16:18:53,107 adding 'lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml' 2024-01-03T16:18:53,108 adding 'lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml' 2024-01-03T16:18:53,109 adding 'lm_eval/tasks/bigbench/generate_until/arithmetic.yaml' 2024-01-03T16:18:53,110 adding 'lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml' 2024-01-03T16:18:53,111 adding 'lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml' 2024-01-03T16:18:53,112 adding 'lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml' 2024-01-03T16:18:53,113 adding 'lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml' 2024-01-03T16:18:53,114 adding 'lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml' 2024-01-03T16:18:53,116 adding 'lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml' 2024-01-03T16:18:53,117 adding 'lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml' 2024-01-03T16:18:53,118 adding 'lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml' 2024-01-03T16:18:53,119 adding 'lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml' 2024-01-03T16:18:53,120 adding 'lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml' 2024-01-03T16:18:53,121 adding 'lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml' 2024-01-03T16:18:53,122 adding 'lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml' 2024-01-03T16:18:53,123 adding 'lm_eval/tasks/bigbench/generate_until/code_line_description.yaml' 2024-01-03T16:18:53,125 adding 'lm_eval/tasks/bigbench/generate_until/codenames.yaml' 2024-01-03T16:18:53,126 adding 'lm_eval/tasks/bigbench/generate_until/color.yaml' 2024-01-03T16:18:53,127 adding 'lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml' 2024-01-03T16:18:53,128 adding 'lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml' 2024-01-03T16:18:53,129 adding 'lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml' 2024-01-03T16:18:53,130 adding 'lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml' 2024-01-03T16:18:53,131 adding 'lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml' 2024-01-03T16:18:53,133 adding 'lm_eval/tasks/bigbench/generate_until/crass_ai.yaml' 2024-01-03T16:18:53,134 adding 'lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml' 2024-01-03T16:18:53,135 adding 'lm_eval/tasks/bigbench/generate_until/cryptonite.yaml' 2024-01-03T16:18:53,136 adding 'lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml' 2024-01-03T16:18:53,137 adding 'lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml' 2024-01-03T16:18:53,139 adding 'lm_eval/tasks/bigbench/generate_until/date_understanding.yaml' 2024-01-03T16:18:53,140 adding 'lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml' 2024-01-03T16:18:53,141 adding 'lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml' 2024-01-03T16:18:53,142 adding 'lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml' 2024-01-03T16:18:53,143 adding 'lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml' 2024-01-03T16:18:53,144 adding 'lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml' 2024-01-03T16:18:53,145 adding 'lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml' 2024-01-03T16:18:53,146 adding 'lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml' 2024-01-03T16:18:53,147 adding 'lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml' 2024-01-03T16:18:53,148 adding 'lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml' 2024-01-03T16:18:53,149 adding 'lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml' 2024-01-03T16:18:53,150 adding 'lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml' 2024-01-03T16:18:53,151 adding 'lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml' 2024-01-03T16:18:53,153 adding 'lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml' 2024-01-03T16:18:53,154 adding 'lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml' 2024-01-03T16:18:53,155 adding 'lm_eval/tasks/bigbench/generate_until/fact_checker.yaml' 2024-01-03T16:18:53,156 adding 'lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml' 2024-01-03T16:18:53,157 adding 'lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml' 2024-01-03T16:18:53,158 adding 'lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml' 2024-01-03T16:18:53,159 adding 'lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml' 2024-01-03T16:18:53,161 adding 'lm_eval/tasks/bigbench/generate_until/gem.yaml' 2024-01-03T16:18:53,162 adding 'lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml' 2024-01-03T16:18:53,163 adding 'lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml' 2024-01-03T16:18:53,164 adding 'lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml' 2024-01-03T16:18:53,165 adding 'lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml' 2024-01-03T16:18:53,166 adding 'lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml' 2024-01-03T16:18:53,167 adding 'lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml' 2024-01-03T16:18:53,168 adding 'lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml' 2024-01-03T16:18:53,170 adding 'lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml' 2024-01-03T16:18:53,171 adding 'lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml' 2024-01-03T16:18:53,172 adding 'lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml' 2024-01-03T16:18:53,173 adding 'lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml' 2024-01-03T16:18:53,174 adding 'lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml' 2024-01-03T16:18:53,175 adding 'lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml' 2024-01-03T16:18:53,176 adding 'lm_eval/tasks/bigbench/generate_until/implicatures.yaml' 2024-01-03T16:18:53,178 adding 'lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml' 2024-01-03T16:18:53,179 adding 'lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml' 2024-01-03T16:18:53,180 adding 'lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml' 2024-01-03T16:18:53,181 adding 'lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml' 2024-01-03T16:18:53,182 adding 'lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml' 2024-01-03T16:18:53,183 adding 'lm_eval/tasks/bigbench/generate_until/irony_identification.yaml' 2024-01-03T16:18:53,185 adding 'lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml' 2024-01-03T16:18:53,186 adding 'lm_eval/tasks/bigbench/generate_until/kannada.yaml' 2024-01-03T16:18:53,187 adding 'lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml' 2024-01-03T16:18:53,188 adding 'lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml' 2024-01-03T16:18:53,189 adding 'lm_eval/tasks/bigbench/generate_until/language_games.yaml' 2024-01-03T16:18:53,190 adding 'lm_eval/tasks/bigbench/generate_until/language_identification.yaml' 2024-01-03T16:18:53,191 adding 'lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml' 2024-01-03T16:18:53,192 adding 'lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml' 2024-01-03T16:18:53,193 adding 'lm_eval/tasks/bigbench/generate_until/list_functions.yaml' 2024-01-03T16:18:53,194 adding 'lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml' 2024-01-03T16:18:53,195 adding 'lm_eval/tasks/bigbench/generate_until/logical_args.yaml' 2024-01-03T16:18:53,196 adding 'lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml' 2024-01-03T16:18:53,198 adding 'lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml' 2024-01-03T16:18:53,199 adding 'lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml' 2024-01-03T16:18:53,200 adding 'lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml' 2024-01-03T16:18:53,201 adding 'lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml' 2024-01-03T16:18:53,202 adding 'lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml' 2024-01-03T16:18:53,203 adding 'lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml' 2024-01-03T16:18:53,204 adding 'lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml' 2024-01-03T16:18:53,205 adding 'lm_eval/tasks/bigbench/generate_until/misconceptions.yaml' 2024-01-03T16:18:53,206 adding 'lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml' 2024-01-03T16:18:53,207 adding 'lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml' 2024-01-03T16:18:53,209 adding 'lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml' 2024-01-03T16:18:53,210 adding 'lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml' 2024-01-03T16:18:53,211 adding 'lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml' 2024-01-03T16:18:53,212 adding 'lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml' 2024-01-03T16:18:53,213 adding 'lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml' 2024-01-03T16:18:53,214 adding 'lm_eval/tasks/bigbench/generate_until/multiemo.yaml' 2024-01-03T16:18:53,215 adding 'lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml' 2024-01-03T16:18:53,216 adding 'lm_eval/tasks/bigbench/generate_until/navigate.yaml' 2024-01-03T16:18:53,218 adding 'lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml' 2024-01-03T16:18:53,219 adding 'lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml' 2024-01-03T16:18:53,220 adding 'lm_eval/tasks/bigbench/generate_until/object_counting.yaml' 2024-01-03T16:18:53,221 adding 'lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml' 2024-01-03T16:18:53,222 adding 'lm_eval/tasks/bigbench/generate_until/operators.yaml' 2024-01-03T16:18:53,223 adding 'lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml' 2024-01-03T16:18:53,224 adding 'lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml' 2024-01-03T16:18:53,226 adding 'lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml' 2024-01-03T16:18:53,227 adding 'lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml' 2024-01-03T16:18:53,228 adding 'lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml' 2024-01-03T16:18:53,229 adding 'lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml' 2024-01-03T16:18:53,230 adding 'lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml' 2024-01-03T16:18:53,231 adding 'lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml' 2024-01-03T16:18:53,232 adding 'lm_eval/tasks/bigbench/generate_until/physics.yaml' 2024-01-03T16:18:53,233 adding 'lm_eval/tasks/bigbench/generate_until/physics_questions.yaml' 2024-01-03T16:18:53,234 adding 'lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml' 2024-01-03T16:18:53,235 adding 'lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml' 2024-01-03T16:18:53,236 adding 'lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml' 2024-01-03T16:18:53,237 adding 'lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml' 2024-01-03T16:18:53,238 adding 'lm_eval/tasks/bigbench/generate_until/question_selection.yaml' 2024-01-03T16:18:53,239 adding 'lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml' 2024-01-03T16:18:53,240 adding 'lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:53,241 adding 'lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml' 2024-01-03T16:18:53,242 adding 'lm_eval/tasks/bigbench/generate_until/rephrase.yaml' 2024-01-03T16:18:53,244 adding 'lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml' 2024-01-03T16:18:53,245 adding 'lm_eval/tasks/bigbench/generate_until/ruin_names.yaml' 2024-01-03T16:18:53,246 adding 'lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml' 2024-01-03T16:18:53,247 adding 'lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml' 2024-01-03T16:18:53,248 adding 'lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml' 2024-01-03T16:18:53,249 adding 'lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml' 2024-01-03T16:18:53,250 adding 'lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml' 2024-01-03T16:18:53,251 adding 'lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml' 2024-01-03T16:18:53,252 adding 'lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml' 2024-01-03T16:18:53,253 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml' 2024-01-03T16:18:53,254 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml' 2024-01-03T16:18:53,256 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml' 2024-01-03T16:18:53,257 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml' 2024-01-03T16:18:53,258 adding 'lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml' 2024-01-03T16:18:53,259 adding 'lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml' 2024-01-03T16:18:53,260 adding 'lm_eval/tasks/bigbench/generate_until/snarks.yaml' 2024-01-03T16:18:53,261 adding 'lm_eval/tasks/bigbench/generate_until/social_iqa.yaml' 2024-01-03T16:18:53,262 adding 'lm_eval/tasks/bigbench/generate_until/social_support.yaml' 2024-01-03T16:18:53,263 adding 'lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml' 2024-01-03T16:18:53,265 adding 'lm_eval/tasks/bigbench/generate_until/strange_stories.yaml' 2024-01-03T16:18:53,266 adding 'lm_eval/tasks/bigbench/generate_until/strategyqa.yaml' 2024-01-03T16:18:53,267 adding 'lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml' 2024-01-03T16:18:53,268 adding 'lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml' 2024-01-03T16:18:53,269 adding 'lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml' 2024-01-03T16:18:53,270 adding 'lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml' 2024-01-03T16:18:53,271 adding 'lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml' 2024-01-03T16:18:53,273 adding 'lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml' 2024-01-03T16:18:53,274 adding 'lm_eval/tasks/bigbench/generate_until/tense.yaml' 2024-01-03T16:18:53,275 adding 'lm_eval/tasks/bigbench/generate_until/timedial.yaml' 2024-01-03T16:18:53,276 adding 'lm_eval/tasks/bigbench/generate_until/topical_chat.yaml' 2024-01-03T16:18:53,277 adding 'lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml' 2024-01-03T16:18:53,278 adding 'lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml' 2024-01-03T16:18:53,280 adding 'lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml' 2024-01-03T16:18:53,281 adding 'lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml' 2024-01-03T16:18:53,282 adding 'lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml' 2024-01-03T16:18:53,283 adding 'lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml' 2024-01-03T16:18:53,284 adding 'lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml' 2024-01-03T16:18:53,285 adding 'lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml' 2024-01-03T16:18:53,286 adding 'lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml' 2024-01-03T16:18:53,287 adding 'lm_eval/tasks/bigbench/generate_until/winowhy.yaml' 2024-01-03T16:18:53,288 adding 'lm_eval/tasks/bigbench/generate_until/word_sorting.yaml' 2024-01-03T16:18:53,289 adding 'lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml' 2024-01-03T16:18:53,294 adding 'lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml' 2024-01-03T16:18:53,295 adding 'lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml' 2024-01-03T16:18:53,296 adding 'lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml' 2024-01-03T16:18:53,297 adding 'lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml' 2024-01-03T16:18:53,298 adding 'lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml' 2024-01-03T16:18:53,299 adding 'lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml' 2024-01-03T16:18:53,300 adding 'lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml' 2024-01-03T16:18:53,302 adding 'lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml' 2024-01-03T16:18:53,303 adding 'lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml' 2024-01-03T16:18:53,304 adding 'lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml' 2024-01-03T16:18:53,305 adding 'lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml' 2024-01-03T16:18:53,306 adding 'lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml' 2024-01-03T16:18:53,307 adding 'lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml' 2024-01-03T16:18:53,308 adding 'lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml' 2024-01-03T16:18:53,309 adding 'lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml' 2024-01-03T16:18:53,311 adding 'lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml' 2024-01-03T16:18:53,312 adding 'lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml' 2024-01-03T16:18:53,313 adding 'lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml' 2024-01-03T16:18:53,314 adding 'lm_eval/tasks/bigbench/multiple_choice/codenames.yaml' 2024-01-03T16:18:53,315 adding 'lm_eval/tasks/bigbench/multiple_choice/color.yaml' 2024-01-03T16:18:53,316 adding 'lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml' 2024-01-03T16:18:53,317 adding 'lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml' 2024-01-03T16:18:53,318 adding 'lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml' 2024-01-03T16:18:53,320 adding 'lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml' 2024-01-03T16:18:53,321 adding 'lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml' 2024-01-03T16:18:53,322 adding 'lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml' 2024-01-03T16:18:53,323 adding 'lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml' 2024-01-03T16:18:53,324 adding 'lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml' 2024-01-03T16:18:53,325 adding 'lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml' 2024-01-03T16:18:53,326 adding 'lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml' 2024-01-03T16:18:53,328 adding 'lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml' 2024-01-03T16:18:53,329 adding 'lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml' 2024-01-03T16:18:53,330 adding 'lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml' 2024-01-03T16:18:53,331 adding 'lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml' 2024-01-03T16:18:53,332 adding 'lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml' 2024-01-03T16:18:53,333 adding 'lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml' 2024-01-03T16:18:53,334 adding 'lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml' 2024-01-03T16:18:53,335 adding 'lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml' 2024-01-03T16:18:53,336 adding 'lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml' 2024-01-03T16:18:53,338 adding 'lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml' 2024-01-03T16:18:53,339 adding 'lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml' 2024-01-03T16:18:53,340 adding 'lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml' 2024-01-03T16:18:53,341 adding 'lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml' 2024-01-03T16:18:53,342 adding 'lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml' 2024-01-03T16:18:53,344 adding 'lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml' 2024-01-03T16:18:53,345 adding 'lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml' 2024-01-03T16:18:53,346 adding 'lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml' 2024-01-03T16:18:53,347 adding 'lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml' 2024-01-03T16:18:53,348 adding 'lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml' 2024-01-03T16:18:53,349 adding 'lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml' 2024-01-03T16:18:53,350 adding 'lm_eval/tasks/bigbench/multiple_choice/gem.yaml' 2024-01-03T16:18:53,351 adding 'lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml' 2024-01-03T16:18:53,352 adding 'lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml' 2024-01-03T16:18:53,353 adding 'lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml' 2024-01-03T16:18:53,354 adding 'lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml' 2024-01-03T16:18:53,356 adding 'lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml' 2024-01-03T16:18:53,357 adding 'lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml' 2024-01-03T16:18:53,358 adding 'lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml' 2024-01-03T16:18:53,359 adding 'lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml' 2024-01-03T16:18:53,360 adding 'lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml' 2024-01-03T16:18:53,361 adding 'lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml' 2024-01-03T16:18:53,362 adding 'lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml' 2024-01-03T16:18:53,363 adding 'lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml' 2024-01-03T16:18:53,365 adding 'lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml' 2024-01-03T16:18:53,366 adding 'lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml' 2024-01-03T16:18:53,367 adding 'lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml' 2024-01-03T16:18:53,368 adding 'lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml' 2024-01-03T16:18:53,369 adding 'lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml' 2024-01-03T16:18:53,370 adding 'lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml' 2024-01-03T16:18:53,371 adding 'lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml' 2024-01-03T16:18:53,373 adding 'lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml' 2024-01-03T16:18:53,374 adding 'lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml' 2024-01-03T16:18:53,375 adding 'lm_eval/tasks/bigbench/multiple_choice/kannada.yaml' 2024-01-03T16:18:53,376 adding 'lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml' 2024-01-03T16:18:53,377 adding 'lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml' 2024-01-03T16:18:53,378 adding 'lm_eval/tasks/bigbench/multiple_choice/language_games.yaml' 2024-01-03T16:18:53,379 adding 'lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml' 2024-01-03T16:18:53,380 adding 'lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml' 2024-01-03T16:18:53,381 adding 'lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml' 2024-01-03T16:18:53,382 adding 'lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml' 2024-01-03T16:18:53,383 adding 'lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml' 2024-01-03T16:18:53,385 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml' 2024-01-03T16:18:53,386 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml' 2024-01-03T16:18:53,387 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml' 2024-01-03T16:18:53,388 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml' 2024-01-03T16:18:53,389 adding 'lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml' 2024-01-03T16:18:53,390 adding 'lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml' 2024-01-03T16:18:53,391 adding 'lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml' 2024-01-03T16:18:53,393 adding 'lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml' 2024-01-03T16:18:53,394 adding 'lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml' 2024-01-03T16:18:53,395 adding 'lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml' 2024-01-03T16:18:53,396 adding 'lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml' 2024-01-03T16:18:53,397 adding 'lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml' 2024-01-03T16:18:53,398 adding 'lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml' 2024-01-03T16:18:53,399 adding 'lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml' 2024-01-03T16:18:53,400 adding 'lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml' 2024-01-03T16:18:53,402 adding 'lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml' 2024-01-03T16:18:53,403 adding 'lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml' 2024-01-03T16:18:53,404 adding 'lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml' 2024-01-03T16:18:53,405 adding 'lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml' 2024-01-03T16:18:53,406 adding 'lm_eval/tasks/bigbench/multiple_choice/navigate.yaml' 2024-01-03T16:18:53,407 adding 'lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml' 2024-01-03T16:18:53,408 adding 'lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml' 2024-01-03T16:18:53,410 adding 'lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml' 2024-01-03T16:18:53,411 adding 'lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml' 2024-01-03T16:18:53,412 adding 'lm_eval/tasks/bigbench/multiple_choice/operators.yaml' 2024-01-03T16:18:53,413 adding 'lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml' 2024-01-03T16:18:53,414 adding 'lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml' 2024-01-03T16:18:53,415 adding 'lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml' 2024-01-03T16:18:53,417 adding 'lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml' 2024-01-03T16:18:53,418 adding 'lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml' 2024-01-03T16:18:53,419 adding 'lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml' 2024-01-03T16:18:53,420 adding 'lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml' 2024-01-03T16:18:53,421 adding 'lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml' 2024-01-03T16:18:53,422 adding 'lm_eval/tasks/bigbench/multiple_choice/physics.yaml' 2024-01-03T16:18:53,424 adding 'lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml' 2024-01-03T16:18:53,425 adding 'lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml' 2024-01-03T16:18:53,426 adding 'lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml' 2024-01-03T16:18:53,427 adding 'lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml' 2024-01-03T16:18:53,428 adding 'lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml' 2024-01-03T16:18:53,429 adding 'lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml' 2024-01-03T16:18:53,430 adding 'lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml' 2024-01-03T16:18:53,431 adding 'lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml' 2024-01-03T16:18:53,432 adding 'lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml' 2024-01-03T16:18:53,433 adding 'lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml' 2024-01-03T16:18:53,435 adding 'lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml' 2024-01-03T16:18:53,436 adding 'lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml' 2024-01-03T16:18:53,437 adding 'lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml' 2024-01-03T16:18:53,438 adding 'lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml' 2024-01-03T16:18:53,439 adding 'lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml' 2024-01-03T16:18:53,440 adding 'lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml' 2024-01-03T16:18:53,441 adding 'lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml' 2024-01-03T16:18:53,442 adding 'lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml' 2024-01-03T16:18:53,443 adding 'lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml' 2024-01-03T16:18:53,445 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml' 2024-01-03T16:18:53,446 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml' 2024-01-03T16:18:53,447 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml' 2024-01-03T16:18:53,448 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml' 2024-01-03T16:18:53,449 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml' 2024-01-03T16:18:53,450 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml' 2024-01-03T16:18:53,451 adding 'lm_eval/tasks/bigbench/multiple_choice/snarks.yaml' 2024-01-03T16:18:53,452 adding 'lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml' 2024-01-03T16:18:53,453 adding 'lm_eval/tasks/bigbench/multiple_choice/social_support.yaml' 2024-01-03T16:18:53,455 adding 'lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml' 2024-01-03T16:18:53,456 adding 'lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml' 2024-01-03T16:18:53,457 adding 'lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml' 2024-01-03T16:18:53,458 adding 'lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml' 2024-01-03T16:18:53,459 adding 'lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml' 2024-01-03T16:18:53,460 adding 'lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml' 2024-01-03T16:18:53,461 adding 'lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml' 2024-01-03T16:18:53,463 adding 'lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml' 2024-01-03T16:18:53,464 adding 'lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml' 2024-01-03T16:18:53,465 adding 'lm_eval/tasks/bigbench/multiple_choice/tense.yaml' 2024-01-03T16:18:53,466 adding 'lm_eval/tasks/bigbench/multiple_choice/timedial.yaml' 2024-01-03T16:18:53,467 adding 'lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml' 2024-01-03T16:18:53,468 adding 'lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml' 2024-01-03T16:18:53,470 adding 'lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml' 2024-01-03T16:18:53,471 adding 'lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml' 2024-01-03T16:18:53,472 adding 'lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml' 2024-01-03T16:18:53,473 adding 'lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml' 2024-01-03T16:18:53,474 adding 'lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml' 2024-01-03T16:18:53,475 adding 'lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml' 2024-01-03T16:18:53,476 adding 'lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml' 2024-01-03T16:18:53,477 adding 'lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml' 2024-01-03T16:18:53,478 adding 'lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml' 2024-01-03T16:18:53,479 adding 'lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml' 2024-01-03T16:18:53,480 adding 'lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml' 2024-01-03T16:18:53,483 adding 'lm_eval/tasks/blimp/README.md' 2024-01-03T16:18:53,484 adding 'lm_eval/tasks/blimp/_template_yaml' 2024-01-03T16:18:53,486 adding 'lm_eval/tasks/blimp/adjunct_island.yaml' 2024-01-03T16:18:53,487 adding 'lm_eval/tasks/blimp/anaphor_gender_agreement.yaml' 2024-01-03T16:18:53,488 adding 'lm_eval/tasks/blimp/anaphor_number_agreement.yaml' 2024-01-03T16:18:53,489 adding 'lm_eval/tasks/blimp/animate_subject_passive.yaml' 2024-01-03T16:18:53,490 adding 'lm_eval/tasks/blimp/animate_subject_trans.yaml' 2024-01-03T16:18:53,491 adding 'lm_eval/tasks/blimp/causative.yaml' 2024-01-03T16:18:53,492 adding 'lm_eval/tasks/blimp/complex_NP_island.yaml' 2024-01-03T16:18:53,493 adding 'lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml' 2024-01-03T16:18:53,494 adding 'lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml' 2024-01-03T16:18:53,495 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml' 2024-01-03T16:18:53,496 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml' 2024-01-03T16:18:53,497 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml' 2024-01-03T16:18:53,499 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml' 2024-01-03T16:18:53,500 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml' 2024-01-03T16:18:53,501 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml' 2024-01-03T16:18:53,502 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml' 2024-01-03T16:18:53,503 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml' 2024-01-03T16:18:53,504 adding 'lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml' 2024-01-03T16:18:53,505 adding 'lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml' 2024-01-03T16:18:53,506 adding 'lm_eval/tasks/blimp/drop_argument.yaml' 2024-01-03T16:18:53,508 adding 'lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml' 2024-01-03T16:18:53,509 adding 'lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml' 2024-01-03T16:18:53,510 adding 'lm_eval/tasks/blimp/existential_there_object_raising.yaml' 2024-01-03T16:18:53,511 adding 'lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml' 2024-01-03T16:18:53,512 adding 'lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml' 2024-01-03T16:18:53,513 adding 'lm_eval/tasks/blimp/existential_there_subject_raising.yaml' 2024-01-03T16:18:53,514 adding 'lm_eval/tasks/blimp/expletive_it_object_raising.yaml' 2024-01-03T16:18:53,516 adding 'lm_eval/tasks/blimp/generate_configs.py' 2024-01-03T16:18:53,517 adding 'lm_eval/tasks/blimp/inchoative.yaml' 2024-01-03T16:18:53,518 adding 'lm_eval/tasks/blimp/intransitive.yaml' 2024-01-03T16:18:53,519 adding 'lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml' 2024-01-03T16:18:53,520 adding 'lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml' 2024-01-03T16:18:53,521 adding 'lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml' 2024-01-03T16:18:53,523 adding 'lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml' 2024-01-03T16:18:53,524 adding 'lm_eval/tasks/blimp/left_branch_island_echo_question.yaml' 2024-01-03T16:18:53,525 adding 'lm_eval/tasks/blimp/left_branch_island_simple_question.yaml' 2024-01-03T16:18:53,526 adding 'lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml' 2024-01-03T16:18:53,527 adding 'lm_eval/tasks/blimp/npi_present_1.yaml' 2024-01-03T16:18:53,528 adding 'lm_eval/tasks/blimp/npi_present_2.yaml' 2024-01-03T16:18:53,529 adding 'lm_eval/tasks/blimp/only_npi_licensor_present.yaml' 2024-01-03T16:18:53,530 adding 'lm_eval/tasks/blimp/only_npi_scope.yaml' 2024-01-03T16:18:53,531 adding 'lm_eval/tasks/blimp/passive_1.yaml' 2024-01-03T16:18:53,532 adding 'lm_eval/tasks/blimp/passive_2.yaml' 2024-01-03T16:18:53,533 adding 'lm_eval/tasks/blimp/principle_A_c_command.yaml' 2024-01-03T16:18:53,534 adding 'lm_eval/tasks/blimp/principle_A_case_1.yaml' 2024-01-03T16:18:53,535 adding 'lm_eval/tasks/blimp/principle_A_case_2.yaml' 2024-01-03T16:18:53,536 adding 'lm_eval/tasks/blimp/principle_A_domain_1.yaml' 2024-01-03T16:18:53,537 adding 'lm_eval/tasks/blimp/principle_A_domain_2.yaml' 2024-01-03T16:18:53,538 adding 'lm_eval/tasks/blimp/principle_A_domain_3.yaml' 2024-01-03T16:18:53,539 adding 'lm_eval/tasks/blimp/principle_A_reconstruction.yaml' 2024-01-03T16:18:53,540 adding 'lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml' 2024-01-03T16:18:53,542 adding 'lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml' 2024-01-03T16:18:53,543 adding 'lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml' 2024-01-03T16:18:53,544 adding 'lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml' 2024-01-03T16:18:53,545 adding 'lm_eval/tasks/blimp/sentential_subject_island.yaml' 2024-01-03T16:18:53,546 adding 'lm_eval/tasks/blimp/superlative_quantifiers_1.yaml' 2024-01-03T16:18:53,547 adding 'lm_eval/tasks/blimp/superlative_quantifiers_2.yaml' 2024-01-03T16:18:53,548 adding 'lm_eval/tasks/blimp/tough_vs_raising_1.yaml' 2024-01-03T16:18:53,549 adding 'lm_eval/tasks/blimp/tough_vs_raising_2.yaml' 2024-01-03T16:18:53,550 adding 'lm_eval/tasks/blimp/transitive.yaml' 2024-01-03T16:18:53,552 adding 'lm_eval/tasks/blimp/wh_island.yaml' 2024-01-03T16:18:53,553 adding 'lm_eval/tasks/blimp/wh_questions_object_gap.yaml' 2024-01-03T16:18:53,554 adding 'lm_eval/tasks/blimp/wh_questions_subject_gap.yaml' 2024-01-03T16:18:53,555 adding 'lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml' 2024-01-03T16:18:53,556 adding 'lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml' 2024-01-03T16:18:53,557 adding 'lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml' 2024-01-03T16:18:53,558 adding 'lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml' 2024-01-03T16:18:53,559 adding 'lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml' 2024-01-03T16:18:53,562 adding 'lm_eval/tasks/ceval/README.md' 2024-01-03T16:18:53,563 adding 'lm_eval/tasks/ceval/_default_ceval_yaml' 2024-01-03T16:18:53,565 adding 'lm_eval/tasks/ceval/_generate_configs.py' 2024-01-03T16:18:53,566 adding 'lm_eval/tasks/ceval/ceval-valid_accountant.yaml' 2024-01-03T16:18:53,567 adding 'lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml' 2024-01-03T16:18:53,568 adding 'lm_eval/tasks/ceval/ceval-valid_art_studies.yaml' 2024-01-03T16:18:53,570 adding 'lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml' 2024-01-03T16:18:53,571 adding 'lm_eval/tasks/ceval/ceval-valid_business_administration.yaml' 2024-01-03T16:18:53,572 adding 'lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml' 2024-01-03T16:18:53,573 adding 'lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml' 2024-01-03T16:18:53,574 adding 'lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml' 2024-01-03T16:18:53,575 adding 'lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml' 2024-01-03T16:18:53,577 adding 'lm_eval/tasks/ceval/ceval-valid_college_economics.yaml' 2024-01-03T16:18:53,578 adding 'lm_eval/tasks/ceval/ceval-valid_college_physics.yaml' 2024-01-03T16:18:53,579 adding 'lm_eval/tasks/ceval/ceval-valid_college_programming.yaml' 2024-01-03T16:18:53,580 adding 'lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml' 2024-01-03T16:18:53,581 adding 'lm_eval/tasks/ceval/ceval-valid_computer_network.yaml' 2024-01-03T16:18:53,582 adding 'lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml' 2024-01-03T16:18:53,583 adding 'lm_eval/tasks/ceval/ceval-valid_education_science.yaml' 2024-01-03T16:18:53,584 adding 'lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml' 2024-01-03T16:18:53,585 adding 'lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml' 2024-01-03T16:18:53,586 adding 'lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml' 2024-01-03T16:18:53,588 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml' 2024-01-03T16:18:53,589 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml' 2024-01-03T16:18:53,590 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml' 2024-01-03T16:18:53,591 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml' 2024-01-03T16:18:53,592 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml' 2024-01-03T16:18:53,593 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml' 2024-01-03T16:18:53,594 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml' 2024-01-03T16:18:53,596 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml' 2024-01-03T16:18:53,597 adding 'lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml' 2024-01-03T16:18:53,598 adding 'lm_eval/tasks/ceval/ceval-valid_law.yaml' 2024-01-03T16:18:53,599 adding 'lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml' 2024-01-03T16:18:53,600 adding 'lm_eval/tasks/ceval/ceval-valid_logic.yaml' 2024-01-03T16:18:53,601 adding 'lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml' 2024-01-03T16:18:53,603 adding 'lm_eval/tasks/ceval/ceval-valid_marxism.yaml' 2024-01-03T16:18:53,604 adding 'lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml' 2024-01-03T16:18:53,605 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml' 2024-01-03T16:18:53,606 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml' 2024-01-03T16:18:53,607 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml' 2024-01-03T16:18:53,608 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml' 2024-01-03T16:18:53,610 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml' 2024-01-03T16:18:53,611 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml' 2024-01-03T16:18:53,612 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml' 2024-01-03T16:18:53,613 adding 'lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml' 2024-01-03T16:18:53,614 adding 'lm_eval/tasks/ceval/ceval-valid_operating_system.yaml' 2024-01-03T16:18:53,615 adding 'lm_eval/tasks/ceval/ceval-valid_physician.yaml' 2024-01-03T16:18:53,616 adding 'lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml' 2024-01-03T16:18:53,618 adding 'lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml' 2024-01-03T16:18:53,619 adding 'lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml' 2024-01-03T16:18:53,620 adding 'lm_eval/tasks/ceval/ceval-valid_sports_science.yaml' 2024-01-03T16:18:53,621 adding 'lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml' 2024-01-03T16:18:53,622 adding 'lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml' 2024-01-03T16:18:53,623 adding 'lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml' 2024-01-03T16:18:53,625 adding 'lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml' 2024-01-03T16:18:53,628 adding 'lm_eval/tasks/cmmlu/README.md' 2024-01-03T16:18:53,629 adding 'lm_eval/tasks/cmmlu/_default_template_yaml' 2024-01-03T16:18:53,630 adding 'lm_eval/tasks/cmmlu/_generate_configs.py' 2024-01-03T16:18:53,631 adding 'lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml' 2024-01-03T16:18:53,632 adding 'lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml' 2024-01-03T16:18:53,634 adding 'lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml' 2024-01-03T16:18:53,635 adding 'lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml' 2024-01-03T16:18:53,636 adding 'lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml' 2024-01-03T16:18:53,637 adding 'lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml' 2024-01-03T16:18:53,638 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml' 2024-01-03T16:18:53,639 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml' 2024-01-03T16:18:53,640 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml' 2024-01-03T16:18:53,642 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml' 2024-01-03T16:18:53,643 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml' 2024-01-03T16:18:53,644 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml' 2024-01-03T16:18:53,645 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml' 2024-01-03T16:18:53,646 adding 'lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml' 2024-01-03T16:18:53,647 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml' 2024-01-03T16:18:53,648 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml' 2024-01-03T16:18:53,649 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml' 2024-01-03T16:18:53,651 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml' 2024-01-03T16:18:53,652 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml' 2024-01-03T16:18:53,653 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml' 2024-01-03T16:18:53,654 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml' 2024-01-03T16:18:53,655 adding 'lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml' 2024-01-03T16:18:53,656 adding 'lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml' 2024-01-03T16:18:53,658 adding 'lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml' 2024-01-03T16:18:53,659 adding 'lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml' 2024-01-03T16:18:53,660 adding 'lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml' 2024-01-03T16:18:53,661 adding 'lm_eval/tasks/cmmlu/cmmlu_default_education.yaml' 2024-01-03T16:18:53,662 adding 'lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml' 2024-01-03T16:18:53,663 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml' 2024-01-03T16:18:53,664 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml' 2024-01-03T16:18:53,666 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml' 2024-01-03T16:18:53,667 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml' 2024-01-03T16:18:53,668 adding 'lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml' 2024-01-03T16:18:53,669 adding 'lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml' 2024-01-03T16:18:53,670 adding 'lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml' 2024-01-03T16:18:53,672 adding 'lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml' 2024-01-03T16:18:53,673 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml' 2024-01-03T16:18:53,674 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml' 2024-01-03T16:18:53,675 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml' 2024-01-03T16:18:53,676 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml' 2024-01-03T16:18:53,677 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml' 2024-01-03T16:18:53,679 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml' 2024-01-03T16:18:53,680 adding 'lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml' 2024-01-03T16:18:53,681 adding 'lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml' 2024-01-03T16:18:53,682 adding 'lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml' 2024-01-03T16:18:53,683 adding 'lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml' 2024-01-03T16:18:53,684 adding 'lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml' 2024-01-03T16:18:53,685 adding 'lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml' 2024-01-03T16:18:53,687 adding 'lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml' 2024-01-03T16:18:53,688 adding 'lm_eval/tasks/cmmlu/cmmlu_default_management.yaml' 2024-01-03T16:18:53,689 adding 'lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml' 2024-01-03T16:18:53,690 adding 'lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml' 2024-01-03T16:18:53,691 adding 'lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml' 2024-01-03T16:18:53,692 adding 'lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml' 2024-01-03T16:18:53,693 adding 'lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml' 2024-01-03T16:18:53,694 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml' 2024-01-03T16:18:53,696 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml' 2024-01-03T16:18:53,697 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml' 2024-01-03T16:18:53,698 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml' 2024-01-03T16:18:53,699 adding 'lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml' 2024-01-03T16:18:53,700 adding 'lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml' 2024-01-03T16:18:53,701 adding 'lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml' 2024-01-03T16:18:53,702 adding 'lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml' 2024-01-03T16:18:53,703 adding 'lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml' 2024-01-03T16:18:53,704 adding 'lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml' 2024-01-03T16:18:53,705 adding 'lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml' 2024-01-03T16:18:53,707 adding 'lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml' 2024-01-03T16:18:53,709 adding 'lm_eval/tasks/code_x_glue/code-text/bleu.py' 2024-01-03T16:18:53,710 adding 'lm_eval/tasks/code_x_glue/code-text/go.yaml' 2024-01-03T16:18:53,711 adding 'lm_eval/tasks/code_x_glue/code-text/java.yaml' 2024-01-03T16:18:53,713 adding 'lm_eval/tasks/code_x_glue/code-text/javascript.yaml' 2024-01-03T16:18:53,714 adding 'lm_eval/tasks/code_x_glue/code-text/php.yaml' 2024-01-03T16:18:53,715 adding 'lm_eval/tasks/code_x_glue/code-text/python.yaml' 2024-01-03T16:18:53,716 adding 'lm_eval/tasks/code_x_glue/code-text/ruby.yaml' 2024-01-03T16:18:53,717 adding 'lm_eval/tasks/code_x_glue/code-text/utils.py' 2024-01-03T16:18:53,719 adding 'lm_eval/tasks/coqa/README.md' 2024-01-03T16:18:53,720 adding 'lm_eval/tasks/coqa/default.yaml' 2024-01-03T16:18:53,721 adding 'lm_eval/tasks/coqa/utils.py' 2024-01-03T16:18:53,724 adding 'lm_eval/tasks/crows_pairs/README.md' 2024-01-03T16:18:53,725 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english.yaml' 2024-01-03T16:18:53,726 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml' 2024-01-03T16:18:53,727 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml' 2024-01-03T16:18:53,729 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml' 2024-01-03T16:18:53,730 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml' 2024-01-03T16:18:53,731 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml' 2024-01-03T16:18:53,732 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml' 2024-01-03T16:18:53,733 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml' 2024-01-03T16:18:53,734 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml' 2024-01-03T16:18:53,735 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml' 2024-01-03T16:18:53,737 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml' 2024-01-03T16:18:53,738 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french.yaml' 2024-01-03T16:18:53,739 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml' 2024-01-03T16:18:53,740 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml' 2024-01-03T16:18:53,741 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml' 2024-01-03T16:18:53,742 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml' 2024-01-03T16:18:53,744 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml' 2024-01-03T16:18:53,745 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml' 2024-01-03T16:18:53,746 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml' 2024-01-03T16:18:53,747 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml' 2024-01-03T16:18:53,748 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml' 2024-01-03T16:18:53,749 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml' 2024-01-03T16:18:53,750 adding 'lm_eval/tasks/crows_pairs/utils.py' 2024-01-03T16:18:53,752 adding 'lm_eval/tasks/csatqa/_default_csatqa_yaml' 2024-01-03T16:18:53,753 adding 'lm_eval/tasks/csatqa/_generate_configs.py' 2024-01-03T16:18:53,754 adding 'lm_eval/tasks/csatqa/csatqa_gr.yaml' 2024-01-03T16:18:53,756 adding 'lm_eval/tasks/csatqa/csatqa_li.yaml' 2024-01-03T16:18:53,757 adding 'lm_eval/tasks/csatqa/csatqa_rch.yaml' 2024-01-03T16:18:53,758 adding 'lm_eval/tasks/csatqa/csatqa_rcs.yaml' 2024-01-03T16:18:53,759 adding 'lm_eval/tasks/csatqa/csatqa_rcss.yaml' 2024-01-03T16:18:53,760 adding 'lm_eval/tasks/csatqa/csatqa_wr.yaml' 2024-01-03T16:18:53,761 adding 'lm_eval/tasks/csatqa/utils.py' 2024-01-03T16:18:53,763 adding 'lm_eval/tasks/drop/README.md' 2024-01-03T16:18:53,764 adding 'lm_eval/tasks/drop/default.yaml' 2024-01-03T16:18:53,766 adding 'lm_eval/tasks/drop/utils.py' 2024-01-03T16:18:53,768 adding 'lm_eval/tasks/glue/README.md' 2024-01-03T16:18:53,770 adding 'lm_eval/tasks/glue/cola/default.yaml' 2024-01-03T16:18:53,771 adding 'lm_eval/tasks/glue/mnli/default.yaml' 2024-01-03T16:18:53,772 adding 'lm_eval/tasks/glue/mnli/mismatch.yaml' 2024-01-03T16:18:53,773 adding 'lm_eval/tasks/glue/mnli/utils.py' 2024-01-03T16:18:53,775 adding 'lm_eval/tasks/glue/mrpc/default.yaml' 2024-01-03T16:18:53,777 adding 'lm_eval/tasks/glue/qnli/default.yaml' 2024-01-03T16:18:53,778 adding 'lm_eval/tasks/glue/qqp/default.yaml' 2024-01-03T16:18:53,780 adding 'lm_eval/tasks/glue/rte/default.yaml' 2024-01-03T16:18:53,781 adding 'lm_eval/tasks/glue/sst/default.yaml' 2024-01-03T16:18:53,782 adding 'lm_eval/tasks/glue/wnli/default.yaml' 2024-01-03T16:18:53,784 adding 'lm_eval/tasks/gsm8k/README.md' 2024-01-03T16:18:53,786 adding 'lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml' 2024-01-03T16:18:53,787 adding 'lm_eval/tasks/gsm8k/gsm8k-cot.yaml' 2024-01-03T16:18:53,788 adding 'lm_eval/tasks/gsm8k/gsm8k.yaml' 2024-01-03T16:18:53,790 adding 'lm_eval/tasks/headqa/README.md' 2024-01-03T16:18:53,791 adding 'lm_eval/tasks/headqa/headqa_en.yaml' 2024-01-03T16:18:53,793 adding 'lm_eval/tasks/headqa/headqa_es.yaml' 2024-01-03T16:18:53,795 adding 'lm_eval/tasks/hellaswag/README.md' 2024-01-03T16:18:53,796 adding 'lm_eval/tasks/hellaswag/hellaswag.yaml' 2024-01-03T16:18:53,797 adding 'lm_eval/tasks/hellaswag/utils.py' 2024-01-03T16:18:53,799 adding 'lm_eval/tasks/hendrycks_ethics/README.md' 2024-01-03T16:18:53,800 adding 'lm_eval/tasks/hendrycks_ethics/commonsense.yaml' 2024-01-03T16:18:53,801 adding 'lm_eval/tasks/hendrycks_ethics/deontology.yaml' 2024-01-03T16:18:53,802 adding 'lm_eval/tasks/hendrycks_ethics/justice.yaml' 2024-01-03T16:18:53,803 adding 'lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml' 2024-01-03T16:18:53,805 adding 'lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml' 2024-01-03T16:18:53,806 adding 'lm_eval/tasks/hendrycks_ethics/utils.py' 2024-01-03T16:18:53,807 adding 'lm_eval/tasks/hendrycks_ethics/virtue.yaml' 2024-01-03T16:18:53,809 adding 'lm_eval/tasks/lambada/README.md' 2024-01-03T16:18:53,810 adding 'lm_eval/tasks/lambada/lambada_openai.yaml' 2024-01-03T16:18:53,811 adding 'lm_eval/tasks/lambada/lambada_standard.yaml' 2024-01-03T16:18:53,813 adding 'lm_eval/tasks/lambada_cloze/README.md' 2024-01-03T16:18:53,814 adding 'lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml' 2024-01-03T16:18:53,816 adding 'lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml' 2024-01-03T16:18:53,817 adding 'lm_eval/tasks/lambada_multilingual/README.md' 2024-01-03T16:18:53,818 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml' 2024-01-03T16:18:53,820 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml' 2024-01-03T16:18:53,821 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml' 2024-01-03T16:18:53,822 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml' 2024-01-03T16:18:53,823 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml' 2024-01-03T16:18:53,825 adding 'lm_eval/tasks/logiqa/README.md' 2024-01-03T16:18:53,826 adding 'lm_eval/tasks/logiqa/logiqa.yaml' 2024-01-03T16:18:53,827 adding 'lm_eval/tasks/logiqa/utils_logiqa.py' 2024-01-03T16:18:53,829 adding 'lm_eval/tasks/logiqa2/README.md' 2024-01-03T16:18:53,830 adding 'lm_eval/tasks/logiqa2/logieval.yaml' 2024-01-03T16:18:53,831 adding 'lm_eval/tasks/logiqa2/logiqa2.yaml' 2024-01-03T16:18:53,832 adding 'lm_eval/tasks/logiqa2/utils_logiqa2.py' 2024-01-03T16:18:53,834 adding 'lm_eval/tasks/mathqa/README.md' 2024-01-03T16:18:53,835 adding 'lm_eval/tasks/mathqa/mathqa.yaml' 2024-01-03T16:18:53,836 adding 'lm_eval/tasks/mathqa/utils.py' 2024-01-03T16:18:53,838 adding 'lm_eval/tasks/mc_taco/README.md' 2024-01-03T16:18:53,839 adding 'lm_eval/tasks/mc_taco/default.yaml' 2024-01-03T16:18:53,841 adding 'lm_eval/tasks/mgsm/README.md' 2024-01-03T16:18:53,843 adding 'lm_eval/tasks/mgsm/utils.py' 2024-01-03T16:18:53,845 adding 'lm_eval/tasks/mgsm/direct/direct_yaml' 2024-01-03T16:18:53,846 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml' 2024-01-03T16:18:53,848 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml' 2024-01-03T16:18:53,849 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml' 2024-01-03T16:18:53,850 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml' 2024-01-03T16:18:53,851 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml' 2024-01-03T16:18:53,852 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml' 2024-01-03T16:18:53,853 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml' 2024-01-03T16:18:53,855 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml' 2024-01-03T16:18:53,856 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml' 2024-01-03T16:18:53,857 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml' 2024-01-03T16:18:53,858 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml' 2024-01-03T16:18:53,860 adding 'lm_eval/tasks/mgsm/en_cot/cot_yaml' 2024-01-03T16:18:53,861 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_bn_en-cot.yaml' 2024-01-03T16:18:53,862 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_de_en-cot.yaml' 2024-01-03T16:18:53,863 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_en-cot.yaml' 2024-01-03T16:18:53,865 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_es_en-cot.yaml' 2024-01-03T16:18:53,866 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_fr_en-cot.yaml' 2024-01-03T16:18:53,867 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_ja_en-cot.yaml' 2024-01-03T16:18:53,868 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_ru_en-cot.yaml' 2024-01-03T16:18:53,869 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_sw_en-cot.yaml' 2024-01-03T16:18:53,870 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_te_en-cot.yaml' 2024-01-03T16:18:53,871 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_th_en-cot.yaml' 2024-01-03T16:18:53,873 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_zh_en-cot.yaml' 2024-01-03T16:18:53,875 adding 'lm_eval/tasks/mgsm/native_cot/cot_yaml' 2024-01-03T16:18:53,876 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_bn.yaml' 2024-01-03T16:18:53,877 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_de.yaml' 2024-01-03T16:18:53,878 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_en.yaml' 2024-01-03T16:18:53,879 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_es.yaml' 2024-01-03T16:18:53,880 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_fr.yaml' 2024-01-03T16:18:53,881 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ja.yaml' 2024-01-03T16:18:53,882 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_ru.yaml' 2024-01-03T16:18:53,883 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_sw.yaml' 2024-01-03T16:18:53,885 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_te.yaml' 2024-01-03T16:18:53,886 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_th.yaml' 2024-01-03T16:18:53,887 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_cot_native_zh.yaml' 2024-01-03T16:18:53,889 adding 'lm_eval/tasks/minerva_math/README.md' 2024-01-03T16:18:53,890 adding 'lm_eval/tasks/minerva_math/minerva_math_algebra.yaml' 2024-01-03T16:18:53,891 adding 'lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml' 2024-01-03T16:18:53,892 adding 'lm_eval/tasks/minerva_math/minerva_math_geometry.yaml' 2024-01-03T16:18:53,893 adding 'lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml' 2024-01-03T16:18:53,894 adding 'lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml' 2024-01-03T16:18:53,895 adding 'lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml' 2024-01-03T16:18:53,896 adding 'lm_eval/tasks/minerva_math/minerva_math_precalc.yaml' 2024-01-03T16:18:53,898 adding 'lm_eval/tasks/minerva_math/utils.py' 2024-01-03T16:18:53,900 adding 'lm_eval/tasks/mmlu/_generate_configs.py' 2024-01-03T16:18:53,903 adding 'lm_eval/tasks/mmlu/default/_default_template_yaml' 2024-01-03T16:18:53,904 adding 'lm_eval/tasks/mmlu/default/_mmlu.yaml' 2024-01-03T16:18:53,905 adding 'lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml' 2024-01-03T16:18:53,906 adding 'lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml' 2024-01-03T16:18:53,907 adding 'lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml' 2024-01-03T16:18:53,909 adding 'lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml' 2024-01-03T16:18:53,910 adding 'lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml' 2024-01-03T16:18:53,911 adding 'lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml' 2024-01-03T16:18:53,912 adding 'lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml' 2024-01-03T16:18:53,913 adding 'lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml' 2024-01-03T16:18:53,915 adding 'lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml' 2024-01-03T16:18:53,916 adding 'lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml' 2024-01-03T16:18:53,917 adding 'lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml' 2024-01-03T16:18:53,918 adding 'lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml' 2024-01-03T16:18:53,919 adding 'lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml' 2024-01-03T16:18:53,920 adding 'lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml' 2024-01-03T16:18:53,922 adding 'lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml' 2024-01-03T16:18:53,923 adding 'lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml' 2024-01-03T16:18:53,924 adding 'lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml' 2024-01-03T16:18:53,926 adding 'lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml' 2024-01-03T16:18:53,927 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml' 2024-01-03T16:18:53,928 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml' 2024-01-03T16:18:53,929 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml' 2024-01-03T16:18:53,930 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml' 2024-01-03T16:18:53,932 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml' 2024-01-03T16:18:53,933 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml' 2024-01-03T16:18:53,934 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml' 2024-01-03T16:18:53,935 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml' 2024-01-03T16:18:53,936 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml' 2024-01-03T16:18:53,937 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml' 2024-01-03T16:18:53,939 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml' 2024-01-03T16:18:53,940 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml' 2024-01-03T16:18:53,941 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml' 2024-01-03T16:18:53,942 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml' 2024-01-03T16:18:53,943 adding 'lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml' 2024-01-03T16:18:53,944 adding 'lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml' 2024-01-03T16:18:53,945 adding 'lm_eval/tasks/mmlu/default/mmlu_international_law.yaml' 2024-01-03T16:18:53,946 adding 'lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml' 2024-01-03T16:18:53,948 adding 'lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml' 2024-01-03T16:18:53,949 adding 'lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml' 2024-01-03T16:18:53,950 adding 'lm_eval/tasks/mmlu/default/mmlu_management.yaml' 2024-01-03T16:18:53,951 adding 'lm_eval/tasks/mmlu/default/mmlu_marketing.yaml' 2024-01-03T16:18:53,952 adding 'lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml' 2024-01-03T16:18:53,953 adding 'lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml' 2024-01-03T16:18:53,955 adding 'lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml' 2024-01-03T16:18:53,956 adding 'lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml' 2024-01-03T16:18:53,957 adding 'lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml' 2024-01-03T16:18:53,958 adding 'lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml' 2024-01-03T16:18:53,959 adding 'lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml' 2024-01-03T16:18:53,960 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml' 2024-01-03T16:18:53,961 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml' 2024-01-03T16:18:53,962 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml' 2024-01-03T16:18:53,964 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml' 2024-01-03T16:18:53,965 adding 'lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml' 2024-01-03T16:18:53,966 adding 'lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml' 2024-01-03T16:18:53,967 adding 'lm_eval/tasks/mmlu/default/mmlu_sociology.yaml' 2024-01-03T16:18:53,968 adding 'lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml' 2024-01-03T16:18:53,969 adding 'lm_eval/tasks/mmlu/default/mmlu_virology.yaml' 2024-01-03T16:18:53,970 adding 'lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml' 2024-01-03T16:18:54,010 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json' 2024-01-03T16:18:54,013 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml' 2024-01-03T16:18:54,014 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml' 2024-01-03T16:18:54,016 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml' 2024-01-03T16:18:54,017 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml' 2024-01-03T16:18:54,019 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml' 2024-01-03T16:18:54,020 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml' 2024-01-03T16:18:54,021 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml' 2024-01-03T16:18:54,023 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml' 2024-01-03T16:18:54,024 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml' 2024-01-03T16:18:54,026 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml' 2024-01-03T16:18:54,027 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml' 2024-01-03T16:18:54,029 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml' 2024-01-03T16:18:54,030 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml' 2024-01-03T16:18:54,031 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml' 2024-01-03T16:18:54,033 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml' 2024-01-03T16:18:54,034 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml' 2024-01-03T16:18:54,035 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml' 2024-01-03T16:18:54,037 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml' 2024-01-03T16:18:54,038 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml' 2024-01-03T16:18:54,040 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml' 2024-01-03T16:18:54,041 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml' 2024-01-03T16:18:54,043 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml' 2024-01-03T16:18:54,044 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml' 2024-01-03T16:18:54,047 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml' 2024-01-03T16:18:54,048 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml' 2024-01-03T16:18:54,050 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml' 2024-01-03T16:18:54,051 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml' 2024-01-03T16:18:54,053 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml' 2024-01-03T16:18:54,054 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml' 2024-01-03T16:18:54,056 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml' 2024-01-03T16:18:54,057 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml' 2024-01-03T16:18:54,058 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml' 2024-01-03T16:18:54,061 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml' 2024-01-03T16:18:54,063 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml' 2024-01-03T16:18:54,064 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml' 2024-01-03T16:18:54,065 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml' 2024-01-03T16:18:54,067 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml' 2024-01-03T16:18:54,068 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml' 2024-01-03T16:18:54,070 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml' 2024-01-03T16:18:54,071 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml' 2024-01-03T16:18:54,073 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml' 2024-01-03T16:18:54,075 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml' 2024-01-03T16:18:54,076 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml' 2024-01-03T16:18:54,078 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml' 2024-01-03T16:18:54,079 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml' 2024-01-03T16:18:54,081 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml' 2024-01-03T16:18:54,083 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml' 2024-01-03T16:18:54,084 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml' 2024-01-03T16:18:54,086 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml' 2024-01-03T16:18:54,087 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml' 2024-01-03T16:18:54,090 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml' 2024-01-03T16:18:54,091 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml' 2024-01-03T16:18:54,093 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml' 2024-01-03T16:18:54,094 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml' 2024-01-03T16:18:54,096 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml' 2024-01-03T16:18:54,097 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml' 2024-01-03T16:18:54,098 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml' 2024-01-03T16:18:54,099 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml' 2024-01-03T16:18:54,101 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml' 2024-01-03T16:18:54,103 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml' 2024-01-03T16:18:54,105 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml' 2024-01-03T16:18:54,106 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml' 2024-01-03T16:18:54,107 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml' 2024-01-03T16:18:54,108 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml' 2024-01-03T16:18:54,109 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml' 2024-01-03T16:18:54,110 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml' 2024-01-03T16:18:54,112 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml' 2024-01-03T16:18:54,113 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml' 2024-01-03T16:18:54,114 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml' 2024-01-03T16:18:54,115 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml' 2024-01-03T16:18:54,116 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml' 2024-01-03T16:18:54,117 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml' 2024-01-03T16:18:54,119 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml' 2024-01-03T16:18:54,120 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml' 2024-01-03T16:18:54,121 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml' 2024-01-03T16:18:54,122 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml' 2024-01-03T16:18:54,123 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml' 2024-01-03T16:18:54,124 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml' 2024-01-03T16:18:54,126 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml' 2024-01-03T16:18:54,127 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml' 2024-01-03T16:18:54,128 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml' 2024-01-03T16:18:54,129 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml' 2024-01-03T16:18:54,130 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml' 2024-01-03T16:18:54,132 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml' 2024-01-03T16:18:54,133 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml' 2024-01-03T16:18:54,134 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml' 2024-01-03T16:18:54,135 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml' 2024-01-03T16:18:54,136 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml' 2024-01-03T16:18:54,137 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml' 2024-01-03T16:18:54,139 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml' 2024-01-03T16:18:54,140 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml' 2024-01-03T16:18:54,141 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml' 2024-01-03T16:18:54,142 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml' 2024-01-03T16:18:54,143 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml' 2024-01-03T16:18:54,144 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml' 2024-01-03T16:18:54,146 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml' 2024-01-03T16:18:54,147 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml' 2024-01-03T16:18:54,148 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml' 2024-01-03T16:18:54,149 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml' 2024-01-03T16:18:54,150 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml' 2024-01-03T16:18:54,151 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml' 2024-01-03T16:18:54,152 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml' 2024-01-03T16:18:54,153 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml' 2024-01-03T16:18:54,155 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml' 2024-01-03T16:18:54,156 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml' 2024-01-03T16:18:54,157 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml' 2024-01-03T16:18:54,158 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml' 2024-01-03T16:18:54,159 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml' 2024-01-03T16:18:54,160 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml' 2024-01-03T16:18:54,161 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml' 2024-01-03T16:18:54,162 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml' 2024-01-03T16:18:54,163 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml' 2024-01-03T16:18:54,165 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml' 2024-01-03T16:18:54,166 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml' 2024-01-03T16:18:54,167 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml' 2024-01-03T16:18:54,168 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml' 2024-01-03T16:18:54,169 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml' 2024-01-03T16:18:54,170 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml' 2024-01-03T16:18:54,173 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml' 2024-01-03T16:18:54,174 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml' 2024-01-03T16:18:54,175 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml' 2024-01-03T16:18:54,177 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml' 2024-01-03T16:18:54,178 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml' 2024-01-03T16:18:54,179 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml' 2024-01-03T16:18:54,180 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml' 2024-01-03T16:18:54,181 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml' 2024-01-03T16:18:54,182 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml' 2024-01-03T16:18:54,183 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml' 2024-01-03T16:18:54,185 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml' 2024-01-03T16:18:54,186 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml' 2024-01-03T16:18:54,187 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml' 2024-01-03T16:18:54,188 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml' 2024-01-03T16:18:54,189 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml' 2024-01-03T16:18:54,190 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml' 2024-01-03T16:18:54,192 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml' 2024-01-03T16:18:54,193 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml' 2024-01-03T16:18:54,194 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml' 2024-01-03T16:18:54,195 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml' 2024-01-03T16:18:54,196 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml' 2024-01-03T16:18:54,197 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml' 2024-01-03T16:18:54,198 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml' 2024-01-03T16:18:54,199 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml' 2024-01-03T16:18:54,200 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml' 2024-01-03T16:18:54,201 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml' 2024-01-03T16:18:54,202 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml' 2024-01-03T16:18:54,204 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml' 2024-01-03T16:18:54,205 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml' 2024-01-03T16:18:54,206 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml' 2024-01-03T16:18:54,207 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml' 2024-01-03T16:18:54,208 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml' 2024-01-03T16:18:54,209 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml' 2024-01-03T16:18:54,211 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml' 2024-01-03T16:18:54,212 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml' 2024-01-03T16:18:54,213 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml' 2024-01-03T16:18:54,214 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml' 2024-01-03T16:18:54,215 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml' 2024-01-03T16:18:54,216 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml' 2024-01-03T16:18:54,220 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml' 2024-01-03T16:18:54,221 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml' 2024-01-03T16:18:54,222 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml' 2024-01-03T16:18:54,224 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml' 2024-01-03T16:18:54,225 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml' 2024-01-03T16:18:54,226 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml' 2024-01-03T16:18:54,227 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml' 2024-01-03T16:18:54,228 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml' 2024-01-03T16:18:54,229 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml' 2024-01-03T16:18:54,231 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml' 2024-01-03T16:18:54,232 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml' 2024-01-03T16:18:54,233 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml' 2024-01-03T16:18:54,234 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml' 2024-01-03T16:18:54,235 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml' 2024-01-03T16:18:54,236 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml' 2024-01-03T16:18:54,238 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml' 2024-01-03T16:18:54,239 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml' 2024-01-03T16:18:54,240 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml' 2024-01-03T16:18:54,241 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml' 2024-01-03T16:18:54,242 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml' 2024-01-03T16:18:54,244 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml' 2024-01-03T16:18:54,245 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml' 2024-01-03T16:18:54,246 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml' 2024-01-03T16:18:54,248 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml' 2024-01-03T16:18:54,249 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml' 2024-01-03T16:18:54,250 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml' 2024-01-03T16:18:54,251 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml' 2024-01-03T16:18:54,252 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml' 2024-01-03T16:18:54,253 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml' 2024-01-03T16:18:54,254 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml' 2024-01-03T16:18:54,255 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml' 2024-01-03T16:18:54,256 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml' 2024-01-03T16:18:54,258 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml' 2024-01-03T16:18:54,259 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml' 2024-01-03T16:18:54,260 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml' 2024-01-03T16:18:54,261 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml' 2024-01-03T16:18:54,262 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml' 2024-01-03T16:18:54,263 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml' 2024-01-03T16:18:54,264 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml' 2024-01-03T16:18:54,266 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml' 2024-01-03T16:18:54,267 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml' 2024-01-03T16:18:54,268 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml' 2024-01-03T16:18:54,269 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml' 2024-01-03T16:18:54,270 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml' 2024-01-03T16:18:54,271 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml' 2024-01-03T16:18:54,273 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml' 2024-01-03T16:18:54,274 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml' 2024-01-03T16:18:54,275 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml' 2024-01-03T16:18:54,276 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml' 2024-01-03T16:18:54,277 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml' 2024-01-03T16:18:54,278 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml' 2024-01-03T16:18:54,279 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml' 2024-01-03T16:18:54,281 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml' 2024-01-03T16:18:54,282 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml' 2024-01-03T16:18:54,283 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml' 2024-01-03T16:18:54,284 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml' 2024-01-03T16:18:54,285 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml' 2024-01-03T16:18:54,286 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml' 2024-01-03T16:18:54,287 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml' 2024-01-03T16:18:54,288 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml' 2024-01-03T16:18:54,290 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml' 2024-01-03T16:18:54,291 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml' 2024-01-03T16:18:54,292 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml' 2024-01-03T16:18:54,293 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml' 2024-01-03T16:18:54,294 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml' 2024-01-03T16:18:54,295 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml' 2024-01-03T16:18:54,296 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml' 2024-01-03T16:18:54,297 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml' 2024-01-03T16:18:54,298 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml' 2024-01-03T16:18:54,300 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml' 2024-01-03T16:18:54,301 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml' 2024-01-03T16:18:54,302 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml' 2024-01-03T16:18:54,303 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml' 2024-01-03T16:18:54,304 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml' 2024-01-03T16:18:54,305 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml' 2024-01-03T16:18:54,306 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml' 2024-01-03T16:18:54,308 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml' 2024-01-03T16:18:54,309 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml' 2024-01-03T16:18:54,310 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml' 2024-01-03T16:18:54,313 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py' 2024-01-03T16:18:54,314 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml' 2024-01-03T16:18:54,315 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml' 2024-01-03T16:18:54,316 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml' 2024-01-03T16:18:54,318 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml' 2024-01-03T16:18:54,319 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml' 2024-01-03T16:18:54,320 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml' 2024-01-03T16:18:54,321 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml' 2024-01-03T16:18:54,322 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml' 2024-01-03T16:18:54,323 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml' 2024-01-03T16:18:54,324 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml' 2024-01-03T16:18:54,326 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml' 2024-01-03T16:18:54,327 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml' 2024-01-03T16:18:54,328 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml' 2024-01-03T16:18:54,329 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml' 2024-01-03T16:18:54,330 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml' 2024-01-03T16:18:54,331 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml' 2024-01-03T16:18:54,332 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml' 2024-01-03T16:18:54,333 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml' 2024-01-03T16:18:54,334 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml' 2024-01-03T16:18:54,336 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml' 2024-01-03T16:18:54,337 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml' 2024-01-03T16:18:54,338 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml' 2024-01-03T16:18:54,339 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml' 2024-01-03T16:18:54,340 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml' 2024-01-03T16:18:54,341 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml' 2024-01-03T16:18:54,342 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml' 2024-01-03T16:18:54,343 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml' 2024-01-03T16:18:54,344 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml' 2024-01-03T16:18:54,346 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml' 2024-01-03T16:18:54,347 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml' 2024-01-03T16:18:54,348 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml' 2024-01-03T16:18:54,349 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml' 2024-01-03T16:18:54,351 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml' 2024-01-03T16:18:54,352 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml' 2024-01-03T16:18:54,353 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml' 2024-01-03T16:18:54,354 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml' 2024-01-03T16:18:54,355 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml' 2024-01-03T16:18:54,356 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml' 2024-01-03T16:18:54,357 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml' 2024-01-03T16:18:54,359 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml' 2024-01-03T16:18:54,360 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml' 2024-01-03T16:18:54,361 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml' 2024-01-03T16:18:54,362 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml' 2024-01-03T16:18:54,363 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml' 2024-01-03T16:18:54,364 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml' 2024-01-03T16:18:54,366 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml' 2024-01-03T16:18:54,367 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml' 2024-01-03T16:18:54,368 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml' 2024-01-03T16:18:54,369 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml' 2024-01-03T16:18:54,370 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml' 2024-01-03T16:18:54,374 adding 'lm_eval/tasks/model_written_evals/persona/_generate_configs.py' 2024-01-03T16:18:54,375 adding 'lm_eval/tasks/model_written_evals/persona/_template_yaml' 2024-01-03T16:18:54,376 adding 'lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml' 2024-01-03T16:18:54,377 adding 'lm_eval/tasks/model_written_evals/persona/agreeableness.yaml' 2024-01-03T16:18:54,379 adding 'lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml' 2024-01-03T16:18:54,380 adding 'lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml' 2024-01-03T16:18:54,381 adding 'lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml' 2024-01-03T16:18:54,382 adding 'lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml' 2024-01-03T16:18:54,383 adding 'lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml' 2024-01-03T16:18:54,384 adding 'lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml' 2024-01-03T16:18:54,385 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml' 2024-01-03T16:18:54,386 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml' 2024-01-03T16:18:54,387 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml' 2024-01-03T16:18:54,388 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml' 2024-01-03T16:18:54,390 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml' 2024-01-03T16:18:54,391 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml' 2024-01-03T16:18:54,392 adding 'lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml' 2024-01-03T16:18:54,393 adding 'lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml' 2024-01-03T16:18:54,394 adding 'lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml' 2024-01-03T16:18:54,395 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml' 2024-01-03T16:18:54,397 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml' 2024-01-03T16:18:54,398 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml' 2024-01-03T16:18:54,399 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml' 2024-01-03T16:18:54,400 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml' 2024-01-03T16:18:54,401 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml' 2024-01-03T16:18:54,402 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml' 2024-01-03T16:18:54,404 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml' 2024-01-03T16:18:54,405 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml' 2024-01-03T16:18:54,406 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml' 2024-01-03T16:18:54,407 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml' 2024-01-03T16:18:54,408 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml' 2024-01-03T16:18:54,409 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml' 2024-01-03T16:18:54,410 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml' 2024-01-03T16:18:54,412 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml' 2024-01-03T16:18:54,413 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml' 2024-01-03T16:18:54,414 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml' 2024-01-03T16:18:54,415 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml' 2024-01-03T16:18:54,416 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml' 2024-01-03T16:18:54,417 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml' 2024-01-03T16:18:54,419 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml' 2024-01-03T16:18:54,420 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml' 2024-01-03T16:18:54,421 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml' 2024-01-03T16:18:54,422 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml' 2024-01-03T16:18:54,423 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml' 2024-01-03T16:18:54,424 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml' 2024-01-03T16:18:54,425 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml' 2024-01-03T16:18:54,426 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml' 2024-01-03T16:18:54,427 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml' 2024-01-03T16:18:54,429 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml' 2024-01-03T16:18:54,430 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml' 2024-01-03T16:18:54,431 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml' 2024-01-03T16:18:54,432 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml' 2024-01-03T16:18:54,433 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml' 2024-01-03T16:18:54,434 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml' 2024-01-03T16:18:54,435 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml' 2024-01-03T16:18:54,436 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml' 2024-01-03T16:18:54,438 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml' 2024-01-03T16:18:54,439 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml' 2024-01-03T16:18:54,440 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml' 2024-01-03T16:18:54,441 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml' 2024-01-03T16:18:54,442 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml' 2024-01-03T16:18:54,443 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml' 2024-01-03T16:18:54,444 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml' 2024-01-03T16:18:54,446 adding 'lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml' 2024-01-03T16:18:54,447 adding 'lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml' 2024-01-03T16:18:54,447 adding 'lm_eval/tasks/model_written_evals/persona/extraversion.yaml' 2024-01-03T16:18:54,448 adding 'lm_eval/tasks/model_written_evals/persona/has-disability.yaml' 2024-01-03T16:18:54,449 adding 'lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml' 2024-01-03T16:18:54,450 adding 'lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml' 2024-01-03T16:18:54,451 adding 'lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml' 2024-01-03T16:18:54,452 adding 'lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml' 2024-01-03T16:18:54,453 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml' 2024-01-03T16:18:54,455 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml' 2024-01-03T16:18:54,456 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml' 2024-01-03T16:18:54,457 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml' 2024-01-03T16:18:54,458 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml' 2024-01-03T16:18:54,459 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml' 2024-01-03T16:18:54,460 adding 'lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml' 2024-01-03T16:18:54,461 adding 'lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml' 2024-01-03T16:18:54,462 adding 'lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml' 2024-01-03T16:18:54,463 adding 'lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml' 2024-01-03T16:18:54,465 adding 'lm_eval/tasks/model_written_evals/persona/narcissism.yaml' 2024-01-03T16:18:54,466 adding 'lm_eval/tasks/model_written_evals/persona/neuroticism.yaml' 2024-01-03T16:18:54,467 adding 'lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml' 2024-01-03T16:18:54,468 adding 'lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml' 2024-01-03T16:18:54,469 adding 'lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml' 2024-01-03T16:18:54,470 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml' 2024-01-03T16:18:54,471 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml' 2024-01-03T16:18:54,472 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml' 2024-01-03T16:18:54,473 adding 'lm_eval/tasks/model_written_evals/persona/openness.yaml' 2024-01-03T16:18:54,475 adding 'lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml' 2024-01-03T16:18:54,476 adding 'lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml' 2024-01-03T16:18:54,477 adding 'lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml' 2024-01-03T16:18:54,478 adding 'lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml' 2024-01-03T16:18:54,479 adding 'lm_eval/tasks/model_written_evals/persona/psychopathy.yaml' 2024-01-03T16:18:54,480 adding 'lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml' 2024-01-03T16:18:54,481 adding 'lm_eval/tasks/model_written_evals/persona/risk-averse.yaml' 2024-01-03T16:18:54,482 adding 'lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml' 2024-01-03T16:18:54,484 adding 'lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml' 2024-01-03T16:18:54,485 adding 'lm_eval/tasks/model_written_evals/persona/self-replication.yaml' 2024-01-03T16:18:54,486 adding 'lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml' 2024-01-03T16:18:54,487 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml' 2024-01-03T16:18:54,488 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml' 2024-01-03T16:18:54,489 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml' 2024-01-03T16:18:54,490 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml' 2024-01-03T16:18:54,491 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml' 2024-01-03T16:18:54,492 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml' 2024-01-03T16:18:54,493 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml' 2024-01-03T16:18:54,494 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml' 2024-01-03T16:18:54,495 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml' 2024-01-03T16:18:54,496 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml' 2024-01-03T16:18:54,497 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml' 2024-01-03T16:18:54,499 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml' 2024-01-03T16:18:54,500 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml' 2024-01-03T16:18:54,501 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml' 2024-01-03T16:18:54,502 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml' 2024-01-03T16:18:54,503 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml' 2024-01-03T16:18:54,504 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml' 2024-01-03T16:18:54,505 adding 'lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml' 2024-01-03T16:18:54,506 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml' 2024-01-03T16:18:54,508 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml' 2024-01-03T16:18:54,509 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml' 2024-01-03T16:18:54,510 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml' 2024-01-03T16:18:54,511 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml' 2024-01-03T16:18:54,512 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml' 2024-01-03T16:18:54,513 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml' 2024-01-03T16:18:54,514 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml' 2024-01-03T16:18:54,515 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml' 2024-01-03T16:18:54,517 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml' 2024-01-03T16:18:54,518 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml' 2024-01-03T16:18:54,519 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml' 2024-01-03T16:18:54,520 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml' 2024-01-03T16:18:54,521 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml' 2024-01-03T16:18:54,522 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml' 2024-01-03T16:18:54,523 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml' 2024-01-03T16:18:54,525 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml' 2024-01-03T16:18:54,526 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml' 2024-01-03T16:18:54,527 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml' 2024-01-03T16:18:54,528 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml' 2024-01-03T16:18:54,529 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml' 2024-01-03T16:18:54,531 adding 'lm_eval/tasks/model_written_evals/winogenerated/_template_yaml' 2024-01-03T16:18:54,533 adding 'lm_eval/tasks/mutual/README.md' 2024-01-03T16:18:54,534 adding 'lm_eval/tasks/mutual/multual_plus.yaml' 2024-01-03T16:18:54,535 adding 'lm_eval/tasks/mutual/mutual.yaml' 2024-01-03T16:18:54,536 adding 'lm_eval/tasks/mutual/utils.py' 2024-01-03T16:18:54,537 adding 'lm_eval/tasks/nq_open/README.md' 2024-01-03T16:18:54,539 adding 'lm_eval/tasks/nq_open/nq_open.yaml' 2024-01-03T16:18:54,540 adding 'lm_eval/tasks/openbookqa/README.md' 2024-01-03T16:18:54,542 adding 'lm_eval/tasks/openbookqa/openbookqa.yaml' 2024-01-03T16:18:54,543 adding 'lm_eval/tasks/paws-x/README.md' 2024-01-03T16:18:54,545 adding 'lm_eval/tasks/paws-x/_generate_config.py' 2024-01-03T16:18:54,546 adding 'lm_eval/tasks/paws-x/paws_de.yaml' 2024-01-03T16:18:54,547 adding 'lm_eval/tasks/paws-x/paws_en.yaml' 2024-01-03T16:18:54,548 adding 'lm_eval/tasks/paws-x/paws_es.yaml' 2024-01-03T16:18:54,549 adding 'lm_eval/tasks/paws-x/paws_fr.yaml' 2024-01-03T16:18:54,550 adding 'lm_eval/tasks/paws-x/paws_ja.yaml' 2024-01-03T16:18:54,551 adding 'lm_eval/tasks/paws-x/paws_ko.yaml' 2024-01-03T16:18:54,552 adding 'lm_eval/tasks/paws-x/paws_zh.yaml' 2024-01-03T16:18:54,554 adding 'lm_eval/tasks/paws-x/pawsx_template_yaml' 2024-01-03T16:18:54,556 adding 'lm_eval/tasks/pile/README.md' 2024-01-03T16:18:54,557 adding 'lm_eval/tasks/pile/pile_arxiv.yaml' 2024-01-03T16:18:54,558 adding 'lm_eval/tasks/pile/pile_bookcorpus2.yaml' 2024-01-03T16:18:54,559 adding 'lm_eval/tasks/pile/pile_books3.yaml' 2024-01-03T16:18:54,560 adding 'lm_eval/tasks/pile/pile_dm-mathematics.yaml' 2024-01-03T16:18:54,561 adding 'lm_eval/tasks/pile/pile_enron.yaml' 2024-01-03T16:18:54,562 adding 'lm_eval/tasks/pile/pile_europarl.yaml' 2024-01-03T16:18:54,564 adding 'lm_eval/tasks/pile/pile_freelaw.yaml' 2024-01-03T16:18:54,565 adding 'lm_eval/tasks/pile/pile_github.yaml' 2024-01-03T16:18:54,566 adding 'lm_eval/tasks/pile/pile_gutenberg.yaml' 2024-01-03T16:18:54,567 adding 'lm_eval/tasks/pile/pile_hackernews.yaml' 2024-01-03T16:18:54,568 adding 'lm_eval/tasks/pile/pile_nih-exporter.yaml' 2024-01-03T16:18:54,569 adding 'lm_eval/tasks/pile/pile_opensubtitles.yaml' 2024-01-03T16:18:54,570 adding 'lm_eval/tasks/pile/pile_openwebtext2.yaml' 2024-01-03T16:18:54,571 adding 'lm_eval/tasks/pile/pile_philpapers.yaml' 2024-01-03T16:18:54,572 adding 'lm_eval/tasks/pile/pile_pile-cc.yaml' 2024-01-03T16:18:54,573 adding 'lm_eval/tasks/pile/pile_pubmed-abstracts.yaml' 2024-01-03T16:18:54,575 adding 'lm_eval/tasks/pile/pile_pubmed-central.yaml' 2024-01-03T16:18:54,576 adding 'lm_eval/tasks/pile/pile_stackexchange.yaml' 2024-01-03T16:18:54,577 adding 'lm_eval/tasks/pile/pile_ubuntu-irc.yaml' 2024-01-03T16:18:54,578 adding 'lm_eval/tasks/pile/pile_uspto.yaml' 2024-01-03T16:18:54,579 adding 'lm_eval/tasks/pile/pile_wikipedia.yaml' 2024-01-03T16:18:54,580 adding 'lm_eval/tasks/pile/pile_youtubesubtitles.yaml' 2024-01-03T16:18:54,582 adding 'lm_eval/tasks/piqa/README.md' 2024-01-03T16:18:54,583 adding 'lm_eval/tasks/piqa/piqa.yaml' 2024-01-03T16:18:54,585 adding 'lm_eval/tasks/polemo2/README.md' 2024-01-03T16:18:54,586 adding 'lm_eval/tasks/polemo2/polemo2_in.yaml' 2024-01-03T16:18:54,587 adding 'lm_eval/tasks/polemo2/polemo2_out.yaml' 2024-01-03T16:18:54,589 adding 'lm_eval/tasks/prost/README.md' 2024-01-03T16:18:54,590 adding 'lm_eval/tasks/prost/corypaik_prost.yaml' 2024-01-03T16:18:54,592 adding 'lm_eval/tasks/pubmedqa/README.md' 2024-01-03T16:18:54,593 adding 'lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py' 2024-01-03T16:18:54,594 adding 'lm_eval/tasks/pubmedqa/pubmedqa.yaml' 2024-01-03T16:18:54,596 adding 'lm_eval/tasks/qa4mre/README.md' 2024-01-03T16:18:54,597 adding 'lm_eval/tasks/qa4mre/preprocess_qa4mre.py' 2024-01-03T16:18:54,598 adding 'lm_eval/tasks/qa4mre/qa4mre_2011.yaml' 2024-01-03T16:18:54,599 adding 'lm_eval/tasks/qa4mre/qa4mre_2012.yaml' 2024-01-03T16:18:54,600 adding 'lm_eval/tasks/qa4mre/qa4mre_2013.yaml' 2024-01-03T16:18:54,602 adding 'lm_eval/tasks/qasper/README.md' 2024-01-03T16:18:54,603 adding 'lm_eval/tasks/qasper/bool.yaml' 2024-01-03T16:18:54,604 adding 'lm_eval/tasks/qasper/freeform.yaml' 2024-01-03T16:18:54,605 adding 'lm_eval/tasks/qasper/metrics.py' 2024-01-03T16:18:54,607 adding 'lm_eval/tasks/qasper/utils.py' 2024-01-03T16:18:54,608 adding 'lm_eval/tasks/race/README.md' 2024-01-03T16:18:54,610 adding 'lm_eval/tasks/race/preprocess_race.py' 2024-01-03T16:18:54,611 adding 'lm_eval/tasks/race/race.yaml' 2024-01-03T16:18:54,612 adding 'lm_eval/tasks/realtoxicityprompts/metric.py' 2024-01-03T16:18:54,614 adding 'lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml' 2024-01-03T16:18:54,615 adding 'lm_eval/tasks/sciq/README.md' 2024-01-03T16:18:54,616 adding 'lm_eval/tasks/sciq/sciq.yaml' 2024-01-03T16:18:54,618 adding 'lm_eval/tasks/scrolls/README.md' 2024-01-03T16:18:54,619 adding 'lm_eval/tasks/scrolls/scrolls.yaml' 2024-01-03T16:18:54,621 adding 'lm_eval/tasks/scrolls/task.py' 2024-01-03T16:18:54,623 adding 'lm_eval/tasks/siqa/README.md' 2024-01-03T16:18:54,624 adding 'lm_eval/tasks/siqa/default.yml' 2024-01-03T16:18:54,626 adding 'lm_eval/tasks/squadv2/README.md' 2024-01-03T16:18:54,628 adding 'lm_eval/tasks/squadv2/task.py' 2024-01-03T16:18:54,630 adding 'lm_eval/tasks/storycloze/README.md' 2024-01-03T16:18:54,631 adding 'lm_eval/tasks/storycloze/storycloze_2016.yaml' 2024-01-03T16:18:54,632 adding 'lm_eval/tasks/storycloze/storycloze_2018.yaml' 2024-01-03T16:18:54,635 adding 'lm_eval/tasks/super_glue/README.md' 2024-01-03T16:18:54,636 adding 'lm_eval/tasks/super_glue/boolq/default.yaml' 2024-01-03T16:18:54,637 adding 'lm_eval/tasks/super_glue/boolq/seq2seq.yaml' 2024-01-03T16:18:54,639 adding 'lm_eval/tasks/super_glue/boolq/t5-prompt.yaml' 2024-01-03T16:18:54,640 adding 'lm_eval/tasks/super_glue/cb/aggregate.py' 2024-01-03T16:18:54,641 adding 'lm_eval/tasks/super_glue/cb/default.yaml' 2024-01-03T16:18:54,643 adding 'lm_eval/tasks/super_glue/cb/t5-prompt.yaml' 2024-01-03T16:18:54,644 adding 'lm_eval/tasks/super_glue/cb/t5_utils.py' 2024-01-03T16:18:54,645 adding 'lm_eval/tasks/super_glue/copa/default.yaml' 2024-01-03T16:18:54,647 adding 'lm_eval/tasks/super_glue/copa/t5-prompt.yaml' 2024-01-03T16:18:54,648 adding 'lm_eval/tasks/super_glue/copa/utils.py' 2024-01-03T16:18:54,649 adding 'lm_eval/tasks/super_glue/multirc/default.yaml' 2024-01-03T16:18:54,651 adding 'lm_eval/tasks/super_glue/multirc/t5-prompt.yaml' 2024-01-03T16:18:54,652 adding 'lm_eval/tasks/super_glue/multirc/t5_utils.py' 2024-01-03T16:18:54,654 adding 'lm_eval/tasks/super_glue/record/default.yaml' 2024-01-03T16:18:54,655 adding 'lm_eval/tasks/super_glue/record/t5-prompt.yaml' 2024-01-03T16:18:54,656 adding 'lm_eval/tasks/super_glue/record/t5_utils.py' 2024-01-03T16:18:54,657 adding 'lm_eval/tasks/super_glue/record/util.py' 2024-01-03T16:18:54,659 adding 'lm_eval/tasks/super_glue/rte/default.yaml' 2024-01-03T16:18:54,660 adding 'lm_eval/tasks/super_glue/rte/t5-prompt.yaml' 2024-01-03T16:18:54,662 adding 'lm_eval/tasks/super_glue/wic/default.yaml' 2024-01-03T16:18:54,663 adding 'lm_eval/tasks/super_glue/wic/t5-prompt.yaml' 2024-01-03T16:18:54,665 adding 'lm_eval/tasks/super_glue/wsc/default.yaml' 2024-01-03T16:18:54,666 adding 'lm_eval/tasks/super_glue/wsc/preprocess_wsc.py' 2024-01-03T16:18:54,667 adding 'lm_eval/tasks/super_glue/wsc/t5-prompt.yaml' 2024-01-03T16:18:54,669 adding 'lm_eval/tasks/super_glue/wsc/t5_utils.py' 2024-01-03T16:18:54,670 adding 'lm_eval/tasks/swag/README.md' 2024-01-03T16:18:54,671 adding 'lm_eval/tasks/swag/swag.yaml' 2024-01-03T16:18:54,673 adding 'lm_eval/tasks/toxigen/README.md' 2024-01-03T16:18:54,674 adding 'lm_eval/tasks/toxigen/toxigen.yaml' 2024-01-03T16:18:54,675 adding 'lm_eval/tasks/toxigen/utils.py' 2024-01-03T16:18:54,677 adding 'lm_eval/tasks/translation/README.md' 2024-01-03T16:18:54,678 adding 'lm_eval/tasks/translation/iwslt2017_ar-en.yaml' 2024-01-03T16:18:54,679 adding 'lm_eval/tasks/translation/iwslt2017_en-ar.yaml' 2024-01-03T16:18:54,680 adding 'lm_eval/tasks/translation/utils.py' 2024-01-03T16:18:54,681 adding 'lm_eval/tasks/translation/wmt14_en-fr.yaml' 2024-01-03T16:18:54,682 adding 'lm_eval/tasks/translation/wmt14_fr-en.yaml' 2024-01-03T16:18:54,683 adding 'lm_eval/tasks/translation/wmt16_de-en.yaml' 2024-01-03T16:18:54,685 adding 'lm_eval/tasks/translation/wmt16_en-de.yaml' 2024-01-03T16:18:54,686 adding 'lm_eval/tasks/translation/wmt16_en-ro.yaml' 2024-01-03T16:18:54,687 adding 'lm_eval/tasks/translation/wmt16_ro-en.yaml' 2024-01-03T16:18:54,688 adding 'lm_eval/tasks/translation/wmt_common_yaml' 2024-01-03T16:18:54,690 adding 'lm_eval/tasks/triviaqa/README.md' 2024-01-03T16:18:54,691 adding 'lm_eval/tasks/triviaqa/default.yaml' 2024-01-03T16:18:54,693 adding 'lm_eval/tasks/truthfulqa/README.md' 2024-01-03T16:18:54,694 adding 'lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml' 2024-01-03T16:18:54,695 adding 'lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml' 2024-01-03T16:18:54,696 adding 'lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml' 2024-01-03T16:18:54,698 adding 'lm_eval/tasks/truthfulqa/utils.py' 2024-01-03T16:18:54,700 adding 'lm_eval/tasks/unscramble/README.md' 2024-01-03T16:18:54,701 adding 'lm_eval/tasks/unscramble/anagrams1.yaml' 2024-01-03T16:18:54,702 adding 'lm_eval/tasks/unscramble/anagrams2.yaml' 2024-01-03T16:18:54,703 adding 'lm_eval/tasks/unscramble/cycle_letters.yaml' 2024-01-03T16:18:54,704 adding 'lm_eval/tasks/unscramble/random_insertion.yaml' 2024-01-03T16:18:54,706 adding 'lm_eval/tasks/unscramble/reversed_words.yaml' 2024-01-03T16:18:54,707 adding 'lm_eval/tasks/webqs/README.md' 2024-01-03T16:18:54,709 adding 'lm_eval/tasks/webqs/utils.py' 2024-01-03T16:18:54,710 adding 'lm_eval/tasks/webqs/webqs.yaml' 2024-01-03T16:18:54,712 adding 'lm_eval/tasks/wikitext/README.md' 2024-01-03T16:18:54,713 adding 'lm_eval/tasks/wikitext/preprocess_wikitext.py' 2024-01-03T16:18:54,714 adding 'lm_eval/tasks/wikitext/wikitext.yaml' 2024-01-03T16:18:54,716 adding 'lm_eval/tasks/winogrande/README.md' 2024-01-03T16:18:54,717 adding 'lm_eval/tasks/winogrande/default.yaml' 2024-01-03T16:18:54,718 adding 'lm_eval/tasks/winogrande/preprocess_winogrande.py' 2024-01-03T16:18:54,720 adding 'lm_eval/tasks/wmt2016/README.md' 2024-01-03T16:18:54,721 adding 'lm_eval/tasks/wmt2016/metrics.py' 2024-01-03T16:18:54,722 adding 'lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml' 2024-01-03T16:18:54,724 adding 'lm_eval/tasks/wsc273/README.md' 2024-01-03T16:18:54,725 adding 'lm_eval/tasks/wsc273/default.yaml' 2024-01-03T16:18:54,727 adding 'lm_eval/tasks/wsc273/utils.py' 2024-01-03T16:18:54,729 adding 'lm_eval/tasks/xcopa/README.md' 2024-01-03T16:18:54,730 adding 'lm_eval/tasks/xcopa/default_et.yaml' 2024-01-03T16:18:54,731 adding 'lm_eval/tasks/xcopa/default_ht.yaml' 2024-01-03T16:18:54,732 adding 'lm_eval/tasks/xcopa/default_id.yaml' 2024-01-03T16:18:54,733 adding 'lm_eval/tasks/xcopa/default_it.yaml' 2024-01-03T16:18:54,734 adding 'lm_eval/tasks/xcopa/default_qu.yaml' 2024-01-03T16:18:54,736 adding 'lm_eval/tasks/xcopa/default_sw.yaml' 2024-01-03T16:18:54,737 adding 'lm_eval/tasks/xcopa/default_ta.yaml' 2024-01-03T16:18:54,738 adding 'lm_eval/tasks/xcopa/default_th.yaml' 2024-01-03T16:18:54,739 adding 'lm_eval/tasks/xcopa/default_tr.yaml' 2024-01-03T16:18:54,740 adding 'lm_eval/tasks/xcopa/default_vi.yaml' 2024-01-03T16:18:54,741 adding 'lm_eval/tasks/xcopa/default_zh.yaml' 2024-01-03T16:18:54,742 adding 'lm_eval/tasks/xcopa/utils.py' 2024-01-03T16:18:54,744 adding 'lm_eval/tasks/xnli/README.md' 2024-01-03T16:18:54,745 adding 'lm_eval/tasks/xnli/utils.py' 2024-01-03T16:18:54,746 adding 'lm_eval/tasks/xnli/xnli_ar.yaml' 2024-01-03T16:18:54,748 adding 'lm_eval/tasks/xnli/xnli_bg.yaml' 2024-01-03T16:18:54,749 adding 'lm_eval/tasks/xnli/xnli_common_yaml' 2024-01-03T16:18:54,750 adding 'lm_eval/tasks/xnli/xnli_de.yaml' 2024-01-03T16:18:54,751 adding 'lm_eval/tasks/xnli/xnli_el.yaml' 2024-01-03T16:18:54,752 adding 'lm_eval/tasks/xnli/xnli_en.yaml' 2024-01-03T16:18:54,753 adding 'lm_eval/tasks/xnli/xnli_es.yaml' 2024-01-03T16:18:54,754 adding 'lm_eval/tasks/xnli/xnli_fr.yaml' 2024-01-03T16:18:54,755 adding 'lm_eval/tasks/xnli/xnli_hi.yaml' 2024-01-03T16:18:54,756 adding 'lm_eval/tasks/xnli/xnli_ru.yaml' 2024-01-03T16:18:54,757 adding 'lm_eval/tasks/xnli/xnli_sw.yaml' 2024-01-03T16:18:54,758 adding 'lm_eval/tasks/xnli/xnli_th.yaml' 2024-01-03T16:18:54,759 adding 'lm_eval/tasks/xnli/xnli_tr.yaml' 2024-01-03T16:18:54,760 adding 'lm_eval/tasks/xnli/xnli_ur.yaml' 2024-01-03T16:18:54,761 adding 'lm_eval/tasks/xnli/xnli_vi.yaml' 2024-01-03T16:18:54,762 adding 'lm_eval/tasks/xnli/xnli_zh.yaml' 2024-01-03T16:18:54,764 adding 'lm_eval/tasks/xstorycloze/README.md' 2024-01-03T16:18:54,765 adding 'lm_eval/tasks/xstorycloze/default_ar.yaml' 2024-01-03T16:18:54,766 adding 'lm_eval/tasks/xstorycloze/default_en.yaml' 2024-01-03T16:18:54,767 adding 'lm_eval/tasks/xstorycloze/default_es.yaml' 2024-01-03T16:18:54,768 adding 'lm_eval/tasks/xstorycloze/default_eu.yaml' 2024-01-03T16:18:54,770 adding 'lm_eval/tasks/xstorycloze/default_hi.yaml' 2024-01-03T16:18:54,771 adding 'lm_eval/tasks/xstorycloze/default_id.yaml' 2024-01-03T16:18:54,772 adding 'lm_eval/tasks/xstorycloze/default_my.yaml' 2024-01-03T16:18:54,773 adding 'lm_eval/tasks/xstorycloze/default_ru.yaml' 2024-01-03T16:18:54,774 adding 'lm_eval/tasks/xstorycloze/default_sw.yaml' 2024-01-03T16:18:54,775 adding 'lm_eval/tasks/xstorycloze/default_te.yaml' 2024-01-03T16:18:54,776 adding 'lm_eval/tasks/xstorycloze/default_zh.yaml' 2024-01-03T16:18:54,778 adding 'lm_eval/tasks/xwinograd/README.md' 2024-01-03T16:18:54,779 adding 'lm_eval/tasks/xwinograd/utils.py' 2024-01-03T16:18:54,781 adding 'lm_eval/tasks/xwinograd/xwinograd_common_yaml' 2024-01-03T16:18:54,782 adding 'lm_eval/tasks/xwinograd/xwinograd_en.yaml' 2024-01-03T16:18:54,783 adding 'lm_eval/tasks/xwinograd/xwinograd_fr.yaml' 2024-01-03T16:18:54,784 adding 'lm_eval/tasks/xwinograd/xwinograd_jp.yaml' 2024-01-03T16:18:54,785 adding 'lm_eval/tasks/xwinograd/xwinograd_pt.yaml' 2024-01-03T16:18:54,786 adding 'lm_eval/tasks/xwinograd/xwinograd_ru.yaml' 2024-01-03T16:18:54,787 adding 'lm_eval/tasks/xwinograd/xwinograd_zh.yaml' 2024-01-03T16:18:54,789 adding 'lm_eval-0.4.0.dist-info/LICENSE.md' 2024-01-03T16:18:54,792 adding 'lm_eval-0.4.0.dist-info/METADATA' 2024-01-03T16:18:54,794 adding 'lm_eval-0.4.0.dist-info/WHEEL' 2024-01-03T16:18:54,794 adding 'lm_eval-0.4.0.dist-info/entry_points.txt' 2024-01-03T16:18:54,795 adding 'lm_eval-0.4.0.dist-info/top_level.txt' 2024-01-03T16:18:54,825 adding 'lm_eval-0.4.0.dist-info/RECORD' 2024-01-03T16:18:54,856 removing build/bdist.linux-armv7l/wheel 2024-01-03T16:18:55,329 Building wheel for lm-eval (pyproject.toml): finished with status 'done' 2024-01-03T16:18:55,351 Created wheel for lm-eval: filename=lm_eval-0.4.0-py3-none-any.whl size=994996 sha256=1a863b5f478f2e66e921dbbf2e4dcda06a923021763f01b59c5b03bdd3af01c2 2024-01-03T16:18:55,352 Stored in directory: /tmp/pip-ephem-wheel-cache-jbj6cmjq/wheels/c0/0c/4a/43c6bc9f3ad182a3fcd95a3a1607f6138f98905e1c5e3dcaaf 2024-01-03T16:18:55,413 Successfully built lm-eval 2024-01-03T16:18:55,440 Removed build tracker: '/tmp/pip-build-tracker-pk8v8s_l'