2024-03-18T14:19:56,850 Created temporary directory: /tmp/pip-build-tracker-9msy7ctu 2024-03-18T14:19:56,851 Initialized build tracking at /tmp/pip-build-tracker-9msy7ctu 2024-03-18T14:19:56,851 Created build tracker: /tmp/pip-build-tracker-9msy7ctu 2024-03-18T14:19:56,852 Entered build tracker: /tmp/pip-build-tracker-9msy7ctu 2024-03-18T14:19:56,853 Created temporary directory: /tmp/pip-wheel-p4s5x_b9 2024-03-18T14:19:56,856 Created temporary directory: /tmp/pip-ephem-wheel-cache-z2uh4avv 2024-03-18T14:19:56,878 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2024-03-18T14:19:56,881 2 location(s) to search for versions of lm-eval: 2024-03-18T14:19:56,881 * https://pypi.org/simple/lm-eval/ 2024-03-18T14:19:56,881 * https://www.piwheels.org/simple/lm-eval/ 2024-03-18T14:19:56,882 Fetching project page and analyzing links: https://pypi.org/simple/lm-eval/ 2024-03-18T14:19:56,883 Getting page https://pypi.org/simple/lm-eval/ 2024-03-18T14:19:56,884 Found index url https://pypi.org/simple/ 2024-03-18T14:19:57,035 Fetched page https://pypi.org/simple/lm-eval/ as application/vnd.pypi.simple.v1+json 2024-03-18T14:19:57,039 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/1b/5f/7841febb99c12ffb453d33a67b9841e89dba18c388b644bf22b81d137fc4/lm_eval-0.0.1-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,040 Found link https://files.pythonhosted.org/packages/21/5a/feb5ff3a1591ca963c54873d39116b0e6a4f80e493e961ac08569709c5d7/lm_eval-0.0.1.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.0.1 2024-03-18T14:19:57,040 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/f3/a7/63cbce8b51de25fabb1c49f3a3fd1704faaacadb5ed816401f800e4d2dbd/lm_eval-0.2.0-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,041 Found link https://files.pythonhosted.org/packages/c5/fd/edd21b0f258b4ec0260f99f5b2ac3864f7cddc8fb7c83bbb2379a6aab975/lm_eval-0.2.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.2.0 2024-03-18T14:19:57,042 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/61/c5/bff92e6b61fc2b0c1b7ac769633731910152e5176a404912ce7c07329ba0/lm_eval-0.3.0-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,043 Found link https://files.pythonhosted.org/packages/c4/f8/58abc65390a758c8c2e5f1d8bb9b58d7885d02535d5f48de27006453d07e/lm_eval-0.3.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.6), version: 0.3.0 2024-03-18T14:19:57,044 Found link https://files.pythonhosted.org/packages/45/e0/05001c2e56e2f8f793189442176432ddb89a2f1f1ef00ea154d0bc00fe37/lm_eval-0.4.0.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8), version: 0.4.0 2024-03-18T14:19:57,045 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/49/2d/39f7a25ab663cb45cfc7773b85980f01df44853cc427d00dce94c90b43e6/lm_eval-0.4.1-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8) 2024-03-18T14:19:57,046 Found link https://files.pythonhosted.org/packages/5a/02/1c7f1ac2f139f4c05af5b94e2c4f88a70404fa0b0c22a5fb04dec0216b03/lm_eval-0.4.1.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8), version: 0.4.1 2024-03-18T14:19:57,047 Skipping link: No binaries permitted for lm-eval: https://files.pythonhosted.org/packages/0c/5a/64cf703b62ac7ada09a514c16c7136bb4ea7ef3030eb4c3d780900a5a634/lm_eval-0.4.2-py3-none-any.whl (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8) 2024-03-18T14:19:57,048 Found link https://files.pythonhosted.org/packages/75/ca/18814743ba3b42d19d8524a4f771e9c3a7aa02cd4c579747a0f513907205/lm_eval-0.4.2.tar.gz (from https://pypi.org/simple/lm-eval/) (requires-python:>=3.8), version: 0.4.2 2024-03-18T14:19:57,049 Fetching project page and analyzing links: https://www.piwheels.org/simple/lm-eval/ 2024-03-18T14:19:57,049 Getting page https://www.piwheels.org/simple/lm-eval/ 2024-03-18T14:19:57,051 Found index url https://www.piwheels.org/simple/ 2024-03-18T14:19:57,278 Fetched page https://www.piwheels.org/simple/lm-eval/ as text/html 2024-03-18T14:19:57,280 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.4.1-py3-none-any.whl#sha256=796615edf52f5673187a884a9d77c7d755fe95ca2d16fe77a1f042730eb957b3 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.8) 2024-03-18T14:19:57,281 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.4.0-py3-none-any.whl#sha256=1a863b5f478f2e66e921dbbf2e4dcda06a923021763f01b59c5b03bdd3af01c2 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.8) 2024-03-18T14:19:57,282 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.3.0-py3-none-any.whl#sha256=498b8b8954c1f9c17f46e3ec096e9be6b9c96ee70560ee613a4eb9c7b9d31644 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,282 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.2.0-py3-none-any.whl#sha256=e06d3d7b6016be832e6889cbc9c4787b99156ba57f8feb31d3aeb27304c6558c (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,282 Skipping link: No binaries permitted for lm-eval: https://www.piwheels.org/simple/lm-eval/lm_eval-0.0.1-py3-none-any.whl#sha256=0afc289f69286f71017fb9811dfea6cda7c703bf693ad43106fbbff1f164cf14 (from https://www.piwheels.org/simple/lm-eval/) (requires-python:>=3.6) 2024-03-18T14:19:57,283 Skipping link: not a file: https://www.piwheels.org/simple/lm-eval/ 2024-03-18T14:19:57,284 Skipping link: not a file: https://pypi.org/simple/lm-eval/ 2024-03-18T14:19:57,302 Given no hashes to check 1 links for project 'lm-eval': discarding no candidates 2024-03-18T14:19:57,320 Collecting lm-eval==0.4.2 2024-03-18T14:19:57,323 Created temporary directory: /tmp/pip-unpack-n4ndz5h4 2024-03-18T14:19:57,559 Downloading lm_eval-0.4.2.tar.gz (698 kB) 2024-03-18T14:20:01,707 Added lm-eval==0.4.2 from https://files.pythonhosted.org/packages/75/ca/18814743ba3b42d19d8524a4f771e9c3a7aa02cd4c579747a0f513907205/lm_eval-0.4.2.tar.gz to build tracker '/tmp/pip-build-tracker-9msy7ctu' 2024-03-18T14:20:01,713 Created temporary directory: /tmp/pip-build-env-zu6uirvz 2024-03-18T14:20:01,719 Installing build dependencies: started 2024-03-18T14:20:01,721 Running command pip subprocess to install build dependencies 2024-03-18T14:20:03,009 Using pip 24.0 from /usr/local/lib/python3.11/dist-packages/pip (python 3.11) 2024-03-18T14:20:03,596 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple 2024-03-18T14:20:05,508 Collecting setuptools>=40.8.0 2024-03-18T14:20:06,250 Using cached https://www.piwheels.org/simple/setuptools/setuptools-69.2.0-py3-none-any.whl (821 kB) 2024-03-18T14:20:06,582 Collecting wheel 2024-03-18T14:20:06,611 Using cached https://www.piwheels.org/simple/wheel/wheel-0.43.0-py3-none-any.whl (65 kB) 2024-03-18T14:20:14,435 Installing collected packages: wheel, setuptools 2024-03-18T14:20:14,703 Creating /tmp/pip-build-env-zu6uirvz/overlay/local/bin 2024-03-18T14:20:14,706 changing mode of /tmp/pip-build-env-zu6uirvz/overlay/local/bin/wheel to 755 2024-03-18T14:20:21,330 Successfully installed setuptools-69.2.0 wheel-0.43.0 2024-03-18T14:20:21,922 Installing build dependencies: finished with status 'done' 2024-03-18T14:20:21,926 Getting requirements to build wheel: started 2024-03-18T14:20:21,927 Running command Getting requirements to build wheel 2024-03-18T14:20:22,897 running egg_info 2024-03-18T14:20:22,901 writing lm_eval.egg-info/PKG-INFO 2024-03-18T14:20:22,922 writing dependency_links to lm_eval.egg-info/dependency_links.txt 2024-03-18T14:20:22,924 writing entry points to lm_eval.egg-info/entry_points.txt 2024-03-18T14:20:22,934 writing requirements to lm_eval.egg-info/requires.txt 2024-03-18T14:20:22,936 writing top-level names to lm_eval.egg-info/top_level.txt 2024-03-18T14:20:23,487 reading manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:23,535 adding license file 'LICENSE.md' 2024-03-18T14:20:23,638 writing manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:23,840 Getting requirements to build wheel: finished with status 'done' 2024-03-18T14:20:23,850 Created temporary directory: /tmp/pip-modern-metadata-_7mkyt94 2024-03-18T14:20:23,853 Preparing metadata (pyproject.toml): started 2024-03-18T14:20:23,854 Running command Preparing metadata (pyproject.toml) 2024-03-18T14:20:24,707 running dist_info 2024-03-18T14:20:24,711 creating /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info 2024-03-18T14:20:24,715 writing /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/PKG-INFO 2024-03-18T14:20:24,735 writing dependency_links to /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/dependency_links.txt 2024-03-18T14:20:24,737 writing entry points to /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/entry_points.txt 2024-03-18T14:20:24,747 writing requirements to /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/requires.txt 2024-03-18T14:20:24,749 writing top-level names to /tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/top_level.txt 2024-03-18T14:20:24,750 writing manifest file '/tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:25,227 reading manifest file '/tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:25,229 adding license file 'LICENSE.md' 2024-03-18T14:20:25,301 writing manifest file '/tmp/pip-modern-metadata-_7mkyt94/lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:25,306 creating '/tmp/pip-modern-metadata-_7mkyt94/lm_eval-0.4.2.dist-info' 2024-03-18T14:20:25,482 Preparing metadata (pyproject.toml): finished with status 'done' 2024-03-18T14:20:25,489 Source in /tmp/pip-wheel-p4s5x_b9/lm-eval_0a1f556720d9405780a230bc45d0c795 has version 0.4.2, which satisfies requirement lm-eval==0.4.2 from https://files.pythonhosted.org/packages/75/ca/18814743ba3b42d19d8524a4f771e9c3a7aa02cd4c579747a0f513907205/lm_eval-0.4.2.tar.gz 2024-03-18T14:20:25,490 Removed lm-eval==0.4.2 from https://files.pythonhosted.org/packages/75/ca/18814743ba3b42d19d8524a4f771e9c3a7aa02cd4c579747a0f513907205/lm_eval-0.4.2.tar.gz from build tracker '/tmp/pip-build-tracker-9msy7ctu' 2024-03-18T14:20:25,500 Created temporary directory: /tmp/pip-unpack-e_2l4l4h 2024-03-18T14:20:25,501 Created temporary directory: /tmp/pip-unpack-1wb7yyow 2024-03-18T14:20:25,654 Building wheels for collected packages: lm-eval 2024-03-18T14:20:25,659 Created temporary directory: /tmp/pip-wheel-r61w3ndf 2024-03-18T14:20:25,660 Destination directory: /tmp/pip-wheel-r61w3ndf 2024-03-18T14:20:25,662 Building wheel for lm-eval (pyproject.toml): started 2024-03-18T14:20:25,664 Running command Building wheel for lm-eval (pyproject.toml) 2024-03-18T14:20:26,489 running bdist_wheel 2024-03-18T14:20:26,505 running build 2024-03-18T14:20:26,506 running build_py 2024-03-18T14:20:26,510 creating build 2024-03-18T14:20:26,511 creating build/lib 2024-03-18T14:20:26,511 creating build/lib/lm_eval 2024-03-18T14:20:26,512 copying lm_eval/evaluator.py -> build/lib/lm_eval 2024-03-18T14:20:26,516 copying lm_eval/__init__.py -> build/lib/lm_eval 2024-03-18T14:20:26,517 copying lm_eval/utils.py -> build/lib/lm_eval 2024-03-18T14:20:26,519 copying lm_eval/evaluator_utils.py -> build/lib/lm_eval 2024-03-18T14:20:26,522 copying lm_eval/__main__.py -> build/lib/lm_eval 2024-03-18T14:20:26,524 copying lm_eval/logging_utils.py -> build/lib/lm_eval 2024-03-18T14:20:26,528 creating build/lib/lm_eval/tasks 2024-03-18T14:20:26,528 copying lm_eval/tasks/__init__.py -> build/lib/lm_eval/tasks 2024-03-18T14:20:26,532 creating build/lib/lm_eval/models 2024-03-18T14:20:26,533 copying lm_eval/models/textsynth.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,535 copying lm_eval/models/optimum_lm.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,537 copying lm_eval/models/__init__.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,539 copying lm_eval/models/utils.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,541 copying lm_eval/models/neuron_optimum.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,544 copying lm_eval/models/anthropic_llms.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,547 copying lm_eval/models/openai_completions.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,550 copying lm_eval/models/mamba_lm.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,552 copying lm_eval/models/huggingface.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,555 copying lm_eval/models/dummy.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,557 copying lm_eval/models/gguf.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,559 copying lm_eval/models/vllm_causallms.py -> build/lib/lm_eval/models 2024-03-18T14:20:26,562 creating build/lib/lm_eval/prompts 2024-03-18T14:20:26,563 copying lm_eval/prompts/__init__.py -> build/lib/lm_eval/prompts 2024-03-18T14:20:26,565 creating build/lib/lm_eval/decontamination 2024-03-18T14:20:26,566 copying lm_eval/decontamination/__init__.py -> build/lib/lm_eval/decontamination 2024-03-18T14:20:26,568 copying lm_eval/decontamination/janitor.py -> build/lib/lm_eval/decontamination 2024-03-18T14:20:26,571 copying lm_eval/decontamination/archiver.py -> build/lib/lm_eval/decontamination 2024-03-18T14:20:26,573 copying lm_eval/decontamination/decontaminate.py -> build/lib/lm_eval/decontamination 2024-03-18T14:20:26,576 creating build/lib/lm_eval/filters 2024-03-18T14:20:26,577 copying lm_eval/filters/selection.py -> build/lib/lm_eval/filters 2024-03-18T14:20:26,578 copying lm_eval/filters/transformation.py -> build/lib/lm_eval/filters 2024-03-18T14:20:26,580 copying lm_eval/filters/decontamination.py -> build/lib/lm_eval/filters 2024-03-18T14:20:26,582 copying lm_eval/filters/__init__.py -> build/lib/lm_eval/filters 2024-03-18T14:20:26,584 copying lm_eval/filters/extraction.py -> build/lib/lm_eval/filters 2024-03-18T14:20:26,587 creating build/lib/lm_eval/api 2024-03-18T14:20:26,588 copying lm_eval/api/samplers.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,590 copying lm_eval/api/metrics.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,593 copying lm_eval/api/__init__.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,595 copying lm_eval/api/model.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,597 copying lm_eval/api/registry.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,600 copying lm_eval/api/filter.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,601 copying lm_eval/api/task.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,605 copying lm_eval/api/instance.py -> build/lib/lm_eval/api 2024-03-18T14:20:26,607 creating build/lib/lm_eval/caching 2024-03-18T14:20:26,608 copying lm_eval/caching/cache.py -> build/lib/lm_eval/caching 2024-03-18T14:20:26,611 creating build/lib/lm_eval/tasks/mathqa 2024-03-18T14:20:26,612 copying lm_eval/tasks/mathqa/utils.py -> build/lib/lm_eval/tasks/mathqa 2024-03-18T14:20:26,615 creating build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:26,616 copying lm_eval/tasks/paws-x/_generate_config.py -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:26,619 creating build/lib/lm_eval/tasks/mutual 2024-03-18T14:20:26,620 copying lm_eval/tasks/mutual/utils.py -> build/lib/lm_eval/tasks/mutual 2024-03-18T14:20:26,623 creating build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:26,624 copying lm_eval/tasks/qasper/metrics.py -> build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:26,626 copying lm_eval/tasks/qasper/utils.py -> build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:26,628 creating build/lib/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:26,629 copying lm_eval/tasks/realtoxicityprompts/metric.py -> build/lib/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:26,632 creating build/lib/lm_eval/tasks/race 2024-03-18T14:20:26,633 copying lm_eval/tasks/race/preprocess_race.py -> build/lib/lm_eval/tasks/race 2024-03-18T14:20:26,636 creating build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:26,638 copying lm_eval/tasks/truthfulqa/utils.py -> build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:26,640 creating build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:26,641 copying lm_eval/tasks/xwinograd/utils.py -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:26,643 creating build/lib/lm_eval/tasks/medqa 2024-03-18T14:20:26,644 copying lm_eval/tasks/medqa/preprocess_medqa.py -> build/lib/lm_eval/tasks/medqa 2024-03-18T14:20:26,647 creating build/lib/lm_eval/tasks/wikitext 2024-03-18T14:20:26,648 copying lm_eval/tasks/wikitext/preprocess_wikitext.py -> build/lib/lm_eval/tasks/wikitext 2024-03-18T14:20:26,652 creating build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:26,653 copying lm_eval/tasks/cmmlu/_generate_configs.py -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:26,656 creating build/lib/lm_eval/tasks/translation 2024-03-18T14:20:26,657 copying lm_eval/tasks/translation/utils.py -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:26,660 creating build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:26,661 copying lm_eval/tasks/ammlu/_generate_configs.py -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:26,664 creating build/lib/lm_eval/tasks/toxigen 2024-03-18T14:20:26,665 copying lm_eval/tasks/toxigen/utils.py -> build/lib/lm_eval/tasks/toxigen 2024-03-18T14:20:26,667 creating build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:26,668 copying lm_eval/tasks/agieval/utils.py -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:26,671 creating build/lib/lm_eval/tasks/wsc273 2024-03-18T14:20:26,672 copying lm_eval/tasks/wsc273/utils.py -> build/lib/lm_eval/tasks/wsc273 2024-03-18T14:20:26,674 creating build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:26,675 copying lm_eval/tasks/xcopa/utils.py -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:26,678 creating build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:26,679 copying lm_eval/tasks/xnli/utils.py -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:26,683 creating build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:26,684 copying lm_eval/tasks/french_bench/preprocess_wikitext.py -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:26,686 copying lm_eval/tasks/french_bench/utils.py -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:26,689 creating build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:26,690 copying lm_eval/tasks/ifeval/utils.py -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:26,692 copying lm_eval/tasks/ifeval/instructions.py -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:26,696 copying lm_eval/tasks/ifeval/instructions_util.py -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:26,698 copying lm_eval/tasks/ifeval/instructions_registry.py -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:26,702 creating build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:26,703 copying lm_eval/tasks/qa4mre/preprocess_qa4mre.py -> build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:26,706 creating build/lib/lm_eval/tasks/mgsm 2024-03-18T14:20:26,707 copying lm_eval/tasks/mgsm/utils.py -> build/lib/lm_eval/tasks/mgsm 2024-03-18T14:20:26,711 creating build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:26,712 copying lm_eval/tasks/bigbench/generate_tasks.py -> build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:26,714 copying lm_eval/tasks/bigbench/push_bigbench_dataset.py -> build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:26,717 creating build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:26,718 copying lm_eval/tasks/blimp/generate_configs.py -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:26,721 creating build/lib/lm_eval/tasks/pubmedqa 2024-03-18T14:20:26,722 copying lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py -> build/lib/lm_eval/tasks/pubmedqa 2024-03-18T14:20:26,724 creating build/lib/lm_eval/tasks/bbh 2024-03-18T14:20:26,725 copying lm_eval/tasks/bbh/_generate_configs.py -> build/lib/lm_eval/tasks/bbh 2024-03-18T14:20:26,727 creating build/lib/lm_eval/tasks/winogrande 2024-03-18T14:20:26,728 copying lm_eval/tasks/winogrande/preprocess_winogrande.py -> build/lib/lm_eval/tasks/winogrande 2024-03-18T14:20:26,732 creating build/lib/lm_eval/tasks/scrolls 2024-03-18T14:20:26,733 copying lm_eval/tasks/scrolls/task.py -> build/lib/lm_eval/tasks/scrolls 2024-03-18T14:20:26,736 creating build/lib/lm_eval/tasks/webqs 2024-03-18T14:20:26,737 copying lm_eval/tasks/webqs/utils.py -> build/lib/lm_eval/tasks/webqs 2024-03-18T14:20:26,740 creating build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:26,741 copying lm_eval/tasks/kobest/utils.py -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:26,743 creating build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:26,744 copying lm_eval/tasks/hendrycks_ethics/utils.py -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:26,747 creating build/lib/lm_eval/tasks/logiqa2 2024-03-18T14:20:26,748 copying lm_eval/tasks/logiqa2/utils_logiqa2.py -> build/lib/lm_eval/tasks/logiqa2 2024-03-18T14:20:26,751 creating build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:26,752 copying lm_eval/tasks/ceval/_generate_configs.py -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:26,755 creating build/lib/lm_eval/tasks/wmt2016 2024-03-18T14:20:26,756 copying lm_eval/tasks/wmt2016/metrics.py -> build/lib/lm_eval/tasks/wmt2016 2024-03-18T14:20:26,759 creating build/lib/lm_eval/tasks/squadv2 2024-03-18T14:20:26,760 copying lm_eval/tasks/squadv2/task.py -> build/lib/lm_eval/tasks/squadv2 2024-03-18T14:20:26,763 creating build/lib/lm_eval/tasks/eq_bench 2024-03-18T14:20:26,764 copying lm_eval/tasks/eq_bench/utils.py -> build/lib/lm_eval/tasks/eq_bench 2024-03-18T14:20:26,766 creating build/lib/lm_eval/tasks/coqa 2024-03-18T14:20:26,767 copying lm_eval/tasks/coqa/utils.py -> build/lib/lm_eval/tasks/coqa 2024-03-18T14:20:26,769 creating build/lib/lm_eval/tasks/medmcqa 2024-03-18T14:20:26,770 copying lm_eval/tasks/medmcqa/utils_medmcqa.py -> build/lib/lm_eval/tasks/medmcqa 2024-03-18T14:20:26,773 creating build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:26,774 copying lm_eval/tasks/minerva_math/utils.py -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:26,777 creating build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:26,778 copying lm_eval/tasks/crows_pairs/utils.py -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:26,780 creating build/lib/lm_eval/tasks/mmlu 2024-03-18T14:20:26,781 copying lm_eval/tasks/mmlu/_generate_configs.py -> build/lib/lm_eval/tasks/mmlu 2024-03-18T14:20:26,784 creating build/lib/lm_eval/tasks/logiqa 2024-03-18T14:20:26,785 copying lm_eval/tasks/logiqa/utils_logiqa.py -> build/lib/lm_eval/tasks/logiqa 2024-03-18T14:20:26,788 creating build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:26,789 copying lm_eval/tasks/belebele/_generate_configs.py -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:26,792 creating build/lib/lm_eval/tasks/drop 2024-03-18T14:20:26,793 copying lm_eval/tasks/drop/utils.py -> build/lib/lm_eval/tasks/drop 2024-03-18T14:20:26,796 creating build/lib/lm_eval/tasks/hellaswag 2024-03-18T14:20:26,797 copying lm_eval/tasks/hellaswag/utils.py -> build/lib/lm_eval/tasks/hellaswag 2024-03-18T14:20:26,799 creating build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:26,800 copying lm_eval/tasks/csatqa/_generate_configs.py -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:26,802 copying lm_eval/tasks/csatqa/utils.py -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:26,807 creating build/lib/lm_eval/tasks/super_glue 2024-03-18T14:20:26,807 creating build/lib/lm_eval/tasks/super_glue/record 2024-03-18T14:20:26,809 copying lm_eval/tasks/super_glue/record/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/record 2024-03-18T14:20:26,811 copying lm_eval/tasks/super_glue/record/util.py -> build/lib/lm_eval/tasks/super_glue/record 2024-03-18T14:20:26,813 creating build/lib/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:26,814 copying lm_eval/tasks/super_glue/copa/utils.py -> build/lib/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:26,817 creating build/lib/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:26,818 copying lm_eval/tasks/super_glue/cb/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:26,820 copying lm_eval/tasks/super_glue/cb/aggregate.py -> build/lib/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:26,822 creating build/lib/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:26,823 copying lm_eval/tasks/super_glue/wsc/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:26,825 copying lm_eval/tasks/super_glue/wsc/preprocess_wsc.py -> build/lib/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:26,828 creating build/lib/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:26,828 copying lm_eval/tasks/super_glue/multirc/t5_utils.py -> build/lib/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:26,831 creating build/lib/lm_eval/tasks/okapi 2024-03-18T14:20:26,832 creating build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:26,833 copying lm_eval/tasks/okapi/truthfulqa_multilingual/utils.py -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:26,836 creating build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:26,836 copying lm_eval/tasks/okapi/hellaswag_multilingual/utils.py -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:26,839 creating build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:26,840 copying lm_eval/tasks/okapi/mmlu_multilingual/_generate_configs.py -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:26,843 creating build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:26,844 copying lm_eval/tasks/okapi/arc_multilingual/utils.py -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:26,847 creating build/lib/lm_eval/tasks/model_written_evals 2024-03-18T14:20:26,847 creating build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:26,848 copying lm_eval/tasks/model_written_evals/persona/_generate_configs.py -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:26,852 creating build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:26,853 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:26,858 creating build/lib/lm_eval/tasks/glue 2024-03-18T14:20:26,858 creating build/lib/lm_eval/tasks/glue/mnli 2024-03-18T14:20:26,859 copying lm_eval/tasks/glue/mnli/utils.py -> build/lib/lm_eval/tasks/glue/mnli 2024-03-18T14:20:26,865 creating build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:26,866 copying lm_eval/tasks/bbh/cot_zeroshot/utils.py -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:26,869 creating build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:26,870 copying lm_eval/tasks/bbh/zeroshot/utils.py -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:26,874 creating build/lib/lm_eval/tasks/gpqa 2024-03-18T14:20:26,875 creating build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:26,876 copying lm_eval/tasks/gpqa/cot_zeroshot/_generate_configs.py -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:26,878 copying lm_eval/tasks/gpqa/cot_zeroshot/utils.py -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:26,880 creating build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:26,881 copying lm_eval/tasks/gpqa/generative/_generate_configs.py -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:26,883 copying lm_eval/tasks/gpqa/generative/utils.py -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:26,885 creating build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:26,886 copying lm_eval/tasks/gpqa/n_shot/_generate_configs.py -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:26,888 copying lm_eval/tasks/gpqa/n_shot/utils.py -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:26,891 creating build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:26,892 copying lm_eval/tasks/gpqa/zeroshot/_generate_configs.py -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:26,894 copying lm_eval/tasks/gpqa/zeroshot/utils.py -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:26,896 creating build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:26,897 copying lm_eval/tasks/gpqa/cot_n_shot/_generate_configs.py -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:26,899 copying lm_eval/tasks/gpqa/cot_n_shot/utils.py -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:26,903 creating build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:26,903 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/utils.py -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:26,907 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot 2024-03-18T14:20:26,908 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:26,909 copying lm_eval/tasks/mmlu/flan_n_shot/generative/utils.py -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:26,912 creating build/lib/lm_eval/tasks/code_x_glue 2024-03-18T14:20:26,912 creating build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:26,913 copying lm_eval/tasks/code_x_glue/code-text/utils.py -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:26,915 copying lm_eval/tasks/code_x_glue/code-text/bleu.py -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:26,919 running egg_info 2024-03-18T14:20:26,922 writing lm_eval.egg-info/PKG-INFO 2024-03-18T14:20:26,942 writing dependency_links to lm_eval.egg-info/dependency_links.txt 2024-03-18T14:20:26,943 writing entry points to lm_eval.egg-info/entry_points.txt 2024-03-18T14:20:26,953 writing requirements to lm_eval.egg-info/requires.txt 2024-03-18T14:20:26,955 writing top-level names to lm_eval.egg-info/top_level.txt 2024-03-18T14:20:27,425 reading manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:27,473 adding license file 'LICENSE.md' 2024-03-18T14:20:27,574 writing manifest file 'lm_eval.egg-info/SOURCES.txt' 2024-03-18T14:20:28,056 copying lm_eval/tasks/mathqa/mathqa.yaml -> build/lib/lm_eval/tasks/mathqa 2024-03-18T14:20:28,058 copying lm_eval/tasks/paws-x/paws_fr.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,060 copying lm_eval/tasks/paws-x/paws_zh.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,063 copying lm_eval/tasks/paws-x/paws_ja.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,065 copying lm_eval/tasks/paws-x/paws_de.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,067 copying lm_eval/tasks/paws-x/paws_es.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,070 copying lm_eval/tasks/paws-x/paws_ko.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,072 copying lm_eval/tasks/paws-x/paws_en.yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:28,074 copying lm_eval/tasks/mutual/multual_plus.yaml -> build/lib/lm_eval/tasks/mutual 2024-03-18T14:20:28,076 copying lm_eval/tasks/mutual/mutual.yaml -> build/lib/lm_eval/tasks/mutual 2024-03-18T14:20:28,079 creating build/lib/lm_eval/tasks/swag 2024-03-18T14:20:28,080 copying lm_eval/tasks/swag/swag.yaml -> build/lib/lm_eval/tasks/swag 2024-03-18T14:20:28,082 copying lm_eval/tasks/qasper/freeform.yaml -> build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:28,085 copying lm_eval/tasks/qasper/bool.yaml -> build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:28,087 copying lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml -> build/lib/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:28,091 creating build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:28,092 copying lm_eval/tasks/gsm8k/gsm8k.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:28,095 copying lm_eval/tasks/gsm8k/gsm8k-cot.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:28,097 copying lm_eval/tasks/gsm8k/gsm8k-cot-zeroshot.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:28,099 copying lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml -> build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:28,102 creating build/lib/lm_eval/tasks/kmmlu 2024-03-18T14:20:28,103 creating build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,104 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_maritime_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,106 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_railway_and_automotive_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,108 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_political_science_and_sociology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,111 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_refrigerating_machinery.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,113 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_taxation.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,116 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_health.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,118 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_construction.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,120 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_nondestructive_testing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,122 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_psychology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,124 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_math.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,127 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_education.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,129 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_food_processing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,131 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_social_welfare.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,133 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_ecology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,135 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_marketing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,138 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,140 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_materials_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,142 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_patent.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,144 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_real_estate.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,146 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_telecommunications_and_wireless_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,148 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_fashion.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,151 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_aviation_engineering_and_maintenance.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,153 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_information_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,155 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_environmental_science.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,158 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemistry.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,160 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_computer_science.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,162 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_economics.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,165 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_machine_design_and_manufacturing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,167 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_energy_management.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,170 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_biology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,172 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_electrical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,174 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_management.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,176 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_korean_history.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,178 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_public_safety.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,180 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_industrial_engineer.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,182 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_mechanical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,185 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_interior_architecture_and_design.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,187 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_accounting.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,189 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_criminal_law.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,191 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_agricultural_sciences.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,193 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_electronics_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,195 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_law.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,197 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_geomatics.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,199 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_civil_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,202 copying lm_eval/tasks/kmmlu/direct/kmmlu_direct_gas_technology_and_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:28,204 creating build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,206 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_education.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,208 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_management.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,210 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_social_welfare.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,212 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_taxation.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,214 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_fashion.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,216 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_geomatics.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,219 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_mechanical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,221 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_civil_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,223 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,225 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_telecommunications_and_wireless_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,227 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_public_safety.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,230 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_criminal_law.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,232 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_materials_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,234 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_food_processing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,236 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_health.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,238 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_real_estate.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,240 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_maritime_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,242 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electrical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,244 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_law.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,246 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_korean_history.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,248 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_nondestructive_testing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,250 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_gas_technology_and_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,252 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_math.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,255 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_computer_science.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,257 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_energy_management.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,260 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_economics.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,262 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_ecology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,264 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemistry.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,266 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_psychology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,268 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electronics_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,270 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_construction.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,272 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_refrigerating_machinery.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,275 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_patent.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,277 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_information_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,279 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_environmental_science.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,281 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_interior_architecture_and_design.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,283 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_railway_and_automotive_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,285 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_aviation_engineering_and_maintenance.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,287 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_industrial_engineer.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,290 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_machine_design_and_manufacturing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,292 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_political_science_and_sociology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,294 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_accounting.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,296 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_biology.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,299 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_agricultural_sciences.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,301 copying lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_marketing.yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:28,303 creating build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,304 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_materials_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,307 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_education.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,310 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_patent.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,313 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_construction.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,316 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_marketing.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,319 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_public_safety.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,322 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_food_processing.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,325 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_social_welfare.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,327 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_computer_science.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,330 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_psychology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,333 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_mechanical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,336 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,339 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_information_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,342 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_geomatics.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,345 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_refrigerating_machinery.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,348 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_economics.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,350 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_aviation_engineering_and_maintenance.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,353 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_korean_history.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,356 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemistry.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,359 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_maritime_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,362 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_energy_management.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,365 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_math.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,367 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_biology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,370 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_machine_design_and_manufacturing.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,374 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electrical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,376 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_criminal_law.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,379 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_nondestructive_testing.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,383 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_gas_technology_and_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,386 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_telecommunications_and_wireless_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,389 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_agricultural_sciences.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,392 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_health.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,395 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_fashion.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,398 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_political_science_and_sociology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,401 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_environmental_science.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,404 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_ecology.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,407 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_taxation.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,410 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_management.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,414 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_railway_and_automotive_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,417 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_real_estate.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,420 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_law.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,423 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_interior_architecture_and_design.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,426 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_accounting.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,429 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_civil_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,432 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_industrial_engineer.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,436 copying lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electronics_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:28,438 creating build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,440 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_ecology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,442 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_electrical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,445 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_psychology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,447 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_geomatics.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,449 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,451 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_health.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,454 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_law.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,456 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_fashion.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,458 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_education.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,460 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemistry.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,462 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_energy_management.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,464 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_real_estate.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,466 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_information_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,468 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_social_welfare.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,470 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_electronics_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,472 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_railway_and_automotive_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,474 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_food_processing.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,477 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_economics.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,479 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_marketing.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,482 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_civil_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,484 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_computer_science.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,487 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_refrigerating_machinery.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,490 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_criminal_law.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,492 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_construction.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,494 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_political_science_and_sociology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,496 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_interior_architecture_and_design.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,499 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_machine_design_and_manufacturing.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,501 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_biology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,504 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_accounting.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,506 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_agricultural_sciences.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,508 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_environmental_science.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,511 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_maritime_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,513 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_gas_technology_and_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,515 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_patent.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,517 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_management.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,519 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_aviation_engineering_and_maintenance.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,522 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_mechanical_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,524 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_telecommunications_and_wireless_technology.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,526 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_industrial_engineer.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,528 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_public_safety.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,531 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_korean_history.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,533 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_math.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,535 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_nondestructive_testing.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,537 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_materials_engineering.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,539 copying lm_eval/tasks/kmmlu/hard/kmmlu_hard_taxation.yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:28,541 copying lm_eval/tasks/race/race.yaml -> build/lib/lm_eval/tasks/race 2024-03-18T14:20:28,543 creating build/lib/lm_eval/tasks/nq_open 2024-03-18T14:20:28,544 copying lm_eval/tasks/nq_open/nq_open.yaml -> build/lib/lm_eval/tasks/nq_open 2024-03-18T14:20:28,547 copying lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:28,549 copying lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:28,551 copying lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml -> build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:28,553 copying lm_eval/tasks/xwinograd/xwinograd_en.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,555 copying lm_eval/tasks/xwinograd/xwinograd_zh.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,558 copying lm_eval/tasks/xwinograd/xwinograd_ru.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,560 copying lm_eval/tasks/xwinograd/xwinograd_fr.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,562 copying lm_eval/tasks/xwinograd/xwinograd_jp.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,565 copying lm_eval/tasks/xwinograd/xwinograd_pt.yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:28,568 copying lm_eval/tasks/medqa/medqa.yaml -> build/lib/lm_eval/tasks/medqa 2024-03-18T14:20:28,572 copying lm_eval/tasks/wikitext/wikitext.yaml -> build/lib/lm_eval/tasks/wikitext 2024-03-18T14:20:28,575 copying lm_eval/tasks/super_glue/record/default.yaml -> build/lib/lm_eval/tasks/super_glue/record 2024-03-18T14:20:28,578 copying lm_eval/tasks/super_glue/record/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/record 2024-03-18T14:20:28,580 copying lm_eval/tasks/super_glue/copa/default.yaml -> build/lib/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:28,583 copying lm_eval/tasks/super_glue/copa/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:28,585 creating build/lib/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:28,587 copying lm_eval/tasks/super_glue/rte/default.yaml -> build/lib/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:28,590 copying lm_eval/tasks/super_glue/rte/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:28,592 copying lm_eval/tasks/super_glue/cb/default.yaml -> build/lib/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:28,595 copying lm_eval/tasks/super_glue/cb/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:28,599 copying lm_eval/tasks/super_glue/wsc/default.yaml -> build/lib/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:28,601 copying lm_eval/tasks/super_glue/wsc/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:28,604 creating build/lib/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:28,605 copying lm_eval/tasks/super_glue/boolq/default.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:28,609 copying lm_eval/tasks/super_glue/boolq/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:28,612 copying lm_eval/tasks/super_glue/boolq/seq2seq.yaml -> build/lib/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:28,615 copying lm_eval/tasks/super_glue/multirc/default.yaml -> build/lib/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:28,618 copying lm_eval/tasks/super_glue/multirc/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:28,621 creating build/lib/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:28,622 copying lm_eval/tasks/super_glue/wic/default.yaml -> build/lib/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:28,625 copying lm_eval/tasks/super_glue/wic/t5-prompt.yaml -> build/lib/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:28,628 creating build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:28,629 copying lm_eval/tasks/wmdp/wmdp_cyber.yaml -> build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:28,633 copying lm_eval/tasks/wmdp/wmdp_bio.yaml -> build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:28,636 copying lm_eval/tasks/wmdp/wmdp_chem.yaml -> build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:28,638 creating build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,640 copying lm_eval/tasks/haerae/haerae_gk.yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,643 copying lm_eval/tasks/haerae/haerae_lw.yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,646 copying lm_eval/tasks/haerae/haerae_sn.yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,648 copying lm_eval/tasks/haerae/haerae_rw.yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,651 copying lm_eval/tasks/haerae/haerae_hi.yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:28,654 copying lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,657 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,660 copying lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,664 copying lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,666 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,669 copying lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,672 copying lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,674 copying lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,677 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,681 copying lm_eval/tasks/cmmlu/cmmlu_default_education.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,684 copying lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,688 copying lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,691 copying lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,693 copying lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,696 copying lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,699 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,702 copying lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,704 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,708 copying lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,711 copying lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,714 copying lm_eval/tasks/cmmlu/cmmlu_default_management.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,716 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,719 copying lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,722 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,724 copying lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,727 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,730 copying lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,734 copying lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,736 copying lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,739 copying lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,742 copying lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,745 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,748 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,751 copying lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,754 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,757 copying lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,759 copying lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,762 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,766 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,769 copying lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,771 copying lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,775 copying lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,777 copying lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,780 copying lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,783 copying lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,786 copying lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,788 copying lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,791 copying lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,794 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,797 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,800 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,803 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,806 copying lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,809 copying lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,811 copying lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,815 copying lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,818 copying lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,820 copying lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,823 copying lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,826 copying lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,828 copying lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,831 copying lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,834 copying lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,837 copying lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,839 copying lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,842 copying lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,845 copying lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:28,848 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,851 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,854 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,856 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,859 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,862 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,865 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,867 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,871 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,873 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,876 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,879 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,882 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,885 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,887 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,890 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,893 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,895 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,898 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,901 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,904 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,907 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,910 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,912 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,915 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,917 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,920 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,923 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,925 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,928 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,931 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,933 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,936 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,939 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,942 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,945 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,947 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,950 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,953 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,956 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,959 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,962 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,964 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,967 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,970 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,972 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,975 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,977 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,980 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,983 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,986 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,988 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,991 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,994 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:28,997 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,000 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,003 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,006 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc2.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,009 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,012 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,015 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,017 copying lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc1.yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:29,021 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ta.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,024 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hi.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,027 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_vi.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,030 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hu.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,033 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_kn.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,036 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_eu.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,040 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ca.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,043 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_gu.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,045 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_pt.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,049 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_id.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,052 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_nl.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,055 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ar.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,057 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_uk.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,061 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ru.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,063 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_mr.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,066 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ml.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,068 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sr.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,071 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hr.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,073 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ne.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,075 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_bn.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,078 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_de.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,080 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ro.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,083 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_te.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,085 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sv.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,087 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hy.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,090 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_da.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,092 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_fr.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,095 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_it.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,097 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sk.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,099 copying lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_es.yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:29,102 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ml.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,105 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ru.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,107 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hu.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,110 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ca.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,112 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hi.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,115 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_is.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,117 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_pt.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,119 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_te.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,121 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ro.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,123 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ar.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,126 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_uk.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,128 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sk.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,130 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_gu.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,132 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_it.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,134 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nl.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,136 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_de.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,139 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_fr.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,141 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_id.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,143 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_kn.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,146 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ne.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,148 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ta.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,150 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hy.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,153 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_bn.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,155 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sv.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,158 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_en.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,161 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hr.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,164 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_es.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,167 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_zh.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,170 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sr.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,173 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_da.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,175 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_mr.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,178 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nb.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,181 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_eu.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,183 copying lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_vi.yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:29,186 copying lm_eval/tasks/okapi/arc_multilingual/arc_kn.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,188 copying lm_eval/tasks/okapi/arc_multilingual/arc_ca.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,191 copying lm_eval/tasks/okapi/arc_multilingual/arc_ml.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,194 copying lm_eval/tasks/okapi/arc_multilingual/arc_uk.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,196 copying lm_eval/tasks/okapi/arc_multilingual/arc_ar.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,199 copying lm_eval/tasks/okapi/arc_multilingual/arc_nl.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,202 copying lm_eval/tasks/okapi/arc_multilingual/arc_hu.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,204 copying lm_eval/tasks/okapi/arc_multilingual/arc_bn.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,207 copying lm_eval/tasks/okapi/arc_multilingual/arc_da.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,209 copying lm_eval/tasks/okapi/arc_multilingual/arc_de.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,212 copying lm_eval/tasks/okapi/arc_multilingual/arc_ne.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,215 copying lm_eval/tasks/okapi/arc_multilingual/arc_hi.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,218 copying lm_eval/tasks/okapi/arc_multilingual/arc_vi.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,220 copying lm_eval/tasks/okapi/arc_multilingual/arc_ta.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,223 copying lm_eval/tasks/okapi/arc_multilingual/arc_id.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,225 copying lm_eval/tasks/okapi/arc_multilingual/arc_ro.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,228 copying lm_eval/tasks/okapi/arc_multilingual/arc_ru.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,231 copying lm_eval/tasks/okapi/arc_multilingual/arc_sr.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,233 copying lm_eval/tasks/okapi/arc_multilingual/arc_it.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,235 copying lm_eval/tasks/okapi/arc_multilingual/arc_es.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,239 copying lm_eval/tasks/okapi/arc_multilingual/arc_sv.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,241 copying lm_eval/tasks/okapi/arc_multilingual/arc_zh.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,243 copying lm_eval/tasks/okapi/arc_multilingual/arc_sk.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,246 copying lm_eval/tasks/okapi/arc_multilingual/arc_te.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,248 copying lm_eval/tasks/okapi/arc_multilingual/arc_fr.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,251 copying lm_eval/tasks/okapi/arc_multilingual/arc_hy.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,253 copying lm_eval/tasks/okapi/arc_multilingual/arc_gu.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,255 copying lm_eval/tasks/okapi/arc_multilingual/arc_pt.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,258 copying lm_eval/tasks/okapi/arc_multilingual/arc_mr.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,260 copying lm_eval/tasks/okapi/arc_multilingual/arc_eu.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,263 copying lm_eval/tasks/okapi/arc_multilingual/arc_hr.yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:29,266 copying lm_eval/tasks/translation/wmt16_en-de.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,268 copying lm_eval/tasks/translation/wmt14_en-fr.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,270 copying lm_eval/tasks/translation/wmt14_fr-en.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,272 copying lm_eval/tasks/translation/wmt16_ro-en.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,274 copying lm_eval/tasks/translation/wmt16_en-ro.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,277 copying lm_eval/tasks/translation/iwslt2017_ar-en.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,279 copying lm_eval/tasks/translation/iwslt2017_en-ar.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,281 copying lm_eval/tasks/translation/wmt16_de-en.yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:29,283 copying lm_eval/tasks/ammlu/ammlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,285 copying lm_eval/tasks/ammlu/ammlu_public_relations.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,287 copying lm_eval/tasks/ammlu/ammlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,289 copying lm_eval/tasks/ammlu/ammlu_formal_logic.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,292 copying lm_eval/tasks/ammlu/ammlu_world_religions.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,294 copying lm_eval/tasks/ammlu/ammlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,296 copying lm_eval/tasks/ammlu/ammlu_anatomy.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,298 copying lm_eval/tasks/ammlu/ammlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,301 copying lm_eval/tasks/ammlu/ammlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,303 copying lm_eval/tasks/ammlu/ammlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,306 copying lm_eval/tasks/ammlu/ammlu_management.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,308 copying lm_eval/tasks/ammlu/ammlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,310 copying lm_eval/tasks/ammlu/ammlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,312 copying lm_eval/tasks/ammlu/ammlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,315 copying lm_eval/tasks/ammlu/ammlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,317 copying lm_eval/tasks/ammlu/ammlu_human_aging.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,319 copying lm_eval/tasks/ammlu/ammlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,322 copying lm_eval/tasks/ammlu/ammlu_philosophy.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,324 copying lm_eval/tasks/ammlu/ammlu_econometrics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,326 copying lm_eval/tasks/ammlu/ammlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,328 copying lm_eval/tasks/ammlu/ammlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,331 copying lm_eval/tasks/ammlu/ammlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,333 copying lm_eval/tasks/ammlu/ammlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,336 copying lm_eval/tasks/ammlu/ammlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,338 copying lm_eval/tasks/ammlu/ammlu_nutrition.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,341 copying lm_eval/tasks/ammlu/ammlu_marketing.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,343 copying lm_eval/tasks/ammlu/ammlu_college_medicine.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,345 copying lm_eval/tasks/ammlu/ammlu_college_biology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,347 copying lm_eval/tasks/ammlu/ammlu_machine_learning.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,349 copying lm_eval/tasks/ammlu/ammlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,352 copying lm_eval/tasks/ammlu/ammlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,354 copying lm_eval/tasks/ammlu/ammlu_college_physics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,356 copying lm_eval/tasks/ammlu/ammlu_professional_law.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,358 copying lm_eval/tasks/ammlu/ammlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,360 copying lm_eval/tasks/ammlu/ammlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,362 copying lm_eval/tasks/ammlu/ammlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,364 copying lm_eval/tasks/ammlu/ammlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,366 copying lm_eval/tasks/ammlu/ammlu_business_ethics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,368 copying lm_eval/tasks/ammlu/ammlu_global_facts.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,370 copying lm_eval/tasks/ammlu/ammlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,373 copying lm_eval/tasks/ammlu/ammlu_sociology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,375 copying lm_eval/tasks/ammlu/ammlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,377 copying lm_eval/tasks/ammlu/ammlu_prehistory.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,380 copying lm_eval/tasks/ammlu/ammlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,382 copying lm_eval/tasks/ammlu/ammlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,384 copying lm_eval/tasks/ammlu/ammlu_computer_security.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,387 copying lm_eval/tasks/ammlu/ammlu_astronomy.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,389 copying lm_eval/tasks/ammlu/ammlu_security_studies.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,391 copying lm_eval/tasks/ammlu/ammlu_international_law.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,394 copying lm_eval/tasks/ammlu/ammlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,396 copying lm_eval/tasks/ammlu/ammlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,399 copying lm_eval/tasks/ammlu/ammlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,401 copying lm_eval/tasks/ammlu/ammlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,403 copying lm_eval/tasks/ammlu/ammlu_virology.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,405 copying lm_eval/tasks/ammlu/ammlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,407 copying lm_eval/tasks/ammlu/ammlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,410 copying lm_eval/tasks/ammlu/ammlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:29,412 creating build/lib/lm_eval/tasks/storycloze 2024-03-18T14:20:29,413 copying lm_eval/tasks/storycloze/storycloze_2016.yaml -> build/lib/lm_eval/tasks/storycloze 2024-03-18T14:20:29,415 copying lm_eval/tasks/storycloze/storycloze_2018.yaml -> build/lib/lm_eval/tasks/storycloze 2024-03-18T14:20:29,418 creating build/lib/lm_eval/tasks/fld 2024-03-18T14:20:29,419 copying lm_eval/tasks/fld/fld_star.yaml -> build/lib/lm_eval/tasks/fld 2024-03-18T14:20:29,421 copying lm_eval/tasks/fld/fld_default.yaml -> build/lib/lm_eval/tasks/fld 2024-03-18T14:20:29,423 copying lm_eval/tasks/toxigen/toxigen.yaml -> build/lib/lm_eval/tasks/toxigen 2024-03-18T14:20:29,425 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,427 copying lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,428 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,431 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,432 copying lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,435 copying lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,436 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,439 copying lm_eval/tasks/model_written_evals/persona/has-disability.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,441 copying lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,443 copying lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,446 copying lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,448 copying lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,450 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,452 copying lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,455 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,457 copying lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,459 copying lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,461 copying lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,463 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,465 copying lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,467 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,470 copying lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,472 copying lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,474 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,476 copying lm_eval/tasks/model_written_evals/persona/agreeableness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,478 copying lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,480 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,482 copying lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,484 copying lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,486 copying lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,488 copying lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,490 copying lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,492 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,495 copying lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,497 copying lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,499 copying lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,502 copying lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,504 copying lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,506 copying lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,508 copying lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,510 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,512 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,514 copying lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,516 copying lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,518 copying lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,520 copying lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,522 copying lm_eval/tasks/model_written_evals/persona/psychopathy.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,525 copying lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,527 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,529 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,531 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,533 copying lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,535 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,537 copying lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,539 copying lm_eval/tasks/model_written_evals/persona/self-replication.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,541 copying lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,543 copying lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,545 copying lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,548 copying lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,551 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,553 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,555 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,558 copying lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,560 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,563 copying lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,567 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,570 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,572 copying lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,575 copying lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,577 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,580 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,583 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,586 copying lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,589 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,592 copying lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,595 copying lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,598 copying lm_eval/tasks/model_written_evals/persona/openness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,601 copying lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,603 copying lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,606 copying lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,609 copying lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,616 copying lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,622 copying lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,627 copying lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,629 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,631 copying lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,634 copying lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,636 copying lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,639 copying lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,641 copying lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,644 copying lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,646 copying lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,648 copying lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,650 copying lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,653 copying lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,655 copying lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,657 copying lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,659 copying lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,661 copying lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,663 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,667 copying lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,670 copying lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,673 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,676 copying lm_eval/tasks/model_written_evals/persona/narcissism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,678 copying lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,681 copying lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,683 copying lm_eval/tasks/model_written_evals/persona/risk-averse.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,686 copying lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,688 copying lm_eval/tasks/model_written_evals/persona/neuroticism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,691 copying lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,693 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,695 copying lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,697 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,700 copying lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,702 copying lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,704 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,706 copying lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,709 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,711 copying lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,714 copying lm_eval/tasks/model_written_evals/persona/extraversion.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,716 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,718 copying lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,720 copying lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,723 copying lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,726 copying lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,728 copying lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,731 copying lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,734 copying lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,736 copying lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,739 copying lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,741 copying lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,744 copying lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,746 copying lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,748 copying lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,750 copying lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:29,752 creating build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:29,753 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:29,756 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:29,758 copying lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml -> build/lib/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:29,760 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,762 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,765 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,767 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,769 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,772 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,774 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,778 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,780 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,784 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,786 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,789 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,791 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,794 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,797 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,800 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,803 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,805 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,808 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,810 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,813 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,815 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,818 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,820 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,823 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,825 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,828 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,831 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,834 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,836 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,839 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,841 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,844 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,847 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,849 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,852 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,855 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,858 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,860 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,863 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,865 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,868 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,871 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,873 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,876 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,878 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,880 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,882 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,884 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:29,886 copying lm_eval/tasks/agieval/gaokao-biology.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,888 copying lm_eval/tasks/agieval/logiqa-en.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,890 copying lm_eval/tasks/agieval/logiqa-zh.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,892 copying lm_eval/tasks/agieval/sat-en-without-passage.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,894 copying lm_eval/tasks/agieval/gaokao-mathcloze.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,896 copying lm_eval/tasks/agieval/gaokao-physics.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,898 copying lm_eval/tasks/agieval/jec-qa-kd.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,900 copying lm_eval/tasks/agieval/aqua-rat.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,902 copying lm_eval/tasks/agieval/lsat-lr.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,904 copying lm_eval/tasks/agieval/lsat-rc.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,906 copying lm_eval/tasks/agieval/gaokao-english.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,908 copying lm_eval/tasks/agieval/math.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,910 copying lm_eval/tasks/agieval/sat-math.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,912 copying lm_eval/tasks/agieval/gaokao-mathqa.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,914 copying lm_eval/tasks/agieval/gaokao-geography.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,917 copying lm_eval/tasks/agieval/gaokao-chemistry.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,919 copying lm_eval/tasks/agieval/lsat-ar.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,921 copying lm_eval/tasks/agieval/jec-qa-ca.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,924 copying lm_eval/tasks/agieval/sat-en.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,926 copying lm_eval/tasks/agieval/gaokao-chinese.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,928 copying lm_eval/tasks/agieval/gaokao-history.yaml -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:29,930 copying lm_eval/tasks/wsc273/default.yaml -> build/lib/lm_eval/tasks/wsc273 2024-03-18T14:20:29,932 creating build/lib/lm_eval/tasks/arc 2024-03-18T14:20:29,933 copying lm_eval/tasks/arc/arc_challenge.yaml -> build/lib/lm_eval/tasks/arc 2024-03-18T14:20:29,935 copying lm_eval/tasks/arc/arc_easy.yaml -> build/lib/lm_eval/tasks/arc 2024-03-18T14:20:29,937 copying lm_eval/tasks/xcopa/default_ht.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,939 copying lm_eval/tasks/xcopa/default_et.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,941 copying lm_eval/tasks/xcopa/default_vi.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,943 copying lm_eval/tasks/xcopa/default_sw.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,946 copying lm_eval/tasks/xcopa/default_ta.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,948 copying lm_eval/tasks/xcopa/default_tr.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,950 copying lm_eval/tasks/xcopa/default_qu.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,952 copying lm_eval/tasks/xcopa/default_id.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,954 copying lm_eval/tasks/xcopa/default_it.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,957 copying lm_eval/tasks/xcopa/default_zh.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,960 copying lm_eval/tasks/xcopa/default_th.yaml -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:29,962 creating build/lib/lm_eval/tasks/triviaqa 2024-03-18T14:20:29,963 copying lm_eval/tasks/triviaqa/default.yaml -> build/lib/lm_eval/tasks/triviaqa 2024-03-18T14:20:29,965 creating build/lib/lm_eval/tasks/benchmarks 2024-03-18T14:20:29,966 copying lm_eval/tasks/benchmarks/pythia.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-03-18T14:20:29,968 copying lm_eval/tasks/benchmarks/t0_eval.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-03-18T14:20:29,970 copying lm_eval/tasks/benchmarks/openllm.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-03-18T14:20:29,972 copying lm_eval/tasks/benchmarks/minerva_math.yaml -> build/lib/lm_eval/tasks/benchmarks 2024-03-18T14:20:29,974 creating build/lib/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:29,975 copying lm_eval/tasks/benchmarks/flan/flan_held_in.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:29,978 copying lm_eval/tasks/benchmarks/flan/flan_held_out.yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:29,980 creating build/lib/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:29,981 copying lm_eval/tasks/benchmarks/multimedqa/multimedqa.yaml -> build/lib/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:29,983 copying lm_eval/tasks/xnli/xnli_vi.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,986 copying lm_eval/tasks/xnli/xnli_es.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,988 copying lm_eval/tasks/xnli/xnli_ar.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,990 copying lm_eval/tasks/xnli/xnli_de.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,992 copying lm_eval/tasks/xnli/xnli_zh.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,994 copying lm_eval/tasks/xnli/xnli_tr.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,996 copying lm_eval/tasks/xnli/xnli_hi.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:29,999 copying lm_eval/tasks/xnli/xnli_ru.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,001 copying lm_eval/tasks/xnli/xnli_en.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,003 copying lm_eval/tasks/xnli/xnli_ur.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,005 copying lm_eval/tasks/xnli/xnli_bg.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,007 copying lm_eval/tasks/xnli/xnli_sw.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,009 copying lm_eval/tasks/xnli/xnli_fr.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,011 copying lm_eval/tasks/xnli/xnli_el.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,014 copying lm_eval/tasks/xnli/xnli_th.yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:30,016 creating build/lib/lm_eval/tasks/piqa 2024-03-18T14:20:30,017 copying lm_eval/tasks/piqa/piqa.yaml -> build/lib/lm_eval/tasks/piqa 2024-03-18T14:20:30,019 copying lm_eval/tasks/french_bench/french_bench_vocab.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,022 copying lm_eval/tasks/french_bench/french_bench_boolqa.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,024 copying lm_eval/tasks/french_bench/french_bench_multifquad.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,026 copying lm_eval/tasks/french_bench/french_bench_wikitext_fr.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,028 copying lm_eval/tasks/french_bench/french_bench_fquadv2_bool.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,031 copying lm_eval/tasks/french_bench/french_bench_xnli.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,033 copying lm_eval/tasks/french_bench/french_bench_reading_comp.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,035 copying lm_eval/tasks/french_bench/french_bench_trivia.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,037 copying lm_eval/tasks/french_bench/french_bench_fquadv2_genq.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,039 copying lm_eval/tasks/french_bench/french_bench_hellaswag.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,042 copying lm_eval/tasks/french_bench/french_bench_fquadv2.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,044 copying lm_eval/tasks/french_bench/french_bench_opus_perplexity.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,046 copying lm_eval/tasks/french_bench/french_bench_grammar.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,048 copying lm_eval/tasks/french_bench/french_bench_orangesum_abstract.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,051 copying lm_eval/tasks/french_bench/french_bench_fquadv2_hasAns.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,053 copying lm_eval/tasks/french_bench/french_bench_orangesum_title.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,055 copying lm_eval/tasks/french_bench/french_bench_topic_based_nli.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,057 copying lm_eval/tasks/french_bench/french_bench_arc_challenge.yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:30,059 copying lm_eval/tasks/ifeval/ifeval.yaml -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:30,061 copying lm_eval/tasks/qa4mre/qa4mre_2012.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:30,063 copying lm_eval/tasks/qa4mre/qa4mre_2013.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:30,065 copying lm_eval/tasks/qa4mre/qa4mre_2011.yaml -> build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:30,068 creating build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,069 copying lm_eval/tasks/aexams/aexams_Physics.yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,072 copying lm_eval/tasks/aexams/aexams_IslamicStudies.yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,074 copying lm_eval/tasks/aexams/aexams_Science.yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,076 copying lm_eval/tasks/aexams/aexams_Biology.yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,078 copying lm_eval/tasks/aexams/aexams_Social.yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:30,080 creating build/lib/lm_eval/tasks/glue/qqp 2024-03-18T14:20:30,081 copying lm_eval/tasks/glue/qqp/default.yaml -> build/lib/lm_eval/tasks/glue/qqp 2024-03-18T14:20:30,083 creating build/lib/lm_eval/tasks/glue/mrpc 2024-03-18T14:20:30,084 copying lm_eval/tasks/glue/mrpc/default.yaml -> build/lib/lm_eval/tasks/glue/mrpc 2024-03-18T14:20:30,087 creating build/lib/lm_eval/tasks/glue/rte 2024-03-18T14:20:30,088 copying lm_eval/tasks/glue/rte/default.yaml -> build/lib/lm_eval/tasks/glue/rte 2024-03-18T14:20:30,092 creating build/lib/lm_eval/tasks/glue/wnli 2024-03-18T14:20:30,093 copying lm_eval/tasks/glue/wnli/default.yaml -> build/lib/lm_eval/tasks/glue/wnli 2024-03-18T14:20:30,096 creating build/lib/lm_eval/tasks/glue/sst2 2024-03-18T14:20:30,097 copying lm_eval/tasks/glue/sst2/default.yaml -> build/lib/lm_eval/tasks/glue/sst2 2024-03-18T14:20:30,099 copying lm_eval/tasks/glue/mnli/default.yaml -> build/lib/lm_eval/tasks/glue/mnli 2024-03-18T14:20:30,102 copying lm_eval/tasks/glue/mnli/mismatch.yaml -> build/lib/lm_eval/tasks/glue/mnli 2024-03-18T14:20:30,107 creating build/lib/lm_eval/tasks/glue/cola 2024-03-18T14:20:30,108 copying lm_eval/tasks/glue/cola/default.yaml -> build/lib/lm_eval/tasks/glue/cola 2024-03-18T14:20:30,110 creating build/lib/lm_eval/tasks/glue/qnli 2024-03-18T14:20:30,111 copying lm_eval/tasks/glue/qnli/default.yaml -> build/lib/lm_eval/tasks/glue/qnli 2024-03-18T14:20:30,113 creating build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,114 copying lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,116 copying lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,119 copying lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,121 copying lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,123 copying lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,126 copying lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,128 copying lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,131 copying lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,133 copying lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,135 copying lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,137 copying lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:30,139 creating build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,140 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ja.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,143 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_sw.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,145 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_fr.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,147 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_zh.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,149 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_th.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,152 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_te.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,155 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_de.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,158 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_es.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,161 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_bn.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,163 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_en.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,166 copying lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ru.yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:30,168 creating build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,169 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_fr.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,171 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ja.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,174 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_sw.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,176 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_th.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,178 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_en.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,180 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_es.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,183 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_te.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,185 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_zh.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,187 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_de.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,190 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ru.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,192 copying lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_bn.yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:30,194 creating build/lib/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:30,195 copying lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml -> build/lib/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:30,197 copying lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml -> build/lib/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:30,199 creating build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,200 copying lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,203 copying lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,205 copying lm_eval/tasks/bigbench/generate_until/cryptonite.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,207 copying lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,209 copying lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,211 copying lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,213 copying lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,215 copying lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,218 copying lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,220 copying lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,223 copying lm_eval/tasks/bigbench/generate_until/social_support.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,225 copying lm_eval/tasks/bigbench/generate_until/social_iqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,228 copying lm_eval/tasks/bigbench/generate_until/code_line_description.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,230 copying lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,232 copying lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,235 copying lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,238 copying lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,240 copying lm_eval/tasks/bigbench/generate_until/codenames.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,243 copying lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,245 copying lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,247 copying lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,249 copying lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,251 copying lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,253 copying lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,256 copying lm_eval/tasks/bigbench/generate_until/operators.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,258 copying lm_eval/tasks/bigbench/generate_until/misconceptions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,260 copying lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,263 copying lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,265 copying lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,267 copying lm_eval/tasks/bigbench/generate_until/fact_checker.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,270 copying lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,271 copying lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,273 copying lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,275 copying lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,277 copying lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,279 copying lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,282 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,284 copying lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,286 copying lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,288 copying lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,290 copying lm_eval/tasks/bigbench/generate_until/anachronisms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,292 copying lm_eval/tasks/bigbench/generate_until/topical_chat.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,295 copying lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,297 copying lm_eval/tasks/bigbench/generate_until/list_functions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,299 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,302 copying lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,305 copying lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,308 copying lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,310 copying lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,312 copying lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,314 copying lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,316 copying lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,319 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,321 copying lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,323 copying lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,325 copying lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,327 copying lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,330 copying lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,332 copying lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,335 copying lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,337 copying lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,339 copying lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,341 copying lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,344 copying lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,347 copying lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,349 copying lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,351 copying lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,354 copying lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,356 copying lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,358 copying lm_eval/tasks/bigbench/generate_until/date_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,360 copying lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,363 copying lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,365 copying lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,367 copying lm_eval/tasks/bigbench/generate_until/logical_args.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,370 copying lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,372 copying lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,375 copying lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,378 copying lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,380 copying lm_eval/tasks/bigbench/generate_until/color.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,382 copying lm_eval/tasks/bigbench/generate_until/gem.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,384 copying lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,387 copying lm_eval/tasks/bigbench/generate_until/multiemo.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,389 copying lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,391 copying lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,394 copying lm_eval/tasks/bigbench/generate_until/kannada.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,611 copying lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,614 copying lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,617 copying lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,621 copying lm_eval/tasks/bigbench/generate_until/physics_questions.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,623 copying lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,626 copying lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,629 copying lm_eval/tasks/bigbench/generate_until/crass_ai.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,631 copying lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,634 copying lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,637 copying lm_eval/tasks/bigbench/generate_until/strategyqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,640 copying lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,643 copying lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,647 copying lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,649 copying lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,652 copying lm_eval/tasks/bigbench/generate_until/language_identification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,655 copying lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,658 copying lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,661 copying lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,664 copying lm_eval/tasks/bigbench/generate_until/irony_identification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,667 copying lm_eval/tasks/bigbench/generate_until/rephrase.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,670 copying lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,672 copying lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,675 copying lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,678 copying lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,681 copying lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,684 copying lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,686 copying lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,689 copying lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,692 copying lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,696 copying lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,698 copying lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,701 copying lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,704 copying lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,707 copying lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,710 copying lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,713 copying lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,716 copying lm_eval/tasks/bigbench/generate_until/object_counting.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,719 copying lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,721 copying lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,724 copying lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,727 copying lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,730 copying lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,733 copying lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,736 copying lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,740 copying lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,744 copying lm_eval/tasks/bigbench/generate_until/arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,747 copying lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,751 copying lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,753 copying lm_eval/tasks/bigbench/generate_until/question_selection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,756 copying lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,760 copying lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,763 copying lm_eval/tasks/bigbench/generate_until/word_sorting.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,766 copying lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,769 copying lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,772 copying lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,775 copying lm_eval/tasks/bigbench/generate_until/timedial.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,779 copying lm_eval/tasks/bigbench/generate_until/navigate.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,782 copying lm_eval/tasks/bigbench/generate_until/physics.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,784 copying lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,786 copying lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,788 copying lm_eval/tasks/bigbench/generate_until/snarks.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,791 copying lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,793 copying lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,795 copying lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,798 copying lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,800 copying lm_eval/tasks/bigbench/generate_until/tense.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,803 copying lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,805 copying lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,808 copying lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,810 copying lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,812 copying lm_eval/tasks/bigbench/generate_until/language_games.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,815 copying lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,818 copying lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,820 copying lm_eval/tasks/bigbench/generate_until/implicatures.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,822 copying lm_eval/tasks/bigbench/generate_until/winowhy.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,825 copying lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,827 copying lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:30,830 copying lm_eval/tasks/bigbench/generate_until/strange_stories.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:31,050 copying lm_eval/tasks/bigbench/generate_until/ruin_names.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:31,053 copying lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:31,055 copying lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:31,057 copying lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml -> build/lib/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:31,059 creating build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,060 copying lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,063 copying lm_eval/tasks/bigbench/multiple_choice/causal_judgement.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,065 copying lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,067 copying lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,069 copying lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,071 copying lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,074 copying lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,077 copying lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,079 copying lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,082 copying lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,084 copying lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,087 copying lm_eval/tasks/bigbench/multiple_choice/social_support.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,089 copying lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,091 copying lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,093 copying lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,096 copying lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,098 copying lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,101 copying lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,103 copying lm_eval/tasks/bigbench/multiple_choice/codenames.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,105 copying lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,107 copying lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,110 copying lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,112 copying lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,115 copying lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,117 copying lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,119 copying lm_eval/tasks/bigbench/multiple_choice/operators.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,122 copying lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,124 copying lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,127 copying lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,129 copying lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,131 copying lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,134 copying lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,136 copying lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,138 copying lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,140 copying lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,143 copying lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,145 copying lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,147 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,149 copying lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,151 copying lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,154 copying lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,156 copying lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,158 copying lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,161 copying lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,163 copying lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,165 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,167 copying lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,170 copying lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,172 copying lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,174 copying lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,177 copying lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,179 copying lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,181 copying lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,183 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,186 copying lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,188 copying lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,190 copying lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,192 copying lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,195 copying lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,197 copying lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,199 copying lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,201 copying lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,204 copying lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,207 copying lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,209 copying lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,211 copying lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,214 copying lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,216 copying lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,218 copying lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,221 copying lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,223 copying lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,226 copying lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,228 copying lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,231 copying lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,233 copying lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,235 copying lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,237 copying lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,240 copying lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,243 copying lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,245 copying lm_eval/tasks/bigbench/multiple_choice/color.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,247 copying lm_eval/tasks/bigbench/multiple_choice/gem.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,249 copying lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,252 copying lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,255 copying lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,257 copying lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,259 copying lm_eval/tasks/bigbench/multiple_choice/kannada.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,261 copying lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,263 copying lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,265 copying lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,267 copying lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,270 copying lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,272 copying lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,274 copying lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,276 copying lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,279 copying lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,281 copying lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,283 copying lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,285 copying lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,288 copying lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,290 copying lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,292 copying lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,295 copying lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,297 copying lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,300 copying lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,303 copying lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,305 copying lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,307 copying lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,310 copying lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,312 copying lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,314 copying lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,316 copying lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,319 copying lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,321 copying lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,323 copying lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,325 copying lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,327 copying lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,329 copying lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,331 copying lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,334 copying lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,336 copying lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,338 copying lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,340 copying lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,343 copying lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,345 copying lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,347 copying lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,350 copying lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,352 copying lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,354 copying lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,356 copying lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,358 copying lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,361 copying lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,363 copying lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,365 copying lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,367 copying lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,370 copying lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,372 copying lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,374 copying lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,376 copying lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,379 copying lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,381 copying lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,383 copying lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,385 copying lm_eval/tasks/bigbench/multiple_choice/timedial.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,388 copying lm_eval/tasks/bigbench/multiple_choice/navigate.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,390 copying lm_eval/tasks/bigbench/multiple_choice/physics.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,392 copying lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,394 copying lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,396 copying lm_eval/tasks/bigbench/multiple_choice/snarks.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,399 copying lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,401 copying lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,403 copying lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,406 copying lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,408 copying lm_eval/tasks/bigbench/multiple_choice/tense.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,410 copying lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,413 copying lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,415 copying lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,417 copying lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,420 copying lm_eval/tasks/bigbench/multiple_choice/language_games.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,422 copying lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,425 copying lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,427 copying lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,430 copying lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,433 copying lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,435 copying lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,437 copying lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,440 copying lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,442 copying lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,445 copying lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,447 copying lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml -> build/lib/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:31,449 creating build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,450 copying lm_eval/tasks/unscramble/anagrams2.yaml -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,452 copying lm_eval/tasks/unscramble/anagrams1.yaml -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,455 copying lm_eval/tasks/unscramble/random_insertion.yaml -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,457 copying lm_eval/tasks/unscramble/reversed_words.yaml -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,460 copying lm_eval/tasks/unscramble/cycle_letters.yaml -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:31,462 creating build/lib/lm_eval/tasks/lambada 2024-03-18T14:20:31,463 copying lm_eval/tasks/lambada/lambada_standard.yaml -> build/lib/lm_eval/tasks/lambada 2024-03-18T14:20:31,466 copying lm_eval/tasks/lambada/lambada_openai.yaml -> build/lib/lm_eval/tasks/lambada 2024-03-18T14:20:31,468 copying lm_eval/tasks/blimp/intransitive.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,470 copying lm_eval/tasks/blimp/existential_there_subject_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,472 copying lm_eval/tasks/blimp/principle_A_domain_3.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,475 copying lm_eval/tasks/blimp/passive_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,477 copying lm_eval/tasks/blimp/npi_present_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,479 copying lm_eval/tasks/blimp/principle_A_case_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,481 copying lm_eval/tasks/blimp/inchoative.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,483 copying lm_eval/tasks/blimp/animate_subject_trans.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,486 copying lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,489 copying lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,492 copying lm_eval/tasks/blimp/drop_argument.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,495 copying lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,498 copying lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,502 copying lm_eval/tasks/blimp/wh_questions_object_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,505 copying lm_eval/tasks/blimp/left_branch_island_echo_question.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,508 copying lm_eval/tasks/blimp/passive_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,512 copying lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,515 copying lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,518 copying lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,522 copying lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,525 copying lm_eval/tasks/blimp/expletive_it_object_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,527 copying lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,530 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,533 copying lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,536 copying lm_eval/tasks/blimp/superlative_quantifiers_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,539 copying lm_eval/tasks/blimp/only_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,542 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,545 copying lm_eval/tasks/blimp/superlative_quantifiers_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,548 copying lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,551 copying lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,554 copying lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,557 copying lm_eval/tasks/blimp/principle_A_domain_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,560 copying lm_eval/tasks/blimp/principle_A_domain_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,563 copying lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,565 copying lm_eval/tasks/blimp/tough_vs_raising_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,568 copying lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,571 copying lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,574 copying lm_eval/tasks/blimp/sentential_subject_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,577 copying lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,580 copying lm_eval/tasks/blimp/transitive.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,583 copying lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,586 copying lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,588 copying lm_eval/tasks/blimp/anaphor_gender_agreement.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,591 copying lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,593 copying lm_eval/tasks/blimp/animate_subject_passive.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,595 copying lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,597 copying lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,599 copying lm_eval/tasks/blimp/causative.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,601 copying lm_eval/tasks/blimp/principle_A_case_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,603 copying lm_eval/tasks/blimp/wh_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,606 copying lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,608 copying lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,610 copying lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,612 copying lm_eval/tasks/blimp/principle_A_reconstruction.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,615 copying lm_eval/tasks/blimp/left_branch_island_simple_question.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,617 copying lm_eval/tasks/blimp/anaphor_number_agreement.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,619 copying lm_eval/tasks/blimp/complex_NP_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,622 copying lm_eval/tasks/blimp/tough_vs_raising_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,624 copying lm_eval/tasks/blimp/existential_there_object_raising.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,626 copying lm_eval/tasks/blimp/only_npi_scope.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,629 copying lm_eval/tasks/blimp/npi_present_1.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,631 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,633 copying lm_eval/tasks/blimp/wh_questions_subject_gap.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,635 copying lm_eval/tasks/blimp/principle_A_c_command.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,637 copying lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,640 copying lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,642 copying lm_eval/tasks/blimp/adjunct_island.yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:31,644 creating build/lib/lm_eval/tasks/anli 2024-03-18T14:20:31,645 copying lm_eval/tasks/anli/anli_r1.yaml -> build/lib/lm_eval/tasks/anli 2024-03-18T14:20:31,648 copying lm_eval/tasks/anli/anli_r2.yaml -> build/lib/lm_eval/tasks/anli 2024-03-18T14:20:31,650 copying lm_eval/tasks/anli/anli_r3.yaml -> build/lib/lm_eval/tasks/anli 2024-03-18T14:20:31,652 copying lm_eval/tasks/pubmedqa/pubmedqa.yaml -> build/lib/lm_eval/tasks/pubmedqa 2024-03-18T14:20:31,655 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,657 copying lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,659 copying lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,661 copying lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,663 copying lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,666 copying lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,668 copying lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,670 copying lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,672 copying lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,675 copying lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,677 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,679 copying lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,681 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,684 copying lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,686 copying lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,688 copying lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,691 copying lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,693 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,695 copying lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,698 copying lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,700 copying lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,702 copying lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,704 copying lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,706 copying lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,708 copying lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,711 copying lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,713 copying lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:31,715 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,718 copying lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,720 copying lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,723 copying lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,725 copying lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,727 copying lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,729 copying lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,732 copying lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,734 copying lm_eval/tasks/bbh/zeroshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,736 copying lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,738 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,740 copying lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,743 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,745 copying lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,747 copying lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,749 copying lm_eval/tasks/bbh/zeroshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,751 copying lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,754 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,756 copying lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,758 copying lm_eval/tasks/bbh/zeroshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,761 copying lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,763 copying lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,766 copying lm_eval/tasks/bbh/zeroshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,768 copying lm_eval/tasks/bbh/zeroshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,770 copying lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,772 copying lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,775 copying lm_eval/tasks/bbh/zeroshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:31,776 creating build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,778 copying lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,780 copying lm_eval/tasks/bbh/fewshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,782 copying lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,785 copying lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,787 copying lm_eval/tasks/bbh/fewshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,789 copying lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,791 copying lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,794 copying lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,796 copying lm_eval/tasks/bbh/fewshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,800 copying lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,802 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,805 copying lm_eval/tasks/bbh/fewshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,807 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,809 copying lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,812 copying lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,815 copying lm_eval/tasks/bbh/fewshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,817 copying lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,820 copying lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,823 copying lm_eval/tasks/bbh/fewshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,825 copying lm_eval/tasks/bbh/fewshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,828 copying lm_eval/tasks/bbh/fewshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,830 copying lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,833 copying lm_eval/tasks/bbh/fewshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,835 copying lm_eval/tasks/bbh/fewshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,838 copying lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,840 copying lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,843 copying lm_eval/tasks/bbh/fewshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:31,845 creating build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,846 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,848 copying lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,851 copying lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,853 copying lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,855 copying lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,858 copying lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,860 copying lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,862 copying lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,865 copying lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,867 copying lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,869 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,872 copying lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,874 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,877 copying lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,880 copying lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,882 copying lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,884 copying lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,887 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,890 copying lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,892 copying lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,895 copying lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,898 copying lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,900 copying lm_eval/tasks/bbh/cot_fewshot/navigate.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,903 copying lm_eval/tasks/bbh/cot_fewshot/snarks.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,905 copying lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,908 copying lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,910 copying lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:31,913 copying lm_eval/tasks/winogrande/default.yaml -> build/lib/lm_eval/tasks/winogrande 2024-03-18T14:20:31,915 creating build/lib/lm_eval/tasks/babi 2024-03-18T14:20:31,916 copying lm_eval/tasks/babi/babi.yaml -> build/lib/lm_eval/tasks/babi 2024-03-18T14:20:31,918 creating build/lib/lm_eval/tasks/asdiv 2024-03-18T14:20:31,919 copying lm_eval/tasks/asdiv/default.yaml -> build/lib/lm_eval/tasks/asdiv 2024-03-18T14:20:31,921 creating build/lib/lm_eval/tasks/mc_taco 2024-03-18T14:20:31,922 copying lm_eval/tasks/mc_taco/default.yaml -> build/lib/lm_eval/tasks/mc_taco 2024-03-18T14:20:31,925 creating build/lib/lm_eval/tasks/siqa 2024-03-18T14:20:31,926 copying lm_eval/tasks/siqa/siqa.yaml -> build/lib/lm_eval/tasks/siqa 2024-03-18T14:20:31,929 copying lm_eval/tasks/scrolls/scrolls.yaml -> build/lib/lm_eval/tasks/scrolls 2024-03-18T14:20:31,931 copying lm_eval/tasks/webqs/webqs.yaml -> build/lib/lm_eval/tasks/webqs 2024-03-18T14:20:31,933 copying lm_eval/tasks/kobest/kobest_hellaswag.yaml -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:31,935 copying lm_eval/tasks/kobest/kobest_boolq.yaml -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:31,938 copying lm_eval/tasks/kobest/kobest_copa.yaml -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:31,940 copying lm_eval/tasks/kobest/kobest_sentineg.yaml -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:31,942 copying lm_eval/tasks/kobest/kobest_wic.yaml -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:31,944 creating build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,945 copying lm_eval/tasks/pile/pile_ubuntu-irc.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,947 copying lm_eval/tasks/pile/pile_arxiv.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,949 copying lm_eval/tasks/pile/pile_opensubtitles.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,951 copying lm_eval/tasks/pile/pile_nih-exporter.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,954 copying lm_eval/tasks/pile/pile_europarl.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,956 copying lm_eval/tasks/pile/pile_dm-mathematics.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,958 copying lm_eval/tasks/pile/pile_enron.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,961 copying lm_eval/tasks/pile/pile_books3.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,963 copying lm_eval/tasks/pile/pile_youtubesubtitles.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,965 copying lm_eval/tasks/pile/pile_stackexchange.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,967 copying lm_eval/tasks/pile/pile_gutenberg.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,969 copying lm_eval/tasks/pile/pile_pubmed-central.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,971 copying lm_eval/tasks/pile/pile_github.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,974 copying lm_eval/tasks/pile/pile_philpapers.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,976 copying lm_eval/tasks/pile/pile_bookcorpus2.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,978 copying lm_eval/tasks/pile/pile_pile-cc.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,981 copying lm_eval/tasks/pile/pile_pubmed-abstracts.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,983 copying lm_eval/tasks/pile/pile_freelaw.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,985 copying lm_eval/tasks/pile/pile_uspto.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,987 copying lm_eval/tasks/pile/pile_openwebtext2.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,990 copying lm_eval/tasks/pile/pile_wikipedia.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,992 copying lm_eval/tasks/pile/pile_hackernews.yaml -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:31,994 copying lm_eval/tasks/hendrycks_ethics/deontology.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:31,996 copying lm_eval/tasks/hendrycks_ethics/justice.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:31,999 copying lm_eval/tasks/hendrycks_ethics/virtue.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:32,001 copying lm_eval/tasks/hendrycks_ethics/commonsense.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:32,003 copying lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:32,005 creating build/lib/lm_eval/tasks/sciq 2024-03-18T14:20:32,006 copying lm_eval/tasks/sciq/sciq.yaml -> build/lib/lm_eval/tasks/sciq 2024-03-18T14:20:32,008 creating build/lib/lm_eval/tasks/openbookqa 2024-03-18T14:20:32,009 copying lm_eval/tasks/openbookqa/openbookqa.yaml -> build/lib/lm_eval/tasks/openbookqa 2024-03-18T14:20:32,011 copying lm_eval/tasks/logiqa2/logiqa2.yaml -> build/lib/lm_eval/tasks/logiqa2 2024-03-18T14:20:32,013 copying lm_eval/tasks/logiqa2/logieval.yaml -> build/lib/lm_eval/tasks/logiqa2 2024-03-18T14:20:32,016 copying lm_eval/tasks/gpqa/cot_zeroshot/gpqa_main_cot_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:32,018 copying lm_eval/tasks/gpqa/cot_zeroshot/gpqa_diamond_cot_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:32,020 copying lm_eval/tasks/gpqa/cot_zeroshot/gpqa_extended_cot_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:32,022 copying lm_eval/tasks/gpqa/generative/gpqa_extended_generative_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:32,024 copying lm_eval/tasks/gpqa/generative/gpqa_diamond_generative_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:32,026 copying lm_eval/tasks/gpqa/generative/gpqa_main_generative_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:32,029 copying lm_eval/tasks/gpqa/n_shot/gpqa_main_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:32,031 copying lm_eval/tasks/gpqa/n_shot/gpqa_extended_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:32,033 copying lm_eval/tasks/gpqa/n_shot/gpqa_diamond_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:32,035 copying lm_eval/tasks/gpqa/zeroshot/gpqa_diamond_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:32,037 copying lm_eval/tasks/gpqa/zeroshot/gpqa_extended_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:32,039 copying lm_eval/tasks/gpqa/zeroshot/gpqa_main_zeroshot.yaml -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:32,042 copying lm_eval/tasks/gpqa/cot_n_shot/gpqa_main_cot_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:32,044 copying lm_eval/tasks/gpqa/cot_n_shot/gpqa_diamond_cot_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:32,046 copying lm_eval/tasks/gpqa/cot_n_shot/gpqa_extended_cot_n_shot.yaml -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:32,048 copying lm_eval/tasks/ceval/ceval-valid_operating_system.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,050 copying lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,052 copying lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,054 copying lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,056 copying lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,059 copying lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,061 copying lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,063 copying lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,065 copying lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,067 copying lm_eval/tasks/ceval/ceval-valid_physician.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,069 copying lm_eval/tasks/ceval/ceval-valid_accountant.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,071 copying lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,073 copying lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,075 copying lm_eval/tasks/ceval/ceval-valid_art_studies.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,078 copying lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,080 copying lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,082 copying lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,084 copying lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,087 copying lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,088 copying lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,091 copying lm_eval/tasks/ceval/ceval-valid_college_programming.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,093 copying lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,095 copying lm_eval/tasks/ceval/ceval-valid_sports_science.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,097 copying lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,099 copying lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,101 copying lm_eval/tasks/ceval/ceval-valid_logic.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,103 copying lm_eval/tasks/ceval/ceval-valid_college_economics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,105 copying lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,107 copying lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,110 copying lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,112 copying lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,114 copying lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,116 copying lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,118 copying lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,121 copying lm_eval/tasks/ceval/ceval-valid_law.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,124 copying lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,126 copying lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,128 copying lm_eval/tasks/ceval/ceval-valid_computer_network.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,130 copying lm_eval/tasks/ceval/ceval-valid_education_science.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,132 copying lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,134 copying lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,137 copying lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,139 copying lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,141 copying lm_eval/tasks/ceval/ceval-valid_marxism.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,143 copying lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,146 copying lm_eval/tasks/ceval/ceval-valid_business_administration.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,148 copying lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,150 copying lm_eval/tasks/ceval/ceval-valid_college_physics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,152 copying lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,155 copying lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,157 copying lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,159 copying lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:32,161 creating build/lib/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:32,162 copying lm_eval/tasks/kormedmcqa/kormedmcqa_doctor.yaml -> build/lib/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:32,164 copying lm_eval/tasks/kormedmcqa/kormedmcqa_pharm.yaml -> build/lib/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:32,166 copying lm_eval/tasks/kormedmcqa/kormedmcqa_nurse.yaml -> build/lib/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:32,168 creating build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,170 copying lm_eval/tasks/arithmetic/arithmetic_2ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,172 copying lm_eval/tasks/arithmetic/arithmetic_4ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,174 copying lm_eval/tasks/arithmetic/arithmetic_2dm.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,176 copying lm_eval/tasks/arithmetic/arithmetic_3da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,179 copying lm_eval/tasks/arithmetic/arithmetic_1dc.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,181 copying lm_eval/tasks/arithmetic/arithmetic_5da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,183 copying lm_eval/tasks/arithmetic/arithmetic_4da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,186 copying lm_eval/tasks/arithmetic/arithmetic_5ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,188 copying lm_eval/tasks/arithmetic/arithmetic_2da.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,191 copying lm_eval/tasks/arithmetic/arithmetic_3ds.yaml -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:32,193 copying lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml -> build/lib/lm_eval/tasks/wmt2016 2024-03-18T14:20:32,195 creating build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,196 copying lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,198 copying lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,201 copying lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,203 copying lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,205 copying lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:32,207 copying lm_eval/tasks/squadv2/squadv2.yaml -> build/lib/lm_eval/tasks/squadv2 2024-03-18T14:20:32,209 copying lm_eval/tasks/eq_bench/default.yaml -> build/lib/lm_eval/tasks/eq_bench 2024-03-18T14:20:32,211 creating build/lib/lm_eval/tasks/prost 2024-03-18T14:20:32,212 copying lm_eval/tasks/prost/corypaik_prost.yaml -> build/lib/lm_eval/tasks/prost 2024-03-18T14:20:32,215 copying lm_eval/tasks/coqa/default.yaml -> build/lib/lm_eval/tasks/coqa 2024-03-18T14:20:32,217 copying lm_eval/tasks/medmcqa/medmcqa.yaml -> build/lib/lm_eval/tasks/medmcqa 2024-03-18T14:20:32,220 copying lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,222 copying lm_eval/tasks/minerva_math/minerva_math_geometry.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,224 copying lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,226 copying lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,228 copying lm_eval/tasks/minerva_math/minerva_math_precalc.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,230 copying lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,232 copying lm_eval/tasks/minerva_math/minerva_math_algebra.yaml -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:32,234 copying lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,236 copying lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,239 copying lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,241 copying lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,243 copying lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,245 copying lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,247 copying lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,250 copying lm_eval/tasks/crows_pairs/crows_pairs_english.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,252 copying lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,254 copying lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,257 copying lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,260 copying lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,262 copying lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,264 copying lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,267 copying lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,269 copying lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,271 copying lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,274 copying lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,276 copying lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,279 copying lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,281 copying lm_eval/tasks/crows_pairs/crows_pairs_french.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,283 copying lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:32,285 creating build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,286 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,289 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,291 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,293 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,296 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,298 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,301 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,303 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,306 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,309 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,312 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,314 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,317 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,320 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,323 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,326 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,328 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,331 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,334 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,336 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,339 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,341 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,344 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,347 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,350 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,352 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,355 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,358 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,361 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,364 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,367 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,369 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,372 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,374 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,377 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,380 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,383 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,385 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,388 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,391 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,394 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,396 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,398 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,400 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,403 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,406 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,408 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,410 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,412 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,415 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,417 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,419 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,421 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,424 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,426 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,428 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,430 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,433 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:32,435 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,437 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,440 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,442 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,444 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,446 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,449 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,451 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,454 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,456 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,458 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,461 copying lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,464 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,466 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,469 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,471 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,474 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,476 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,478 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,481 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,484 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,487 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,489 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,491 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,493 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,495 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,498 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,500 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,504 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,506 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,509 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,511 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,513 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,516 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,518 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,521 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,523 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,526 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,528 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,530 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,533 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,535 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,538 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,540 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,543 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,545 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,547 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,550 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,552 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,554 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,557 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,559 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,561 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,563 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,566 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,568 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,571 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,573 copying lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:32,575 creating build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,576 copying lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,579 copying lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,581 copying lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,584 copying lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,586 copying lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,588 copying lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,591 copying lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,594 copying lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,596 copying lm_eval/tasks/mmlu/default/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,599 copying lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,601 copying lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,604 copying lm_eval/tasks/mmlu/default/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,606 copying lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,608 copying lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,611 copying lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,614 copying lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,616 copying lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,619 copying lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,621 copying lm_eval/tasks/mmlu/default/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,624 copying lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,626 copying lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,628 copying lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,630 copying lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,632 copying lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,635 copying lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,637 copying lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,639 copying lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,642 copying lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,644 copying lm_eval/tasks/mmlu/default/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,646 copying lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,648 copying lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,651 copying lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,654 copying lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,656 copying lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,659 copying lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,661 copying lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,664 copying lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,666 copying lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,669 copying lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,671 copying lm_eval/tasks/mmlu/default/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,674 copying lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,676 copying lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,678 copying lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,681 copying lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,683 copying lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,686 copying lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,688 copying lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,690 copying lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,692 copying lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,695 copying lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,697 copying lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,699 copying lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,702 copying lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,704 copying lm_eval/tasks/mmlu/default/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,707 copying lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,710 copying lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,712 copying lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,714 copying lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:32,716 creating build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,717 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,719 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,722 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,724 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,726 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,729 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,732 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,734 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,737 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,739 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,742 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,744 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,746 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,749 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,751 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,754 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,756 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,759 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,761 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,763 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,766 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,768 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,771 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,774 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,777 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,780 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,782 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,785 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,787 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,790 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,792 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,795 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,797 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,800 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,803 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,805 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,807 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,810 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,812 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,815 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,817 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,820 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,823 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,825 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,828 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,830 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,833 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,835 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,838 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,841 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,843 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,846 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,848 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,850 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,853 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,856 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,858 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,860 copying lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:32,863 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,865 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,868 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,870 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,872 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,874 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,876 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,878 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,881 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,883 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,885 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,888 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,891 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,894 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,896 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,898 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,900 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,903 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,905 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,907 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,910 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,912 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,915 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,917 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,919 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,922 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,924 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,926 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,929 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,932 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,934 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,936 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,938 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,941 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,943 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,945 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,948 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,950 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,952 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,955 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,957 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,959 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,962 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,964 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,966 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,968 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,971 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,973 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,975 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,977 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,980 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,983 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,985 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,987 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,989 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,992 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,994 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,996 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:32,998 copying lm_eval/tasks/logiqa/logiqa.yaml -> build/lib/lm_eval/tasks/logiqa 2024-03-18T14:20:33,000 creating build/lib/lm_eval/tasks/polemo2 2024-03-18T14:20:33,001 copying lm_eval/tasks/polemo2/polemo2_out.yaml -> build/lib/lm_eval/tasks/polemo2 2024-03-18T14:20:33,004 copying lm_eval/tasks/polemo2/polemo2_in.yaml -> build/lib/lm_eval/tasks/polemo2 2024-03-18T14:20:33,006 copying lm_eval/tasks/belebele/belebele_lin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,008 copying lm_eval/tasks/belebele/belebele_mar_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,010 copying lm_eval/tasks/belebele/belebele_fra_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,012 copying lm_eval/tasks/belebele/belebele_swe_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,014 copying lm_eval/tasks/belebele/belebele_pbt_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,017 copying lm_eval/tasks/belebele/belebele_jav_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,019 copying lm_eval/tasks/belebele/belebele_fuv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,021 copying lm_eval/tasks/belebele/belebele_plt_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,023 copying lm_eval/tasks/belebele/belebele_spa_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,025 copying lm_eval/tasks/belebele/belebele_war_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,028 copying lm_eval/tasks/belebele/belebele_ben_Beng.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,030 copying lm_eval/tasks/belebele/belebele_ibo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,032 copying lm_eval/tasks/belebele/belebele_tam_Taml.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,034 copying lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,036 copying lm_eval/tasks/belebele/belebele_eus_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,038 copying lm_eval/tasks/belebele/belebele_bam_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,040 copying lm_eval/tasks/belebele/belebele_arb_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,042 copying lm_eval/tasks/belebele/belebele_kan_Knda.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,044 copying lm_eval/tasks/belebele/belebele_mya_Mymr.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,046 copying lm_eval/tasks/belebele/belebele_hin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,048 copying lm_eval/tasks/belebele/belebele_dan_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,050 copying lm_eval/tasks/belebele/belebele_ssw_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,052 copying lm_eval/tasks/belebele/belebele_ita_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,054 copying lm_eval/tasks/belebele/belebele_hin_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,057 copying lm_eval/tasks/belebele/belebele_pan_Guru.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,059 copying lm_eval/tasks/belebele/belebele_shn_Mymr.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,061 copying lm_eval/tasks/belebele/belebele_tgl_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,063 copying lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,066 copying lm_eval/tasks/belebele/belebele_kea_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,068 copying lm_eval/tasks/belebele/belebele_arz_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,070 copying lm_eval/tasks/belebele/belebele_eng_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,072 copying lm_eval/tasks/belebele/belebele_tel_Telu.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,074 copying lm_eval/tasks/belebele/belebele_tso_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,076 copying lm_eval/tasks/belebele/belebele_luo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,079 copying lm_eval/tasks/belebele/belebele_zho_Hant.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,082 copying lm_eval/tasks/belebele/belebele_ces_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,084 copying lm_eval/tasks/belebele/belebele_nya_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,086 copying lm_eval/tasks/belebele/belebele_pol_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,088 copying lm_eval/tasks/belebele/belebele_mal_Mlym.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,091 copying lm_eval/tasks/belebele/belebele_nld_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,094 copying lm_eval/tasks/belebele/belebele_npi_Deva.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,098 copying lm_eval/tasks/belebele/belebele_pes_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,104 copying lm_eval/tasks/belebele/belebele_kat_Geor.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,111 copying lm_eval/tasks/belebele/belebele_ind_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,115 copying lm_eval/tasks/belebele/belebele_hun_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,118 copying lm_eval/tasks/belebele/belebele_ceb_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,121 copying lm_eval/tasks/belebele/belebele_tir_Ethi.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,125 copying lm_eval/tasks/belebele/belebele_npi_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,128 copying lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,131 copying lm_eval/tasks/belebele/belebele_apc_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,134 copying lm_eval/tasks/belebele/belebele_tsn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,137 copying lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,140 copying lm_eval/tasks/belebele/belebele_tha_Thai.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,144 copying lm_eval/tasks/belebele/belebele_sun_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,147 copying lm_eval/tasks/belebele/belebele_ron_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,151 copying lm_eval/tasks/belebele/belebele_wol_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,154 copying lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,157 copying lm_eval/tasks/belebele/belebele_deu_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,160 copying lm_eval/tasks/belebele/belebele_ell_Grek.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,164 copying lm_eval/tasks/belebele/belebele_swh_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,167 copying lm_eval/tasks/belebele/belebele_als_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,170 copying lm_eval/tasks/belebele/belebele_acm_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,173 copying lm_eval/tasks/belebele/belebele_nob_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,176 copying lm_eval/tasks/belebele/belebele_ckb_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,179 copying lm_eval/tasks/belebele/belebele_kin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,182 copying lm_eval/tasks/belebele/belebele_sin_Sinh.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,185 copying lm_eval/tasks/belebele/belebele_xho_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,188 copying lm_eval/tasks/belebele/belebele_fin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,192 copying lm_eval/tasks/belebele/belebele_kac_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,195 copying lm_eval/tasks/belebele/belebele_mlt_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,198 copying lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,201 copying lm_eval/tasks/belebele/belebele_uzn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,204 copying lm_eval/tasks/belebele/belebele_azj_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,206 copying lm_eval/tasks/belebele/belebele_lao_Laoo.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,209 copying lm_eval/tasks/belebele/belebele_slv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,212 copying lm_eval/tasks/belebele/belebele_est_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,214 copying lm_eval/tasks/belebele/belebele_sna_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,217 copying lm_eval/tasks/belebele/belebele_urd_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,219 copying lm_eval/tasks/belebele/belebele_guj_Gujr.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,222 copying lm_eval/tasks/belebele/belebele_som_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,224 copying lm_eval/tasks/belebele/belebele_tur_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,227 copying lm_eval/tasks/belebele/belebele_amh_Ethi.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,229 copying lm_eval/tasks/belebele/belebele_cat_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,232 copying lm_eval/tasks/belebele/belebele_khm_Khmr.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,235 copying lm_eval/tasks/belebele/belebele_mri_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,238 copying lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,240 copying lm_eval/tasks/belebele/belebele_urd_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,242 copying lm_eval/tasks/belebele/belebele_ars_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,244 copying lm_eval/tasks/belebele/belebele_heb_Hebr.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,247 copying lm_eval/tasks/belebele/belebele_lug_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,249 copying lm_eval/tasks/belebele/belebele_sot_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,251 copying lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,253 copying lm_eval/tasks/belebele/belebele_slk_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,256 copying lm_eval/tasks/belebele/belebele_por_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,258 copying lm_eval/tasks/belebele/belebele_ary_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,260 copying lm_eval/tasks/belebele/belebele_afr_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,263 copying lm_eval/tasks/belebele/belebele_hrv_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,265 copying lm_eval/tasks/belebele/belebele_ilo_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,267 copying lm_eval/tasks/belebele/belebele_zul_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,270 copying lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,272 copying lm_eval/tasks/belebele/belebele_snd_Arab.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,275 copying lm_eval/tasks/belebele/belebele_isl_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,278 copying lm_eval/tasks/belebele/belebele_arb_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,281 copying lm_eval/tasks/belebele/belebele_sin_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,283 copying lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,286 copying lm_eval/tasks/belebele/belebele_hat_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,289 copying lm_eval/tasks/belebele/belebele_lvs_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,292 copying lm_eval/tasks/belebele/belebele_ory_Orya.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,294 copying lm_eval/tasks/belebele/belebele_kor_Hang.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,297 copying lm_eval/tasks/belebele/belebele_vie_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,300 copying lm_eval/tasks/belebele/belebele_ben_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,302 copying lm_eval/tasks/belebele/belebele_nso_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,305 copying lm_eval/tasks/belebele/belebele_zho_Hans.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,308 copying lm_eval/tasks/belebele/belebele_lit_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,310 copying lm_eval/tasks/belebele/belebele_zsm_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,314 copying lm_eval/tasks/belebele/belebele_yor_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,316 copying lm_eval/tasks/belebele/belebele_bod_Tibt.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,319 copying lm_eval/tasks/belebele/belebele_asm_Beng.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,322 copying lm_eval/tasks/belebele/belebele_grn_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,324 copying lm_eval/tasks/belebele/belebele_gaz_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,327 copying lm_eval/tasks/belebele/belebele_hau_Latn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,331 copying lm_eval/tasks/belebele/belebele_hye_Armn.yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,333 creating build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,335 copying lm_eval/tasks/xstorycloze/default_es.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,338 copying lm_eval/tasks/xstorycloze/default_te.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,341 copying lm_eval/tasks/xstorycloze/default_ru.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,343 copying lm_eval/tasks/xstorycloze/default_my.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,345 copying lm_eval/tasks/xstorycloze/default_sw.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,348 copying lm_eval/tasks/xstorycloze/default_hi.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,351 copying lm_eval/tasks/xstorycloze/default_eu.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,353 copying lm_eval/tasks/xstorycloze/default_ar.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,356 copying lm_eval/tasks/xstorycloze/default_id.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,358 copying lm_eval/tasks/xstorycloze/default_en.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,360 copying lm_eval/tasks/xstorycloze/default_zh.yaml -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,363 creating build/lib/lm_eval/tasks/headqa 2024-03-18T14:20:33,364 copying lm_eval/tasks/headqa/headqa_es.yaml -> build/lib/lm_eval/tasks/headqa 2024-03-18T14:20:33,366 copying lm_eval/tasks/headqa/headqa_en.yaml -> build/lib/lm_eval/tasks/headqa 2024-03-18T14:20:33,368 copying lm_eval/tasks/drop/default.yaml -> build/lib/lm_eval/tasks/drop 2024-03-18T14:20:33,371 copying lm_eval/tasks/hellaswag/hellaswag.yaml -> build/lib/lm_eval/tasks/hellaswag 2024-03-18T14:20:33,373 copying lm_eval/tasks/csatqa/csatqa_rcs.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,376 copying lm_eval/tasks/csatqa/csatqa_rcss.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,378 copying lm_eval/tasks/csatqa/csatqa_wr.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,381 copying lm_eval/tasks/csatqa/csatqa_gr.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,383 copying lm_eval/tasks/csatqa/csatqa_rch.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,385 copying lm_eval/tasks/csatqa/csatqa_li.yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:33,388 copying lm_eval/tasks/code_x_glue/code-text/ruby.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,390 copying lm_eval/tasks/code_x_glue/code-text/go.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,392 copying lm_eval/tasks/code_x_glue/code-text/javascript.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,395 copying lm_eval/tasks/code_x_glue/code-text/php.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,403 copying lm_eval/tasks/code_x_glue/code-text/java.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,405 copying lm_eval/tasks/code_x_glue/code-text/python.yaml -> build/lib/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:33,409 copying lm_eval/tasks/mathqa/README.md -> build/lib/lm_eval/tasks/mathqa 2024-03-18T14:20:33,412 copying lm_eval/tasks/paws-x/README.md -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:33,415 copying lm_eval/tasks/paws-x/pawsx_template_yaml -> build/lib/lm_eval/tasks/paws-x 2024-03-18T14:20:33,418 copying lm_eval/tasks/mutual/README.md -> build/lib/lm_eval/tasks/mutual 2024-03-18T14:20:33,421 copying lm_eval/tasks/swag/README.md -> build/lib/lm_eval/tasks/swag 2024-03-18T14:20:33,425 copying lm_eval/tasks/qasper/README.md -> build/lib/lm_eval/tasks/qasper 2024-03-18T14:20:33,428 copying lm_eval/tasks/gsm8k/README.md -> build/lib/lm_eval/tasks/gsm8k 2024-03-18T14:20:33,431 copying lm_eval/tasks/kmmlu/README.md -> build/lib/lm_eval/tasks/kmmlu 2024-03-18T14:20:33,434 copying lm_eval/tasks/kmmlu/direct/_direct_kmmlu_yaml -> build/lib/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:33,436 copying lm_eval/tasks/kmmlu/direct_hard/_direct_hard_kmmlu_yaml -> build/lib/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:33,438 copying lm_eval/tasks/kmmlu/cot_hard/_cot_kmmlu_yaml -> build/lib/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:33,441 copying lm_eval/tasks/kmmlu/hard/_hard_kmmlu_yaml -> build/lib/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:33,444 copying lm_eval/tasks/race/README.md -> build/lib/lm_eval/tasks/race 2024-03-18T14:20:33,446 copying lm_eval/tasks/nq_open/README.md -> build/lib/lm_eval/tasks/nq_open 2024-03-18T14:20:33,449 copying lm_eval/tasks/truthfulqa/README.md -> build/lib/lm_eval/tasks/truthfulqa 2024-03-18T14:20:33,452 copying lm_eval/tasks/xwinograd/xwinograd_common_yaml -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:33,455 copying lm_eval/tasks/xwinograd/README.md -> build/lib/lm_eval/tasks/xwinograd 2024-03-18T14:20:33,459 copying lm_eval/tasks/wikitext/README.md -> build/lib/lm_eval/tasks/wikitext 2024-03-18T14:20:33,461 copying lm_eval/tasks/super_glue/README.md -> build/lib/lm_eval/tasks/super_glue 2024-03-18T14:20:33,468 copying lm_eval/tasks/wmdp/README.md -> build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:33,470 copying lm_eval/tasks/wmdp/_default_template_yaml -> build/lib/lm_eval/tasks/wmdp 2024-03-18T14:20:33,472 copying lm_eval/tasks/haerae/_default_haerae_yaml -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:33,474 copying lm_eval/tasks/haerae/README.md -> build/lib/lm_eval/tasks/haerae 2024-03-18T14:20:33,477 copying lm_eval/tasks/cmmlu/README.md -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:33,479 copying lm_eval/tasks/cmmlu/_default_template_yaml -> build/lib/lm_eval/tasks/cmmlu 2024-03-18T14:20:33,481 copying lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc1_yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:33,484 copying lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc2_yaml -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:33,486 copying lm_eval/tasks/okapi/truthfulqa_multilingual/README.md -> build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:33,489 copying lm_eval/tasks/okapi/hellaswag_multilingual/_hellaswag_yaml -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:33,491 copying lm_eval/tasks/okapi/hellaswag_multilingual/README.md -> build/lib/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:33,494 copying lm_eval/tasks/okapi/mmlu_multilingual/_default_yaml -> build/lib/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:33,496 copying lm_eval/tasks/okapi/arc_multilingual/README.md -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:33,498 copying lm_eval/tasks/okapi/arc_multilingual/_arc_yaml -> build/lib/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:33,501 copying lm_eval/tasks/translation/wmt_common_yaml -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:33,503 copying lm_eval/tasks/translation/README.md -> build/lib/lm_eval/tasks/translation 2024-03-18T14:20:33,506 copying lm_eval/tasks/ammlu/README.md -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:33,508 copying lm_eval/tasks/ammlu/_default_template_yaml -> build/lib/lm_eval/tasks/ammlu 2024-03-18T14:20:33,511 copying lm_eval/tasks/storycloze/README.md -> build/lib/lm_eval/tasks/storycloze 2024-03-18T14:20:33,513 copying lm_eval/tasks/fld/README.md -> build/lib/lm_eval/tasks/fld 2024-03-18T14:20:33,516 copying lm_eval/tasks/toxigen/README.md -> build/lib/lm_eval/tasks/toxigen 2024-03-18T14:20:33,518 copying lm_eval/tasks/model_written_evals/persona/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:33,521 creating build/lib/lm_eval/tasks/model_written_evals/winogenerated 2024-03-18T14:20:33,522 copying lm_eval/tasks/model_written_evals/winogenerated/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/winogenerated 2024-03-18T14:20:33,524 copying lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml -> build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:33,527 copying lm_eval/tasks/agieval/README.md -> build/lib/lm_eval/tasks/agieval 2024-03-18T14:20:33,530 copying lm_eval/tasks/wsc273/README.md -> build/lib/lm_eval/tasks/wsc273 2024-03-18T14:20:33,532 copying lm_eval/tasks/arc/README.md -> build/lib/lm_eval/tasks/arc 2024-03-18T14:20:33,535 copying lm_eval/tasks/xcopa/README.md -> build/lib/lm_eval/tasks/xcopa 2024-03-18T14:20:33,538 copying lm_eval/tasks/triviaqa/README.md -> build/lib/lm_eval/tasks/triviaqa 2024-03-18T14:20:33,540 copying lm_eval/tasks/benchmarks/flan/_held_in_template_yaml -> build/lib/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:33,542 copying lm_eval/tasks/benchmarks/multimedqa/README.md -> build/lib/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:33,544 copying lm_eval/tasks/xnli/xnli_common_yaml -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:33,547 copying lm_eval/tasks/xnli/README.md -> build/lib/lm_eval/tasks/xnli 2024-03-18T14:20:33,550 copying lm_eval/tasks/piqa/README.md -> build/lib/lm_eval/tasks/piqa 2024-03-18T14:20:33,553 copying lm_eval/tasks/french_bench/README.md -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:33,556 copying lm_eval/tasks/french_bench/_default_template_yaml -> build/lib/lm_eval/tasks/french_bench 2024-03-18T14:20:33,560 copying lm_eval/tasks/ifeval/README.md -> build/lib/lm_eval/tasks/ifeval 2024-03-18T14:20:33,563 copying lm_eval/tasks/qa4mre/README.md -> build/lib/lm_eval/tasks/qa4mre 2024-03-18T14:20:33,566 copying lm_eval/tasks/aexams/README.md -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:33,568 copying lm_eval/tasks/aexams/_default_template_yaml -> build/lib/lm_eval/tasks/aexams 2024-03-18T14:20:33,571 copying lm_eval/tasks/glue/README.md -> build/lib/lm_eval/tasks/glue 2024-03-18T14:20:33,574 copying lm_eval/tasks/mgsm/gen_yaml.sh -> build/lib/lm_eval/tasks/mgsm 2024-03-18T14:20:33,576 copying lm_eval/tasks/mgsm/README.md -> build/lib/lm_eval/tasks/mgsm 2024-03-18T14:20:33,579 copying lm_eval/tasks/mgsm/direct/direct_yaml -> build/lib/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:33,581 copying lm_eval/tasks/mgsm/en_cot/cot_yaml -> build/lib/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:33,584 copying lm_eval/tasks/mgsm/native_cot/cot_yaml -> build/lib/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:33,586 copying lm_eval/tasks/lambada_cloze/README.md -> build/lib/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:33,588 copying lm_eval/tasks/bigbench/multiple_choice_template_yaml -> build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:33,592 copying lm_eval/tasks/bigbench/README.md -> build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:33,595 copying lm_eval/tasks/bigbench/generate_until_template_yaml -> build/lib/lm_eval/tasks/bigbench 2024-03-18T14:20:33,597 copying lm_eval/tasks/unscramble/README.md -> build/lib/lm_eval/tasks/unscramble 2024-03-18T14:20:33,600 copying lm_eval/tasks/lambada/README.md -> build/lib/lm_eval/tasks/lambada 2024-03-18T14:20:33,602 copying lm_eval/tasks/blimp/_template_yaml -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:33,605 copying lm_eval/tasks/blimp/README.md -> build/lib/lm_eval/tasks/blimp 2024-03-18T14:20:33,607 copying lm_eval/tasks/anli/README.md -> build/lib/lm_eval/tasks/anli 2024-03-18T14:20:33,610 copying lm_eval/tasks/pubmedqa/README.md -> build/lib/lm_eval/tasks/pubmedqa 2024-03-18T14:20:33,613 copying lm_eval/tasks/bbh/README.md -> build/lib/lm_eval/tasks/bbh 2024-03-18T14:20:33,615 copying lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml -> build/lib/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:33,618 copying lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml -> build/lib/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:33,620 copying lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml -> build/lib/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:33,622 copying lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml -> build/lib/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:33,625 copying lm_eval/tasks/winogrande/README.md -> build/lib/lm_eval/tasks/winogrande 2024-03-18T14:20:33,627 copying lm_eval/tasks/babi/README.md -> build/lib/lm_eval/tasks/babi 2024-03-18T14:20:33,629 copying lm_eval/tasks/asdiv/README.md -> build/lib/lm_eval/tasks/asdiv 2024-03-18T14:20:33,632 copying lm_eval/tasks/mc_taco/README.md -> build/lib/lm_eval/tasks/mc_taco 2024-03-18T14:20:33,634 copying lm_eval/tasks/siqa/README.md -> build/lib/lm_eval/tasks/siqa 2024-03-18T14:20:33,636 copying lm_eval/tasks/scrolls/README.md -> build/lib/lm_eval/tasks/scrolls 2024-03-18T14:20:33,640 copying lm_eval/tasks/webqs/README.md -> build/lib/lm_eval/tasks/webqs 2024-03-18T14:20:33,643 copying lm_eval/tasks/kobest/README.md -> build/lib/lm_eval/tasks/kobest 2024-03-18T14:20:33,645 copying lm_eval/tasks/pile/README.md -> build/lib/lm_eval/tasks/pile 2024-03-18T14:20:33,648 copying lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:33,650 copying lm_eval/tasks/hendrycks_ethics/README.md -> build/lib/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:33,653 copying lm_eval/tasks/sciq/README.md -> build/lib/lm_eval/tasks/sciq 2024-03-18T14:20:33,655 copying lm_eval/tasks/openbookqa/README.md -> build/lib/lm_eval/tasks/openbookqa 2024-03-18T14:20:33,659 copying lm_eval/tasks/logiqa2/README.md -> build/lib/lm_eval/tasks/logiqa2 2024-03-18T14:20:33,661 copying lm_eval/tasks/gpqa/README.md -> build/lib/lm_eval/tasks/gpqa 2024-03-18T14:20:33,665 copying lm_eval/tasks/gpqa/cot_zeroshot/_gpqa_cot_zeroshot_yaml -> build/lib/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:33,668 copying lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml -> build/lib/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:33,673 copying lm_eval/tasks/gpqa/n_shot/_gpqa_n_shot_yaml -> build/lib/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:33,676 copying lm_eval/tasks/gpqa/zeroshot/_gpqa_zeroshot_yaml -> build/lib/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:33,680 copying lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml -> build/lib/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:33,682 copying lm_eval/tasks/ceval/_default_ceval_yaml -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:33,684 copying lm_eval/tasks/ceval/README.md -> build/lib/lm_eval/tasks/ceval 2024-03-18T14:20:33,687 copying lm_eval/tasks/kormedmcqa/README.md -> build/lib/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:33,689 copying lm_eval/tasks/arithmetic/README.md -> build/lib/lm_eval/tasks/arithmetic 2024-03-18T14:20:33,692 copying lm_eval/tasks/wmt2016/README.md -> build/lib/lm_eval/tasks/wmt2016 2024-03-18T14:20:33,694 copying lm_eval/tasks/lambada_multilingual/README.md -> build/lib/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:33,696 copying lm_eval/tasks/squadv2/README.md -> build/lib/lm_eval/tasks/squadv2 2024-03-18T14:20:33,700 copying lm_eval/tasks/eq_bench/README.md -> build/lib/lm_eval/tasks/eq_bench 2024-03-18T14:20:33,702 copying lm_eval/tasks/prost/README.md -> build/lib/lm_eval/tasks/prost 2024-03-18T14:20:33,705 copying lm_eval/tasks/coqa/README.md -> build/lib/lm_eval/tasks/coqa 2024-03-18T14:20:33,709 copying lm_eval/tasks/minerva_math/README.md -> build/lib/lm_eval/tasks/minerva_math 2024-03-18T14:20:33,712 copying lm_eval/tasks/crows_pairs/README.md -> build/lib/lm_eval/tasks/crows_pairs 2024-03-18T14:20:33,715 copying lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:33,717 copying lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:33,720 copying lm_eval/tasks/mmlu/default/_default_template_yaml -> build/lib/lm_eval/tasks/mmlu/default 2024-03-18T14:20:33,722 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:33,724 copying lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json -> build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:33,732 copying lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml -> build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:33,735 copying lm_eval/tasks/logiqa/README.md -> build/lib/lm_eval/tasks/logiqa 2024-03-18T14:20:33,737 copying lm_eval/tasks/polemo2/README.md -> build/lib/lm_eval/tasks/polemo2 2024-03-18T14:20:33,741 copying lm_eval/tasks/belebele/README.md -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,743 copying lm_eval/tasks/belebele/_default_template_yaml -> build/lib/lm_eval/tasks/belebele 2024-03-18T14:20:33,745 copying lm_eval/tasks/xstorycloze/README.md -> build/lib/lm_eval/tasks/xstorycloze 2024-03-18T14:20:33,748 copying lm_eval/tasks/headqa/README.md -> build/lib/lm_eval/tasks/headqa 2024-03-18T14:20:33,751 copying lm_eval/tasks/drop/README.md -> build/lib/lm_eval/tasks/drop 2024-03-18T14:20:33,754 copying lm_eval/tasks/hellaswag/README.md -> build/lib/lm_eval/tasks/hellaswag 2024-03-18T14:20:33,756 copying lm_eval/tasks/csatqa/_default_csatqa_yaml -> build/lib/lm_eval/tasks/csatqa 2024-03-18T14:20:34,754 installing to build/bdist.linux-armv7l/wheel 2024-03-18T14:20:34,754 running install 2024-03-18T14:20:34,778 running install_lib 2024-03-18T14:20:34,783 creating build/bdist.linux-armv7l 2024-03-18T14:20:34,783 creating build/bdist.linux-armv7l/wheel 2024-03-18T14:20:34,785 creating build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:34,786 copying build/lib/lm_eval/evaluator.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:34,791 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks 2024-03-18T14:20:34,792 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-03-18T14:20:34,793 copying build/lib/lm_eval/tasks/mathqa/mathqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-03-18T14:20:34,795 copying build/lib/lm_eval/tasks/mathqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-03-18T14:20:34,797 copying build/lib/lm_eval/tasks/mathqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mathqa 2024-03-18T14:20:34,799 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,800 copying build/lib/lm_eval/tasks/paws-x/paws_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,802 copying build/lib/lm_eval/tasks/paws-x/paws_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,804 copying build/lib/lm_eval/tasks/paws-x/_generate_config.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,806 copying build/lib/lm_eval/tasks/paws-x/paws_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,808 copying build/lib/lm_eval/tasks/paws-x/paws_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,810 copying build/lib/lm_eval/tasks/paws-x/paws_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,812 copying build/lib/lm_eval/tasks/paws-x/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,814 copying build/lib/lm_eval/tasks/paws-x/paws_ko.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,816 copying build/lib/lm_eval/tasks/paws-x/pawsx_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,818 copying build/lib/lm_eval/tasks/paws-x/paws_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/paws-x 2024-03-18T14:20:34,820 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-03-18T14:20:34,821 copying build/lib/lm_eval/tasks/mutual/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-03-18T14:20:34,823 copying build/lib/lm_eval/tasks/mutual/multual_plus.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-03-18T14:20:34,824 copying build/lib/lm_eval/tasks/mutual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-03-18T14:20:34,826 copying build/lib/lm_eval/tasks/mutual/mutual.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mutual 2024-03-18T14:20:34,828 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-03-18T14:20:34,829 copying build/lib/lm_eval/tasks/swag/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-03-18T14:20:34,831 copying build/lib/lm_eval/tasks/swag/swag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/swag 2024-03-18T14:20:34,834 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,834 copying build/lib/lm_eval/tasks/qasper/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,836 copying build/lib/lm_eval/tasks/qasper/freeform.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,838 copying build/lib/lm_eval/tasks/qasper/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,840 copying build/lib/lm_eval/tasks/qasper/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,842 copying build/lib/lm_eval/tasks/qasper/bool.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qasper 2024-03-18T14:20:34,844 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:34,845 copying build/lib/lm_eval/tasks/realtoxicityprompts/metric.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:34,847 copying build/lib/lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/realtoxicityprompts 2024-03-18T14:20:34,850 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,851 copying build/lib/lm_eval/tasks/gsm8k/gsm8k.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,852 copying build/lib/lm_eval/tasks/gsm8k/gsm8k-cot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,854 copying build/lib/lm_eval/tasks/gsm8k/gsm8k-cot-zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,856 copying build/lib/lm_eval/tasks/gsm8k/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,858 copying build/lib/lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gsm8k 2024-03-18T14:20:34,860 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu 2024-03-18T14:20:34,863 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,864 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_maritime_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,866 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_railway_and_automotive_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,868 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_political_science_and_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,869 copying build/lib/lm_eval/tasks/kmmlu/direct/_direct_kmmlu_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,871 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_refrigerating_machinery.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,873 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_taxation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,875 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_health.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,877 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_construction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,878 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_nondestructive_testing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,880 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,882 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,884 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,885 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_food_processing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,887 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_social_welfare.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,889 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_ecology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,890 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,892 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,894 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_materials_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,896 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_patent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,897 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_real_estate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,899 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_telecommunications_and_wireless_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,901 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_fashion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,903 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_aviation_engineering_and_maintenance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,905 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_information_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,907 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_environmental_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,908 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,910 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,912 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,914 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_machine_design_and_manufacturing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,916 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_energy_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,918 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,919 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,921 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,923 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_korean_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,924 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_public_safety.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,926 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_industrial_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,928 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_mechanical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,930 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_interior_architecture_and_design.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,932 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,934 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_criminal_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,936 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_agricultural_sciences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,938 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_electronics_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,940 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,942 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_geomatics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,944 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_civil_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,946 copying build/lib/lm_eval/tasks/kmmlu/direct/kmmlu_direct_gas_technology_and_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct 2024-03-18T14:20:34,948 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,950 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,951 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,953 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_social_welfare.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,955 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_taxation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,957 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_fashion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,960 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_geomatics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,962 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_mechanical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,964 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_civil_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,966 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,967 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_telecommunications_and_wireless_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,969 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_public_safety.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,971 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_criminal_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,973 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_materials_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,975 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_food_processing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,977 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_health.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,979 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_real_estate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,981 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_maritime_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,982 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,984 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,986 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_korean_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,987 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_nondestructive_testing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,989 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_gas_technology_and_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,991 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,993 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,995 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_energy_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,997 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:34,999 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_ecology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,000 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,002 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,004 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electronics_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,006 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_construction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,008 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_refrigerating_machinery.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,009 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/_direct_hard_kmmlu_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,011 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_patent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,013 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_information_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,014 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_environmental_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,016 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_interior_architecture_and_design.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,018 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_railway_and_automotive_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,020 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_aviation_engineering_and_maintenance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,022 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_industrial_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,024 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_machine_design_and_manufacturing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,026 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_political_science_and_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,028 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,030 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,032 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_agricultural_sciences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,034 copying build/lib/lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/direct_hard 2024-03-18T14:20:35,037 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,038 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_materials_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,041 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,043 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_patent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,046 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_construction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,049 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,051 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_public_safety.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,054 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_food_processing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,056 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_social_welfare.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,059 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,062 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,065 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_mechanical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,067 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,070 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_information_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,072 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_geomatics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,075 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/_cot_kmmlu_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,077 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_refrigerating_machinery.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,079 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,082 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_aviation_engineering_and_maintenance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,084 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_korean_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,087 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,089 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_maritime_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,092 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_energy_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,094 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,097 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,100 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_machine_design_and_manufacturing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,103 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,105 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_criminal_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,108 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_nondestructive_testing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,111 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_gas_technology_and_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,113 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_telecommunications_and_wireless_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,116 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_agricultural_sciences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,118 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_health.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,120 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_fashion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,123 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_political_science_and_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,125 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_environmental_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,133 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_ecology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,135 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_taxation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,138 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,140 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_railway_and_automotive_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,143 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_real_estate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,145 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,147 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_interior_architecture_and_design.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,150 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,152 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_civil_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,155 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_industrial_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,157 copying build/lib/lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electronics_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/cot_hard 2024-03-18T14:20:35,160 copying build/lib/lm_eval/tasks/kmmlu/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu 2024-03-18T14:20:35,163 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,164 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_ecology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,167 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,168 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,171 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_geomatics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,173 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,174 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_health.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,176 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,178 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_fashion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,180 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,182 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,183 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_energy_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,185 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_real_estate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,187 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_information_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,189 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_social_welfare.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,190 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_electronics_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,192 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_railway_and_automotive_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,194 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_food_processing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,196 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,198 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,200 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_civil_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,202 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,204 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_refrigerating_machinery.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,205 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_criminal_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,207 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_construction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,209 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_political_science_and_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,211 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_interior_architecture_and_design.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,213 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_machine_design_and_manufacturing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,215 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,217 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,219 copying build/lib/lm_eval/tasks/kmmlu/hard/_hard_kmmlu_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,221 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_agricultural_sciences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,223 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_environmental_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,224 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_maritime_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,226 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_gas_technology_and_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,228 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_patent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,230 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,232 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_aviation_engineering_and_maintenance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,234 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_mechanical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,236 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_telecommunications_and_wireless_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,238 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_industrial_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,239 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_public_safety.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,241 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_korean_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,243 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,245 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_nondestructive_testing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,247 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_materials_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,249 copying build/lib/lm_eval/tasks/kmmlu/hard/kmmlu_hard_taxation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kmmlu/hard 2024-03-18T14:20:35,251 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-03-18T14:20:35,252 copying build/lib/lm_eval/tasks/race/preprocess_race.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-03-18T14:20:35,254 copying build/lib/lm_eval/tasks/race/race.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-03-18T14:20:35,256 copying build/lib/lm_eval/tasks/race/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/race 2024-03-18T14:20:35,258 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-03-18T14:20:35,259 copying build/lib/lm_eval/tasks/nq_open/nq_open.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-03-18T14:20:35,262 copying build/lib/lm_eval/tasks/nq_open/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/nq_open 2024-03-18T14:20:35,264 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,265 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,267 copying build/lib/lm_eval/tasks/truthfulqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,269 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,271 copying build/lib/lm_eval/tasks/truthfulqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,274 copying build/lib/lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/truthfulqa 2024-03-18T14:20:35,276 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,277 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,279 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,281 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,283 copying build/lib/lm_eval/tasks/xwinograd/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,284 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,286 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,288 copying build/lib/lm_eval/tasks/xwinograd/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,290 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_jp.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,292 copying build/lib/lm_eval/tasks/xwinograd/xwinograd_pt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xwinograd 2024-03-18T14:20:35,294 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/medqa 2024-03-18T14:20:35,295 copying build/lib/lm_eval/tasks/medqa/medqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/medqa 2024-03-18T14:20:35,297 copying build/lib/lm_eval/tasks/medqa/preprocess_medqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/medqa 2024-03-18T14:20:35,299 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-03-18T14:20:35,300 copying build/lib/lm_eval/tasks/wikitext/preprocess_wikitext.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-03-18T14:20:35,302 copying build/lib/lm_eval/tasks/wikitext/wikitext.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-03-18T14:20:35,304 copying build/lib/lm_eval/tasks/wikitext/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wikitext 2024-03-18T14:20:35,307 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue 2024-03-18T14:20:35,308 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-03-18T14:20:35,310 copying build/lib/lm_eval/tasks/super_glue/record/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-03-18T14:20:35,311 copying build/lib/lm_eval/tasks/super_glue/record/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-03-18T14:20:35,314 copying build/lib/lm_eval/tasks/super_glue/record/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-03-18T14:20:35,315 copying build/lib/lm_eval/tasks/super_glue/record/util.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/record 2024-03-18T14:20:35,318 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:35,319 copying build/lib/lm_eval/tasks/super_glue/copa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:35,321 copying build/lib/lm_eval/tasks/super_glue/copa/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:35,323 copying build/lib/lm_eval/tasks/super_glue/copa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/copa 2024-03-18T14:20:35,325 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:35,326 copying build/lib/lm_eval/tasks/super_glue/rte/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:35,328 copying build/lib/lm_eval/tasks/super_glue/rte/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/rte 2024-03-18T14:20:35,331 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:35,332 copying build/lib/lm_eval/tasks/super_glue/cb/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:35,334 copying build/lib/lm_eval/tasks/super_glue/cb/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:35,335 copying build/lib/lm_eval/tasks/super_glue/cb/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:35,337 copying build/lib/lm_eval/tasks/super_glue/cb/aggregate.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/cb 2024-03-18T14:20:35,339 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:35,340 copying build/lib/lm_eval/tasks/super_glue/wsc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:35,342 copying build/lib/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:35,344 copying build/lib/lm_eval/tasks/super_glue/wsc/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:35,346 copying build/lib/lm_eval/tasks/super_glue/wsc/preprocess_wsc.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wsc 2024-03-18T14:20:35,348 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:35,349 copying build/lib/lm_eval/tasks/super_glue/boolq/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:35,351 copying build/lib/lm_eval/tasks/super_glue/boolq/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:35,353 copying build/lib/lm_eval/tasks/super_glue/boolq/seq2seq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/boolq 2024-03-18T14:20:35,355 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:35,356 copying build/lib/lm_eval/tasks/super_glue/multirc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:35,358 copying build/lib/lm_eval/tasks/super_glue/multirc/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:35,360 copying build/lib/lm_eval/tasks/super_glue/multirc/t5_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/multirc 2024-03-18T14:20:35,362 copying build/lib/lm_eval/tasks/super_glue/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue 2024-03-18T14:20:35,364 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:35,365 copying build/lib/lm_eval/tasks/super_glue/wic/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:35,367 copying build/lib/lm_eval/tasks/super_glue/wic/t5-prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/super_glue/wic 2024-03-18T14:20:35,370 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,371 copying build/lib/lm_eval/tasks/wmdp/wmdp_cyber.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,373 copying build/lib/lm_eval/tasks/wmdp/wmdp_bio.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,375 copying build/lib/lm_eval/tasks/wmdp/wmdp_chem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,376 copying build/lib/lm_eval/tasks/wmdp/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,378 copying build/lib/lm_eval/tasks/wmdp/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmdp 2024-03-18T14:20:35,381 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,382 copying build/lib/lm_eval/tasks/haerae/haerae_gk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,384 copying build/lib/lm_eval/tasks/haerae/haerae_lw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,386 copying build/lib/lm_eval/tasks/haerae/haerae_sn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,388 copying build/lib/lm_eval/tasks/haerae/_default_haerae_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,389 copying build/lib/lm_eval/tasks/haerae/haerae_rw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,391 copying build/lib/lm_eval/tasks/haerae/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,393 copying build/lib/lm_eval/tasks/haerae/haerae_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/haerae 2024-03-18T14:20:35,397 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,398 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,400 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,402 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,403 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,405 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,407 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,408 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,411 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,412 copying build/lib/lm_eval/tasks/cmmlu/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,415 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,417 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,420 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,422 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,424 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,426 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,428 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,431 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,433 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,435 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,438 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,440 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,443 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,445 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,447 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,449 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,452 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,454 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,456 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,458 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,460 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,462 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,465 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,467 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,469 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,471 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,473 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,475 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,477 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,480 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,482 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,484 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,487 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,489 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,492 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,495 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,497 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,499 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,501 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,504 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,506 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,509 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,511 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,513 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,515 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,517 copying build/lib/lm_eval/tasks/cmmlu/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,519 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,521 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,522 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,524 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,526 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,528 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,530 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,531 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,533 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,535 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,537 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,539 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,541 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,543 copying build/lib/lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,544 copying build/lib/lm_eval/tasks/cmmlu/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/cmmlu 2024-03-18T14:20:35,547 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi 2024-03-18T14:20:35,549 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,550 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,552 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,554 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,556 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,558 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,560 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,561 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,563 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,565 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,567 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,568 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,570 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,572 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,574 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,576 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,578 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,580 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,583 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,586 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,590 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,593 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,595 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,598 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,600 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,603 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc1_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,606 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,609 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,611 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,613 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,615 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,617 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,619 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,621 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,623 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc2_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,625 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,627 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,629 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,630 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,632 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,634 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,635 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,637 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,639 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,641 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,643 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,644 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,647 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,649 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,651 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,653 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,655 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,656 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,658 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,660 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,662 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,664 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,666 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,669 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,670 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,673 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,675 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,677 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,679 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,682 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,684 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,686 copying build/lib/lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/truthfulqa_multilingual 2024-03-18T14:20:35,689 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,690 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ta.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,692 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,695 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,696 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,698 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_kn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,700 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_eu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,701 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ca.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,703 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_gu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,705 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_pt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,706 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,708 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_nl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,710 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,712 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_uk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,714 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,715 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,717 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_mr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,719 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ml.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,721 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,722 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,724 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ne.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,726 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,728 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/_hellaswag_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,730 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,732 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ro.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,734 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,736 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,738 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sv.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,739 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,741 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,743 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,744 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,746 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,748 copying build/lib/lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/hellaswag_multilingual 2024-03-18T14:20:35,751 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,752 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ml.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,754 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,755 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,757 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,759 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ca.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,761 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,763 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_is.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,764 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_pt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,766 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,768 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ro.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,770 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,772 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_uk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,773 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,775 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_gu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,777 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,779 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,781 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,783 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,784 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,786 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/_default_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,788 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_kn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,789 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ne.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,791 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ta.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,792 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,794 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,796 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sv.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,798 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,800 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,802 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,804 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,805 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,807 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,809 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_mr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,810 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nb.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,812 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_eu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,814 copying build/lib/lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/mmlu_multilingual 2024-03-18T14:20:35,817 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,818 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_kn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,820 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ca.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,822 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ml.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,823 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_uk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,825 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,827 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_nl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,828 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_hu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,830 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,831 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,833 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,835 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ne.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,837 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,838 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,840 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ta.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,842 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,843 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ro.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,845 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,847 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,849 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_sr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,851 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,853 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,854 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_sv.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,856 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,858 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_sk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,860 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,861 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,863 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_hy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,865 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_gu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,867 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_pt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,868 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,870 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/_arc_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,872 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_mr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,874 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_eu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,876 copying build/lib/lm_eval/tasks/okapi/arc_multilingual/arc_hr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/okapi/arc_multilingual 2024-03-18T14:20:35,878 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,879 copying build/lib/lm_eval/tasks/translation/wmt16_en-de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,881 copying build/lib/lm_eval/tasks/translation/wmt14_en-fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,883 copying build/lib/lm_eval/tasks/translation/wmt_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,885 copying build/lib/lm_eval/tasks/translation/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,887 copying build/lib/lm_eval/tasks/translation/wmt14_fr-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,889 copying build/lib/lm_eval/tasks/translation/wmt16_ro-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,890 copying build/lib/lm_eval/tasks/translation/wmt16_en-ro.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,892 copying build/lib/lm_eval/tasks/translation/iwslt2017_ar-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,894 copying build/lib/lm_eval/tasks/translation/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,896 copying build/lib/lm_eval/tasks/translation/iwslt2017_en-ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,898 copying build/lib/lm_eval/tasks/translation/wmt16_de-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/translation 2024-03-18T14:20:35,901 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,902 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,904 copying build/lib/lm_eval/tasks/ammlu/ammlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,905 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,907 copying build/lib/lm_eval/tasks/ammlu/ammlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,909 copying build/lib/lm_eval/tasks/ammlu/ammlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,911 copying build/lib/lm_eval/tasks/ammlu/ammlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,913 copying build/lib/lm_eval/tasks/ammlu/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,915 copying build/lib/lm_eval/tasks/ammlu/ammlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,917 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,919 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,920 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,922 copying build/lib/lm_eval/tasks/ammlu/ammlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,924 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,925 copying build/lib/lm_eval/tasks/ammlu/ammlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,927 copying build/lib/lm_eval/tasks/ammlu/ammlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,929 copying build/lib/lm_eval/tasks/ammlu/ammlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,931 copying build/lib/lm_eval/tasks/ammlu/ammlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,932 copying build/lib/lm_eval/tasks/ammlu/ammlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,934 copying build/lib/lm_eval/tasks/ammlu/ammlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,936 copying build/lib/lm_eval/tasks/ammlu/ammlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,938 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,940 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,942 copying build/lib/lm_eval/tasks/ammlu/ammlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,943 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,945 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,947 copying build/lib/lm_eval/tasks/ammlu/ammlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,948 copying build/lib/lm_eval/tasks/ammlu/ammlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,950 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,952 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,954 copying build/lib/lm_eval/tasks/ammlu/ammlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,955 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,957 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,959 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,960 copying build/lib/lm_eval/tasks/ammlu/ammlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,962 copying build/lib/lm_eval/tasks/ammlu/ammlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,964 copying build/lib/lm_eval/tasks/ammlu/ammlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,966 copying build/lib/lm_eval/tasks/ammlu/ammlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,968 copying build/lib/lm_eval/tasks/ammlu/ammlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,970 copying build/lib/lm_eval/tasks/ammlu/ammlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,971 copying build/lib/lm_eval/tasks/ammlu/ammlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,973 copying build/lib/lm_eval/tasks/ammlu/ammlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,974 copying build/lib/lm_eval/tasks/ammlu/ammlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,976 copying build/lib/lm_eval/tasks/ammlu/ammlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,978 copying build/lib/lm_eval/tasks/ammlu/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,980 copying build/lib/lm_eval/tasks/ammlu/ammlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,982 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,983 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,985 copying build/lib/lm_eval/tasks/ammlu/ammlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,987 copying build/lib/lm_eval/tasks/ammlu/ammlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,989 copying build/lib/lm_eval/tasks/ammlu/ammlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,990 copying build/lib/lm_eval/tasks/ammlu/ammlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,992 copying build/lib/lm_eval/tasks/ammlu/ammlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,994 copying build/lib/lm_eval/tasks/ammlu/ammlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,996 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,998 copying build/lib/lm_eval/tasks/ammlu/ammlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:35,999 copying build/lib/lm_eval/tasks/ammlu/ammlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:36,001 copying build/lib/lm_eval/tasks/ammlu/ammlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:36,003 copying build/lib/lm_eval/tasks/ammlu/ammlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:36,005 copying build/lib/lm_eval/tasks/ammlu/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:36,006 copying build/lib/lm_eval/tasks/ammlu/ammlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ammlu 2024-03-18T14:20:36,009 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-03-18T14:20:36,010 copying build/lib/lm_eval/tasks/storycloze/storycloze_2016.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-03-18T14:20:36,012 copying build/lib/lm_eval/tasks/storycloze/storycloze_2018.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-03-18T14:20:36,014 copying build/lib/lm_eval/tasks/storycloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/storycloze 2024-03-18T14:20:36,016 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/fld 2024-03-18T14:20:36,017 copying build/lib/lm_eval/tasks/fld/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/fld 2024-03-18T14:20:36,019 copying build/lib/lm_eval/tasks/fld/fld_star.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/fld 2024-03-18T14:20:36,021 copying build/lib/lm_eval/tasks/fld/fld_default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/fld 2024-03-18T14:20:36,023 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-03-18T14:20:36,024 copying build/lib/lm_eval/tasks/toxigen/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-03-18T14:20:36,026 copying build/lib/lm_eval/tasks/toxigen/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-03-18T14:20:36,028 copying build/lib/lm_eval/tasks/toxigen/toxigen.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/toxigen 2024-03-18T14:20:36,030 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals 2024-03-18T14:20:36,034 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,035 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,037 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,038 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,040 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,042 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,044 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,045 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,047 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,049 copying build/lib/lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,051 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,053 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,054 copying build/lib/lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,056 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,058 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,059 copying build/lib/lm_eval/tasks/model_written_evals/persona/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,061 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,063 copying build/lib/lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,064 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,066 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,068 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,070 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,072 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,073 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,075 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,077 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,079 copying build/lib/lm_eval/tasks/model_written_evals/persona/agreeableness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,081 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,082 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,085 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,087 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,089 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,091 copying build/lib/lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,093 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,095 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,097 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,098 copying build/lib/lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,100 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,102 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,104 copying build/lib/lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,105 copying build/lib/lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,107 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,109 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,111 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,113 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,115 copying build/lib/lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,117 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,118 copying build/lib/lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,120 copying build/lib/lm_eval/tasks/model_written_evals/persona/psychopathy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,122 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,124 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,126 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,128 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,130 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,132 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,133 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,135 copying build/lib/lm_eval/tasks/model_written_evals/persona/self-replication.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,137 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,138 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,140 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,142 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,144 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,146 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,147 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,149 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,151 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,153 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,155 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,157 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,158 copying build/lib/lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,160 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,162 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,164 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,166 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,168 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,170 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,171 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,173 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,175 copying build/lib/lm_eval/tasks/model_written_evals/persona/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,176 copying build/lib/lm_eval/tasks/model_written_evals/persona/openness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,178 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,180 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,182 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,183 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,185 copying build/lib/lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,187 copying build/lib/lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,189 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,191 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,193 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,195 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,197 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,199 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,200 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,202 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,204 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,206 copying build/lib/lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,208 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,210 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,212 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,214 copying build/lib/lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,216 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,218 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,221 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,223 copying build/lib/lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,225 copying build/lib/lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,227 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,230 copying build/lib/lm_eval/tasks/model_written_evals/persona/narcissism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,232 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,235 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,237 copying build/lib/lm_eval/tasks/model_written_evals/persona/risk-averse.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,240 copying build/lib/lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,242 copying build/lib/lm_eval/tasks/model_written_evals/persona/neuroticism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,245 copying build/lib/lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,247 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,250 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,252 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,255 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,257 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,261 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,263 copying build/lib/lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,265 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,268 copying build/lib/lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,270 copying build/lib/lm_eval/tasks/model_written_evals/persona/extraversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,272 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,275 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,277 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,280 copying build/lib/lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,282 copying build/lib/lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,285 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,288 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,290 copying build/lib/lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,293 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,296 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,298 copying build/lib/lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,302 copying build/lib/lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,305 copying build/lib/lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,308 copying build/lib/lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,310 copying build/lib/lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/persona 2024-03-18T14:20:36,313 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/winogenerated 2024-03-18T14:20:36,315 copying build/lib/lm_eval/tasks/model_written_evals/winogenerated/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/winogenerated 2024-03-18T14:20:36,318 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:36,319 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:36,322 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:36,324 copying build/lib/lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/sycophancy 2024-03-18T14:20:36,328 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,329 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,331 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,333 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,335 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,338 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,340 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,343 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,345 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,347 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,349 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,352 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,354 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,356 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,359 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,361 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,364 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,366 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,368 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,371 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,373 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,375 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,377 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,380 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,382 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,384 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,387 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,389 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,391 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,394 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,396 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,398 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,400 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,403 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,405 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,408 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,410 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,412 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,415 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,417 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,419 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,421 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,422 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,424 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,426 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,428 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,430 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,431 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,433 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,435 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,436 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,438 copying build/lib/lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/model_written_evals/advanced_ai_risk 2024-03-18T14:20:36,441 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,442 copying build/lib/lm_eval/tasks/agieval/gaokao-biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,444 copying build/lib/lm_eval/tasks/agieval/logiqa-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,446 copying build/lib/lm_eval/tasks/agieval/logiqa-zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,447 copying build/lib/lm_eval/tasks/agieval/sat-en-without-passage.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,449 copying build/lib/lm_eval/tasks/agieval/gaokao-mathcloze.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,451 copying build/lib/lm_eval/tasks/agieval/gaokao-physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,452 copying build/lib/lm_eval/tasks/agieval/jec-qa-kd.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,454 copying build/lib/lm_eval/tasks/agieval/aqua-rat.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,456 copying build/lib/lm_eval/tasks/agieval/lsat-lr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,457 copying build/lib/lm_eval/tasks/agieval/lsat-rc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,459 copying build/lib/lm_eval/tasks/agieval/gaokao-english.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,461 copying build/lib/lm_eval/tasks/agieval/math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,463 copying build/lib/lm_eval/tasks/agieval/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,465 copying build/lib/lm_eval/tasks/agieval/sat-math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,466 copying build/lib/lm_eval/tasks/agieval/gaokao-mathqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,468 copying build/lib/lm_eval/tasks/agieval/gaokao-geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,470 copying build/lib/lm_eval/tasks/agieval/gaokao-chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,472 copying build/lib/lm_eval/tasks/agieval/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,474 copying build/lib/lm_eval/tasks/agieval/lsat-ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,476 copying build/lib/lm_eval/tasks/agieval/jec-qa-ca.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,477 copying build/lib/lm_eval/tasks/agieval/sat-en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,479 copying build/lib/lm_eval/tasks/agieval/gaokao-chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,481 copying build/lib/lm_eval/tasks/agieval/gaokao-history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/agieval 2024-03-18T14:20:36,483 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-03-18T14:20:36,484 copying build/lib/lm_eval/tasks/wsc273/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-03-18T14:20:36,486 copying build/lib/lm_eval/tasks/wsc273/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-03-18T14:20:36,488 copying build/lib/lm_eval/tasks/wsc273/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wsc273 2024-03-18T14:20:36,490 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-03-18T14:20:36,491 copying build/lib/lm_eval/tasks/arc/arc_challenge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-03-18T14:20:36,493 copying build/lib/lm_eval/tasks/arc/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-03-18T14:20:36,495 copying build/lib/lm_eval/tasks/arc/arc_easy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arc 2024-03-18T14:20:36,497 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,498 copying build/lib/lm_eval/tasks/xcopa/default_ht.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,500 copying build/lib/lm_eval/tasks/xcopa/default_et.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,502 copying build/lib/lm_eval/tasks/xcopa/default_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,504 copying build/lib/lm_eval/tasks/xcopa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,506 copying build/lib/lm_eval/tasks/xcopa/default_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,507 copying build/lib/lm_eval/tasks/xcopa/default_ta.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,509 copying build/lib/lm_eval/tasks/xcopa/default_tr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,511 copying build/lib/lm_eval/tasks/xcopa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,513 copying build/lib/lm_eval/tasks/xcopa/default_qu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,515 copying build/lib/lm_eval/tasks/xcopa/default_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,517 copying build/lib/lm_eval/tasks/xcopa/default_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,519 copying build/lib/lm_eval/tasks/xcopa/default_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,520 copying build/lib/lm_eval/tasks/xcopa/default_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xcopa 2024-03-18T14:20:36,523 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-03-18T14:20:36,523 copying build/lib/lm_eval/tasks/triviaqa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-03-18T14:20:36,526 copying build/lib/lm_eval/tasks/triviaqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/triviaqa 2024-03-18T14:20:36,528 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-03-18T14:20:36,529 copying build/lib/lm_eval/tasks/benchmarks/pythia.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-03-18T14:20:36,531 copying build/lib/lm_eval/tasks/benchmarks/t0_eval.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-03-18T14:20:36,533 copying build/lib/lm_eval/tasks/benchmarks/openllm.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-03-18T14:20:36,534 copying build/lib/lm_eval/tasks/benchmarks/minerva_math.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks 2024-03-18T14:20:36,537 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:36,538 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:36,541 copying build/lib/lm_eval/tasks/benchmarks/flan/_held_in_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:36,543 copying build/lib/lm_eval/tasks/benchmarks/flan/flan_held_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/flan 2024-03-18T14:20:36,545 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:36,546 copying build/lib/lm_eval/tasks/benchmarks/multimedqa/multimedqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:36,548 copying build/lib/lm_eval/tasks/benchmarks/multimedqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/benchmarks/multimedqa 2024-03-18T14:20:36,551 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,552 copying build/lib/lm_eval/tasks/xnli/xnli_vi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,554 copying build/lib/lm_eval/tasks/xnli/xnli_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,556 copying build/lib/lm_eval/tasks/xnli/xnli_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,557 copying build/lib/lm_eval/tasks/xnli/xnli_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,559 copying build/lib/lm_eval/tasks/xnli/xnli_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,561 copying build/lib/lm_eval/tasks/xnli/xnli_tr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,562 copying build/lib/lm_eval/tasks/xnli/xnli_common_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,564 copying build/lib/lm_eval/tasks/xnli/xnli_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,566 copying build/lib/lm_eval/tasks/xnli/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,568 copying build/lib/lm_eval/tasks/xnli/xnli_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,569 copying build/lib/lm_eval/tasks/xnli/xnli_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,571 copying build/lib/lm_eval/tasks/xnli/xnli_ur.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,573 copying build/lib/lm_eval/tasks/xnli/xnli_bg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,575 copying build/lib/lm_eval/tasks/xnli/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,576 copying build/lib/lm_eval/tasks/xnli/xnli_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,578 copying build/lib/lm_eval/tasks/xnli/xnli_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,580 copying build/lib/lm_eval/tasks/xnli/xnli_el.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,582 copying build/lib/lm_eval/tasks/xnli/xnli_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xnli 2024-03-18T14:20:36,584 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-03-18T14:20:36,585 copying build/lib/lm_eval/tasks/piqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-03-18T14:20:36,587 copying build/lib/lm_eval/tasks/piqa/piqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/piqa 2024-03-18T14:20:36,589 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,590 copying build/lib/lm_eval/tasks/french_bench/french_bench_vocab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,592 copying build/lib/lm_eval/tasks/french_bench/preprocess_wikitext.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,594 copying build/lib/lm_eval/tasks/french_bench/french_bench_boolqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,596 copying build/lib/lm_eval/tasks/french_bench/french_bench_multifquad.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,598 copying build/lib/lm_eval/tasks/french_bench/french_bench_wikitext_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,600 copying build/lib/lm_eval/tasks/french_bench/french_bench_fquadv2_bool.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,602 copying build/lib/lm_eval/tasks/french_bench/french_bench_xnli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,604 copying build/lib/lm_eval/tasks/french_bench/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,606 copying build/lib/lm_eval/tasks/french_bench/french_bench_reading_comp.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,608 copying build/lib/lm_eval/tasks/french_bench/french_bench_trivia.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,610 copying build/lib/lm_eval/tasks/french_bench/french_bench_fquadv2_genq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,612 copying build/lib/lm_eval/tasks/french_bench/french_bench_hellaswag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,614 copying build/lib/lm_eval/tasks/french_bench/french_bench_fquadv2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,615 copying build/lib/lm_eval/tasks/french_bench/french_bench_opus_perplexity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,617 copying build/lib/lm_eval/tasks/french_bench/french_bench_grammar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,619 copying build/lib/lm_eval/tasks/french_bench/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,621 copying build/lib/lm_eval/tasks/french_bench/french_bench_orangesum_abstract.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,623 copying build/lib/lm_eval/tasks/french_bench/french_bench_fquadv2_hasAns.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,625 copying build/lib/lm_eval/tasks/french_bench/french_bench_orangesum_title.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,626 copying build/lib/lm_eval/tasks/french_bench/french_bench_topic_based_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,628 copying build/lib/lm_eval/tasks/french_bench/french_bench_arc_challenge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,630 copying build/lib/lm_eval/tasks/french_bench/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/french_bench 2024-03-18T14:20:36,631 copying build/lib/lm_eval/tasks/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks 2024-03-18T14:20:36,635 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,636 copying build/lib/lm_eval/tasks/ifeval/ifeval.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,638 copying build/lib/lm_eval/tasks/ifeval/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,640 copying build/lib/lm_eval/tasks/ifeval/instructions.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,643 copying build/lib/lm_eval/tasks/ifeval/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,645 copying build/lib/lm_eval/tasks/ifeval/instructions_util.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,648 copying build/lib/lm_eval/tasks/ifeval/instructions_registry.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ifeval 2024-03-18T14:20:36,650 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,651 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2012.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,653 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2013.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,654 copying build/lib/lm_eval/tasks/qa4mre/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,656 copying build/lib/lm_eval/tasks/qa4mre/qa4mre_2011.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,658 copying build/lib/lm_eval/tasks/qa4mre/preprocess_qa4mre.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/qa4mre 2024-03-18T14:20:36,660 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,661 copying build/lib/lm_eval/tasks/aexams/aexams_Physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,663 copying build/lib/lm_eval/tasks/aexams/aexams_IslamicStudies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,665 copying build/lib/lm_eval/tasks/aexams/aexams_Science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,666 copying build/lib/lm_eval/tasks/aexams/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,668 copying build/lib/lm_eval/tasks/aexams/aexams_Biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,670 copying build/lib/lm_eval/tasks/aexams/aexams_Social.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,671 copying build/lib/lm_eval/tasks/aexams/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/aexams 2024-03-18T14:20:36,673 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue 2024-03-18T14:20:36,674 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qqp 2024-03-18T14:20:36,675 copying build/lib/lm_eval/tasks/glue/qqp/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qqp 2024-03-18T14:20:36,677 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mrpc 2024-03-18T14:20:36,678 copying build/lib/lm_eval/tasks/glue/mrpc/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mrpc 2024-03-18T14:20:36,680 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/rte 2024-03-18T14:20:36,681 copying build/lib/lm_eval/tasks/glue/rte/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/rte 2024-03-18T14:20:36,683 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/wnli 2024-03-18T14:20:36,684 copying build/lib/lm_eval/tasks/glue/wnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/wnli 2024-03-18T14:20:36,687 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/sst2 2024-03-18T14:20:36,687 copying build/lib/lm_eval/tasks/glue/sst2/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/sst2 2024-03-18T14:20:36,690 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-03-18T14:20:36,691 copying build/lib/lm_eval/tasks/glue/mnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-03-18T14:20:36,693 copying build/lib/lm_eval/tasks/glue/mnli/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-03-18T14:20:36,694 copying build/lib/lm_eval/tasks/glue/mnli/mismatch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/mnli 2024-03-18T14:20:36,697 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/cola 2024-03-18T14:20:36,698 copying build/lib/lm_eval/tasks/glue/cola/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/cola 2024-03-18T14:20:36,700 copying build/lib/lm_eval/tasks/glue/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue 2024-03-18T14:20:36,702 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qnli 2024-03-18T14:20:36,703 copying build/lib/lm_eval/tasks/glue/qnli/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/glue/qnli 2024-03-18T14:20:36,705 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-03-18T14:20:36,707 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,708 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,710 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,711 copying build/lib/lm_eval/tasks/mgsm/direct/direct_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,713 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,715 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,717 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,718 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,720 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,722 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,724 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,726 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,727 copying build/lib/lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/direct 2024-03-18T14:20:36,729 copying build/lib/lm_eval/tasks/mgsm/gen_yaml.sh -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-03-18T14:20:36,731 copying build/lib/lm_eval/tasks/mgsm/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-03-18T14:20:36,734 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,734 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,736 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,738 copying build/lib/lm_eval/tasks/mgsm/en_cot/cot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,740 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,742 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,743 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,745 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,747 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,748 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,750 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,752 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,754 copying build/lib/lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/en_cot 2024-03-18T14:20:36,756 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,756 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,758 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ja.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,760 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,762 copying build/lib/lm_eval/tasks/mgsm/native_cot/cot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,764 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_th.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,766 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,767 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,769 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,771 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,773 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,775 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,777 copying build/lib/lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_bn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm/native_cot 2024-03-18T14:20:36,779 copying build/lib/lm_eval/tasks/mgsm/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mgsm 2024-03-18T14:20:36,781 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:36,782 copying build/lib/lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:36,784 copying build/lib/lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:36,785 copying build/lib/lm_eval/tasks/lambada_cloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_cloze 2024-03-18T14:20:36,788 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:36,792 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,794 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,795 copying build/lib/lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,797 copying build/lib/lm_eval/tasks/bigbench/generate_until/cryptonite.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,799 copying build/lib/lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,801 copying build/lib/lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,803 copying build/lib/lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,805 copying build/lib/lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,807 copying build/lib/lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,809 copying build/lib/lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,810 copying build/lib/lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,812 copying build/lib/lm_eval/tasks/bigbench/generate_until/social_support.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,814 copying build/lib/lm_eval/tasks/bigbench/generate_until/social_iqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,816 copying build/lib/lm_eval/tasks/bigbench/generate_until/code_line_description.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,818 copying build/lib/lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,820 copying build/lib/lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,822 copying build/lib/lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,823 copying build/lib/lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,825 copying build/lib/lm_eval/tasks/bigbench/generate_until/codenames.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,826 copying build/lib/lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,828 copying build/lib/lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,830 copying build/lib/lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,831 copying build/lib/lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,833 copying build/lib/lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,834 copying build/lib/lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,836 copying build/lib/lm_eval/tasks/bigbench/generate_until/operators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,838 copying build/lib/lm_eval/tasks/bigbench/generate_until/misconceptions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,839 copying build/lib/lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,841 copying build/lib/lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,843 copying build/lib/lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,845 copying build/lib/lm_eval/tasks/bigbench/generate_until/fact_checker.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,847 copying build/lib/lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,849 copying build/lib/lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,850 copying build/lib/lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,852 copying build/lib/lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,854 copying build/lib/lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,856 copying build/lib/lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,858 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,860 copying build/lib/lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,861 copying build/lib/lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,863 copying build/lib/lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,865 copying build/lib/lm_eval/tasks/bigbench/generate_until/anachronisms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,867 copying build/lib/lm_eval/tasks/bigbench/generate_until/topical_chat.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,868 copying build/lib/lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,870 copying build/lib/lm_eval/tasks/bigbench/generate_until/list_functions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,872 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,874 copying build/lib/lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,875 copying build/lib/lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,877 copying build/lib/lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,879 copying build/lib/lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,881 copying build/lib/lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,883 copying build/lib/lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,884 copying build/lib/lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,886 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,888 copying build/lib/lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,889 copying build/lib/lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,891 copying build/lib/lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,893 copying build/lib/lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,895 copying build/lib/lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,897 copying build/lib/lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,899 copying build/lib/lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,901 copying build/lib/lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,902 copying build/lib/lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,904 copying build/lib/lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,905 copying build/lib/lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,907 copying build/lib/lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,908 copying build/lib/lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,910 copying build/lib/lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,912 copying build/lib/lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,914 copying build/lib/lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,915 copying build/lib/lm_eval/tasks/bigbench/generate_until/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,917 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,919 copying build/lib/lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,921 copying build/lib/lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,922 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_args.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,924 copying build/lib/lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,926 copying build/lib/lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,928 copying build/lib/lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,929 copying build/lib/lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,931 copying build/lib/lm_eval/tasks/bigbench/generate_until/color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,933 copying build/lib/lm_eval/tasks/bigbench/generate_until/gem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,934 copying build/lib/lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,936 copying build/lib/lm_eval/tasks/bigbench/generate_until/multiemo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,938 copying build/lib/lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,940 copying build/lib/lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,942 copying build/lib/lm_eval/tasks/bigbench/generate_until/kannada.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,943 copying build/lib/lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,945 copying build/lib/lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,947 copying build/lib/lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,948 copying build/lib/lm_eval/tasks/bigbench/generate_until/physics_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,950 copying build/lib/lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,952 copying build/lib/lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,953 copying build/lib/lm_eval/tasks/bigbench/generate_until/crass_ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,955 copying build/lib/lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,957 copying build/lib/lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,958 copying build/lib/lm_eval/tasks/bigbench/generate_until/strategyqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,960 copying build/lib/lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,962 copying build/lib/lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,964 copying build/lib/lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,965 copying build/lib/lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,967 copying build/lib/lm_eval/tasks/bigbench/generate_until/language_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,970 copying build/lib/lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,972 copying build/lib/lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,974 copying build/lib/lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,975 copying build/lib/lm_eval/tasks/bigbench/generate_until/irony_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,977 copying build/lib/lm_eval/tasks/bigbench/generate_until/rephrase.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,979 copying build/lib/lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,981 copying build/lib/lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,983 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,985 copying build/lib/lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,986 copying build/lib/lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,988 copying build/lib/lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,990 copying build/lib/lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,991 copying build/lib/lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,993 copying build/lib/lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,995 copying build/lib/lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,996 copying build/lib/lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:36,998 copying build/lib/lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,000 copying build/lib/lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,002 copying build/lib/lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,004 copying build/lib/lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,005 copying build/lib/lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,007 copying build/lib/lm_eval/tasks/bigbench/generate_until/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,009 copying build/lib/lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,010 copying build/lib/lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,012 copying build/lib/lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,014 copying build/lib/lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,016 copying build/lib/lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,017 copying build/lib/lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,019 copying build/lib/lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,021 copying build/lib/lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,022 copying build/lib/lm_eval/tasks/bigbench/generate_until/arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,024 copying build/lib/lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,026 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,027 copying build/lib/lm_eval/tasks/bigbench/generate_until/question_selection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,029 copying build/lib/lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,031 copying build/lib/lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,032 copying build/lib/lm_eval/tasks/bigbench/generate_until/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,034 copying build/lib/lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,036 copying build/lib/lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,037 copying build/lib/lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,039 copying build/lib/lm_eval/tasks/bigbench/generate_until/timedial.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,041 copying build/lib/lm_eval/tasks/bigbench/generate_until/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,042 copying build/lib/lm_eval/tasks/bigbench/generate_until/physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,044 copying build/lib/lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,046 copying build/lib/lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,048 copying build/lib/lm_eval/tasks/bigbench/generate_until/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,049 copying build/lib/lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,051 copying build/lib/lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,053 copying build/lib/lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,055 copying build/lib/lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,057 copying build/lib/lm_eval/tasks/bigbench/generate_until/tense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,059 copying build/lib/lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,060 copying build/lib/lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,062 copying build/lib/lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,064 copying build/lib/lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,065 copying build/lib/lm_eval/tasks/bigbench/generate_until/language_games.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,067 copying build/lib/lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,069 copying build/lib/lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,070 copying build/lib/lm_eval/tasks/bigbench/generate_until/implicatures.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,072 copying build/lib/lm_eval/tasks/bigbench/generate_until/winowhy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,074 copying build/lib/lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,076 copying build/lib/lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,078 copying build/lib/lm_eval/tasks/bigbench/generate_until/strange_stories.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,079 copying build/lib/lm_eval/tasks/bigbench/generate_until/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,082 copying build/lib/lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,084 copying build/lib/lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,085 copying build/lib/lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/generate_until 2024-03-18T14:20:37,090 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,091 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,093 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,095 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,097 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,099 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,100 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,102 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,104 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,106 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,107 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,109 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,111 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/social_support.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,112 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,114 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,116 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,118 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,120 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,122 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,124 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/codenames.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,126 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,129 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,131 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,133 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,135 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,138 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,140 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/operators.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,143 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,145 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,148 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,150 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,152 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,155 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,157 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,159 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,161 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,164 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,166 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,169 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,171 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,173 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,176 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,178 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,181 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,183 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,185 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,188 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,190 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,193 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,195 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,198 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,201 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,203 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,205 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,207 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,209 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,212 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,214 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,216 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,219 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,221 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,223 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,225 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,227 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,228 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,230 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,232 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,234 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,236 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,238 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,240 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,241 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,243 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,245 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,247 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,248 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,250 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,252 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,253 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,255 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,257 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,259 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gem.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,261 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,263 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,265 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,266 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,269 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/kannada.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,271 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,272 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,274 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,276 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,278 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,280 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,282 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,284 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,286 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,288 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,290 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,291 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,294 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,295 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,297 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,299 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,301 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,303 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,305 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,306 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,308 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,310 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,312 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,314 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,315 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,317 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,319 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,321 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,323 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,324 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,327 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,329 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,330 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,332 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,334 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,336 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,337 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,339 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,341 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,343 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,345 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,346 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,348 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,350 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,352 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,354 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,356 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,358 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,359 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,361 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,363 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,365 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,367 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,368 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,370 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,372 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/timedial.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,374 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,376 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,378 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,380 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,382 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,383 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,385 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,387 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,389 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,390 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/tense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,392 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,394 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,396 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,397 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,399 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/language_games.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,401 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,402 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,404 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,406 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,409 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,411 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,413 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,415 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,417 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,419 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,420 copying build/lib/lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench/multiple_choice 2024-03-18T14:20:37,422 copying build/lib/lm_eval/tasks/bigbench/multiple_choice_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:37,424 copying build/lib/lm_eval/tasks/bigbench/generate_tasks.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:37,426 copying build/lib/lm_eval/tasks/bigbench/push_bigbench_dataset.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:37,427 copying build/lib/lm_eval/tasks/bigbench/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:37,430 copying build/lib/lm_eval/tasks/bigbench/generate_until_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bigbench 2024-03-18T14:20:37,432 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,433 copying build/lib/lm_eval/tasks/unscramble/anagrams2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,435 copying build/lib/lm_eval/tasks/unscramble/anagrams1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,437 copying build/lib/lm_eval/tasks/unscramble/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,439 copying build/lib/lm_eval/tasks/unscramble/random_insertion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,441 copying build/lib/lm_eval/tasks/unscramble/reversed_words.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,443 copying build/lib/lm_eval/tasks/unscramble/cycle_letters.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/unscramble 2024-03-18T14:20:37,446 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-03-18T14:20:37,447 copying build/lib/lm_eval/tasks/lambada/lambada_standard.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-03-18T14:20:37,449 copying build/lib/lm_eval/tasks/lambada/lambada_openai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-03-18T14:20:37,451 copying build/lib/lm_eval/tasks/lambada/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada 2024-03-18T14:20:37,455 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,456 copying build/lib/lm_eval/tasks/blimp/intransitive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,458 copying build/lib/lm_eval/tasks/blimp/existential_there_subject_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,459 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,461 copying build/lib/lm_eval/tasks/blimp/passive_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,463 copying build/lib/lm_eval/tasks/blimp/npi_present_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,465 copying build/lib/lm_eval/tasks/blimp/principle_A_case_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,467 copying build/lib/lm_eval/tasks/blimp/inchoative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,469 copying build/lib/lm_eval/tasks/blimp/generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,471 copying build/lib/lm_eval/tasks/blimp/animate_subject_trans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,472 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,474 copying build/lib/lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,476 copying build/lib/lm_eval/tasks/blimp/drop_argument.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,478 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,480 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,482 copying build/lib/lm_eval/tasks/blimp/wh_questions_object_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,484 copying build/lib/lm_eval/tasks/blimp/left_branch_island_echo_question.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,485 copying build/lib/lm_eval/tasks/blimp/passive_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,487 copying build/lib/lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,489 copying build/lib/lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,490 copying build/lib/lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,492 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,494 copying build/lib/lm_eval/tasks/blimp/expletive_it_object_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,496 copying build/lib/lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,497 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,499 copying build/lib/lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,501 copying build/lib/lm_eval/tasks/blimp/superlative_quantifiers_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,503 copying build/lib/lm_eval/tasks/blimp/only_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,505 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,507 copying build/lib/lm_eval/tasks/blimp/superlative_quantifiers_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,509 copying build/lib/lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,511 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,512 copying build/lib/lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,514 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,516 copying build/lib/lm_eval/tasks/blimp/principle_A_domain_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,519 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,521 copying build/lib/lm_eval/tasks/blimp/tough_vs_raising_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,523 copying build/lib/lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,524 copying build/lib/lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,526 copying build/lib/lm_eval/tasks/blimp/sentential_subject_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,528 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,530 copying build/lib/lm_eval/tasks/blimp/_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,532 copying build/lib/lm_eval/tasks/blimp/transitive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,533 copying build/lib/lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,535 copying build/lib/lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,537 copying build/lib/lm_eval/tasks/blimp/anaphor_gender_agreement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,538 copying build/lib/lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,540 copying build/lib/lm_eval/tasks/blimp/animate_subject_passive.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,542 copying build/lib/lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,544 copying build/lib/lm_eval/tasks/blimp/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,546 copying build/lib/lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,547 copying build/lib/lm_eval/tasks/blimp/causative.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,549 copying build/lib/lm_eval/tasks/blimp/principle_A_case_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,551 copying build/lib/lm_eval/tasks/blimp/wh_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,553 copying build/lib/lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,555 copying build/lib/lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,556 copying build/lib/lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,558 copying build/lib/lm_eval/tasks/blimp/principle_A_reconstruction.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,560 copying build/lib/lm_eval/tasks/blimp/left_branch_island_simple_question.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,562 copying build/lib/lm_eval/tasks/blimp/anaphor_number_agreement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,564 copying build/lib/lm_eval/tasks/blimp/complex_NP_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,566 copying build/lib/lm_eval/tasks/blimp/tough_vs_raising_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,568 copying build/lib/lm_eval/tasks/blimp/existential_there_object_raising.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,570 copying build/lib/lm_eval/tasks/blimp/only_npi_scope.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,571 copying build/lib/lm_eval/tasks/blimp/npi_present_1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,573 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,575 copying build/lib/lm_eval/tasks/blimp/wh_questions_subject_gap.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,577 copying build/lib/lm_eval/tasks/blimp/principle_A_c_command.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,579 copying build/lib/lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,581 copying build/lib/lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,583 copying build/lib/lm_eval/tasks/blimp/adjunct_island.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/blimp 2024-03-18T14:20:37,585 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-03-18T14:20:37,586 copying build/lib/lm_eval/tasks/anli/anli_r1.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-03-18T14:20:37,588 copying build/lib/lm_eval/tasks/anli/anli_r2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-03-18T14:20:37,590 copying build/lib/lm_eval/tasks/anli/anli_r3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-03-18T14:20:37,592 copying build/lib/lm_eval/tasks/anli/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/anli 2024-03-18T14:20:37,595 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-03-18T14:20:37,596 copying build/lib/lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-03-18T14:20:37,598 copying build/lib/lm_eval/tasks/pubmedqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-03-18T14:20:37,600 copying build/lib/lm_eval/tasks/pubmedqa/pubmedqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pubmedqa 2024-03-18T14:20:37,602 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-03-18T14:20:37,603 copying build/lib/lm_eval/tasks/bbh/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-03-18T14:20:37,606 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,607 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,609 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,611 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,613 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,615 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,616 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,619 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,621 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,623 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,624 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,626 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,628 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,631 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,632 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,634 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,636 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,638 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,639 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,642 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,643 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,645 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,648 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,650 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,652 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,654 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,656 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,658 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,659 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,661 copying build/lib/lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_zeroshot 2024-03-18T14:20:37,665 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,666 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,668 copying build/lib/lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,670 copying build/lib/lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,672 copying build/lib/lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,674 copying build/lib/lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,676 copying build/lib/lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,678 copying build/lib/lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,680 copying build/lib/lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,681 copying build/lib/lm_eval/tasks/bbh/zeroshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,683 copying build/lib/lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,685 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,687 copying build/lib/lm_eval/tasks/bbh/zeroshot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,689 copying build/lib/lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,691 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,693 copying build/lib/lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,695 copying build/lib/lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,697 copying build/lib/lm_eval/tasks/bbh/zeroshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,699 copying build/lib/lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,701 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,703 copying build/lib/lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,705 copying build/lib/lm_eval/tasks/bbh/zeroshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,707 copying build/lib/lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,710 copying build/lib/lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,712 copying build/lib/lm_eval/tasks/bbh/zeroshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,714 copying build/lib/lm_eval/tasks/bbh/zeroshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,717 copying build/lib/lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,719 copying build/lib/lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,721 copying build/lib/lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,722 copying build/lib/lm_eval/tasks/bbh/zeroshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/zeroshot 2024-03-18T14:20:37,725 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,726 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,729 copying build/lib/lm_eval/tasks/bbh/fewshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,731 copying build/lib/lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,733 copying build/lib/lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,735 copying build/lib/lm_eval/tasks/bbh/fewshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,737 copying build/lib/lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,739 copying build/lib/lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,741 copying build/lib/lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,742 copying build/lib/lm_eval/tasks/bbh/fewshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,744 copying build/lib/lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,747 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,749 copying build/lib/lm_eval/tasks/bbh/fewshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,751 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,754 copying build/lib/lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,756 copying build/lib/lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,758 copying build/lib/lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,760 copying build/lib/lm_eval/tasks/bbh/fewshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,762 copying build/lib/lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,764 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,767 copying build/lib/lm_eval/tasks/bbh/fewshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,769 copying build/lib/lm_eval/tasks/bbh/fewshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,771 copying build/lib/lm_eval/tasks/bbh/fewshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,773 copying build/lib/lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,775 copying build/lib/lm_eval/tasks/bbh/fewshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,777 copying build/lib/lm_eval/tasks/bbh/fewshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,779 copying build/lib/lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,782 copying build/lib/lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,783 copying build/lib/lm_eval/tasks/bbh/fewshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/fewshot 2024-03-18T14:20:37,786 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,787 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,790 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,792 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,795 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,797 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,799 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,801 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,803 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,805 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,808 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,810 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,812 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,814 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,816 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,818 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,821 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,824 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,827 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,830 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,832 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,834 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,836 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,839 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,841 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/navigate.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,844 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/snarks.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,846 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,849 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,851 copying build/lib/lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh/cot_fewshot 2024-03-18T14:20:37,854 copying build/lib/lm_eval/tasks/bbh/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/bbh 2024-03-18T14:20:37,857 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-03-18T14:20:37,858 copying build/lib/lm_eval/tasks/winogrande/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-03-18T14:20:37,860 copying build/lib/lm_eval/tasks/winogrande/preprocess_winogrande.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-03-18T14:20:37,862 copying build/lib/lm_eval/tasks/winogrande/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/winogrande 2024-03-18T14:20:37,865 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-03-18T14:20:37,866 copying build/lib/lm_eval/tasks/babi/babi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-03-18T14:20:37,868 copying build/lib/lm_eval/tasks/babi/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/babi 2024-03-18T14:20:37,871 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-03-18T14:20:37,871 copying build/lib/lm_eval/tasks/asdiv/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-03-18T14:20:37,873 copying build/lib/lm_eval/tasks/asdiv/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/asdiv 2024-03-18T14:20:37,876 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-03-18T14:20:37,877 copying build/lib/lm_eval/tasks/mc_taco/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-03-18T14:20:37,879 copying build/lib/lm_eval/tasks/mc_taco/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mc_taco 2024-03-18T14:20:37,881 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-03-18T14:20:37,882 copying build/lib/lm_eval/tasks/siqa/siqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-03-18T14:20:37,885 copying build/lib/lm_eval/tasks/siqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/siqa 2024-03-18T14:20:37,887 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-03-18T14:20:37,888 copying build/lib/lm_eval/tasks/scrolls/scrolls.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-03-18T14:20:37,891 copying build/lib/lm_eval/tasks/scrolls/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-03-18T14:20:37,893 copying build/lib/lm_eval/tasks/scrolls/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/scrolls 2024-03-18T14:20:37,896 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-03-18T14:20:37,898 copying build/lib/lm_eval/tasks/webqs/webqs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-03-18T14:20:37,900 copying build/lib/lm_eval/tasks/webqs/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-03-18T14:20:37,902 copying build/lib/lm_eval/tasks/webqs/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/webqs 2024-03-18T14:20:37,906 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,907 copying build/lib/lm_eval/tasks/kobest/kobest_hellaswag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,910 copying build/lib/lm_eval/tasks/kobest/kobest_boolq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,912 copying build/lib/lm_eval/tasks/kobest/kobest_copa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,914 copying build/lib/lm_eval/tasks/kobest/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,916 copying build/lib/lm_eval/tasks/kobest/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,919 copying build/lib/lm_eval/tasks/kobest/kobest_sentineg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,921 copying build/lib/lm_eval/tasks/kobest/kobest_wic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kobest 2024-03-18T14:20:37,924 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,926 copying build/lib/lm_eval/tasks/pile/pile_ubuntu-irc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,928 copying build/lib/lm_eval/tasks/pile/pile_arxiv.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,931 copying build/lib/lm_eval/tasks/pile/pile_opensubtitles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,933 copying build/lib/lm_eval/tasks/pile/pile_nih-exporter.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,935 copying build/lib/lm_eval/tasks/pile/pile_europarl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,938 copying build/lib/lm_eval/tasks/pile/pile_dm-mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,941 copying build/lib/lm_eval/tasks/pile/pile_enron.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,943 copying build/lib/lm_eval/tasks/pile/pile_books3.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,946 copying build/lib/lm_eval/tasks/pile/pile_youtubesubtitles.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,948 copying build/lib/lm_eval/tasks/pile/pile_stackexchange.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,950 copying build/lib/lm_eval/tasks/pile/pile_gutenberg.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,952 copying build/lib/lm_eval/tasks/pile/pile_pubmed-central.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,955 copying build/lib/lm_eval/tasks/pile/pile_github.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,957 copying build/lib/lm_eval/tasks/pile/pile_philpapers.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,960 copying build/lib/lm_eval/tasks/pile/pile_bookcorpus2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,962 copying build/lib/lm_eval/tasks/pile/pile_pile-cc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,964 copying build/lib/lm_eval/tasks/pile/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,967 copying build/lib/lm_eval/tasks/pile/pile_pubmed-abstracts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,969 copying build/lib/lm_eval/tasks/pile/pile_freelaw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,972 copying build/lib/lm_eval/tasks/pile/pile_uspto.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,974 copying build/lib/lm_eval/tasks/pile/pile_openwebtext2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,976 copying build/lib/lm_eval/tasks/pile/pile_wikipedia.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,979 copying build/lib/lm_eval/tasks/pile/pile_hackernews.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/pile 2024-03-18T14:20:37,982 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,984 copying build/lib/lm_eval/tasks/hendrycks_ethics/deontology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,986 copying build/lib/lm_eval/tasks/hendrycks_ethics/justice.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,989 copying build/lib/lm_eval/tasks/hendrycks_ethics/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,991 copying build/lib/lm_eval/tasks/hendrycks_ethics/virtue.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,994 copying build/lib/lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,996 copying build/lib/lm_eval/tasks/hendrycks_ethics/commonsense.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:37,998 copying build/lib/lm_eval/tasks/hendrycks_ethics/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:38,001 copying build/lib/lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hendrycks_ethics 2024-03-18T14:20:38,003 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-03-18T14:20:38,005 copying build/lib/lm_eval/tasks/sciq/sciq.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-03-18T14:20:38,007 copying build/lib/lm_eval/tasks/sciq/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/sciq 2024-03-18T14:20:38,010 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-03-18T14:20:38,011 copying build/lib/lm_eval/tasks/openbookqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-03-18T14:20:38,014 copying build/lib/lm_eval/tasks/openbookqa/openbookqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/openbookqa 2024-03-18T14:20:38,017 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-03-18T14:20:38,018 copying build/lib/lm_eval/tasks/logiqa2/utils_logiqa2.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-03-18T14:20:38,021 copying build/lib/lm_eval/tasks/logiqa2/logiqa2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-03-18T14:20:38,023 copying build/lib/lm_eval/tasks/logiqa2/logieval.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-03-18T14:20:38,025 copying build/lib/lm_eval/tasks/logiqa2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa2 2024-03-18T14:20:38,029 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa 2024-03-18T14:20:38,031 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,032 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,034 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,036 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/gpqa_main_cot_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,038 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/_gpqa_cot_zeroshot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,040 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/gpqa_diamond_cot_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,042 copying build/lib/lm_eval/tasks/gpqa/cot_zeroshot/gpqa_extended_cot_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_zeroshot 2024-03-18T14:20:38,044 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,045 copying build/lib/lm_eval/tasks/gpqa/generative/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,047 copying build/lib/lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,049 copying build/lib/lm_eval/tasks/gpqa/generative/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,051 copying build/lib/lm_eval/tasks/gpqa/generative/gpqa_extended_generative_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,053 copying build/lib/lm_eval/tasks/gpqa/generative/gpqa_diamond_generative_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,055 copying build/lib/lm_eval/tasks/gpqa/generative/gpqa_main_generative_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/generative 2024-03-18T14:20:38,057 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,058 copying build/lib/lm_eval/tasks/gpqa/n_shot/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,060 copying build/lib/lm_eval/tasks/gpqa/n_shot/gpqa_main_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,062 copying build/lib/lm_eval/tasks/gpqa/n_shot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,064 copying build/lib/lm_eval/tasks/gpqa/n_shot/gpqa_extended_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,065 copying build/lib/lm_eval/tasks/gpqa/n_shot/gpqa_diamond_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,067 copying build/lib/lm_eval/tasks/gpqa/n_shot/_gpqa_n_shot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/n_shot 2024-03-18T14:20:38,069 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,070 copying build/lib/lm_eval/tasks/gpqa/zeroshot/gpqa_diamond_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,073 copying build/lib/lm_eval/tasks/gpqa/zeroshot/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,075 copying build/lib/lm_eval/tasks/gpqa/zeroshot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,077 copying build/lib/lm_eval/tasks/gpqa/zeroshot/_gpqa_zeroshot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,079 copying build/lib/lm_eval/tasks/gpqa/zeroshot/gpqa_extended_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,082 copying build/lib/lm_eval/tasks/gpqa/zeroshot/gpqa_main_zeroshot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/zeroshot 2024-03-18T14:20:38,084 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,085 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/gpqa_main_cot_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,087 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,089 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/gpqa_diamond_cot_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,090 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,092 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,094 copying build/lib/lm_eval/tasks/gpqa/cot_n_shot/gpqa_extended_cot_n_shot.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa/cot_n_shot 2024-03-18T14:20:38,096 copying build/lib/lm_eval/tasks/gpqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/gpqa 2024-03-18T14:20:38,099 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,100 copying build/lib/lm_eval/tasks/ceval/ceval-valid_operating_system.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,103 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,106 copying build/lib/lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,108 copying build/lib/lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,110 copying build/lib/lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,112 copying build/lib/lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,113 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,115 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,117 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,118 copying build/lib/lm_eval/tasks/ceval/ceval-valid_physician.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,120 copying build/lib/lm_eval/tasks/ceval/ceval-valid_accountant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,122 copying build/lib/lm_eval/tasks/ceval/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,125 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,127 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,129 copying build/lib/lm_eval/tasks/ceval/ceval-valid_art_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,130 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,132 copying build/lib/lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,134 copying build/lib/lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,136 copying build/lib/lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,138 copying build/lib/lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,140 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,141 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_programming.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,143 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,145 copying build/lib/lm_eval/tasks/ceval/ceval-valid_sports_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,147 copying build/lib/lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,149 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,151 copying build/lib/lm_eval/tasks/ceval/ceval-valid_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,152 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_economics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,154 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,156 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,158 copying build/lib/lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,159 copying build/lib/lm_eval/tasks/ceval/_default_ceval_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,161 copying build/lib/lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,163 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,165 copying build/lib/lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,174 copying build/lib/lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,176 copying build/lib/lm_eval/tasks/ceval/ceval-valid_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,178 copying build/lib/lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,179 copying build/lib/lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,182 copying build/lib/lm_eval/tasks/ceval/ceval-valid_computer_network.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,183 copying build/lib/lm_eval/tasks/ceval/ceval-valid_education_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,185 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,187 copying build/lib/lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,188 copying build/lib/lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,190 copying build/lib/lm_eval/tasks/ceval/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,193 copying build/lib/lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,194 copying build/lib/lm_eval/tasks/ceval/ceval-valid_marxism.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,196 copying build/lib/lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,198 copying build/lib/lm_eval/tasks/ceval/ceval-valid_business_administration.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,200 copying build/lib/lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,202 copying build/lib/lm_eval/tasks/ceval/ceval-valid_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,204 copying build/lib/lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,206 copying build/lib/lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,208 copying build/lib/lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,211 copying build/lib/lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/ceval 2024-03-18T14:20:38,215 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:38,216 copying build/lib/lm_eval/tasks/kormedmcqa/kormedmcqa_doctor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:38,220 copying build/lib/lm_eval/tasks/kormedmcqa/kormedmcqa_pharm.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:38,222 copying build/lib/lm_eval/tasks/kormedmcqa/kormedmcqa_nurse.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:38,223 copying build/lib/lm_eval/tasks/kormedmcqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/kormedmcqa 2024-03-18T14:20:38,226 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,226 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,228 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_4ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,230 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2dm.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,232 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_3da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,234 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_1dc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,236 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_5da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,238 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_4da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,240 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_5ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,242 copying build/lib/lm_eval/tasks/arithmetic/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,245 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_2da.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,246 copying build/lib/lm_eval/tasks/arithmetic/arithmetic_3ds.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/arithmetic 2024-03-18T14:20:38,249 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-03-18T14:20:38,250 copying build/lib/lm_eval/tasks/wmt2016/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-03-18T14:20:38,252 copying build/lib/lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-03-18T14:20:38,253 copying build/lib/lm_eval/tasks/wmt2016/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/wmt2016 2024-03-18T14:20:38,256 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,257 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,259 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,261 copying build/lib/lm_eval/tasks/lambada_multilingual/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,263 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,266 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,268 copying build/lib/lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/lambada_multilingual 2024-03-18T14:20:38,270 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-03-18T14:20:38,271 copying build/lib/lm_eval/tasks/squadv2/squadv2.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-03-18T14:20:38,273 copying build/lib/lm_eval/tasks/squadv2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-03-18T14:20:38,275 copying build/lib/lm_eval/tasks/squadv2/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/squadv2 2024-03-18T14:20:38,278 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/eq_bench 2024-03-18T14:20:38,279 copying build/lib/lm_eval/tasks/eq_bench/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/eq_bench 2024-03-18T14:20:38,281 copying build/lib/lm_eval/tasks/eq_bench/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/eq_bench 2024-03-18T14:20:38,283 copying build/lib/lm_eval/tasks/eq_bench/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/eq_bench 2024-03-18T14:20:38,285 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-03-18T14:20:38,286 copying build/lib/lm_eval/tasks/prost/corypaik_prost.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-03-18T14:20:38,288 copying build/lib/lm_eval/tasks/prost/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/prost 2024-03-18T14:20:38,290 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-03-18T14:20:38,291 copying build/lib/lm_eval/tasks/coqa/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-03-18T14:20:38,293 copying build/lib/lm_eval/tasks/coqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-03-18T14:20:38,295 copying build/lib/lm_eval/tasks/coqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/coqa 2024-03-18T14:20:38,298 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/medmcqa 2024-03-18T14:20:38,299 copying build/lib/lm_eval/tasks/medmcqa/utils_medmcqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/medmcqa 2024-03-18T14:20:38,301 copying build/lib/lm_eval/tasks/medmcqa/medmcqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/medmcqa 2024-03-18T14:20:38,303 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,304 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,306 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_geometry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,308 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,310 copying build/lib/lm_eval/tasks/minerva_math/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,312 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,314 copying build/lib/lm_eval/tasks/minerva_math/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,317 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_precalc.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,319 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,321 copying build/lib/lm_eval/tasks/minerva_math/minerva_math_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/minerva_math 2024-03-18T14:20:38,324 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,326 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,330 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,333 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,336 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,340 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,344 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,348 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,351 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,352 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,354 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,356 copying build/lib/lm_eval/tasks/crows_pairs/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,358 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,360 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,362 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,363 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,365 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,367 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,369 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,370 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,372 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,374 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,376 copying build/lib/lm_eval/tasks/crows_pairs/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,379 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,381 copying build/lib/lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/crows_pairs 2024-03-18T14:20:38,383 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu 2024-03-18T14:20:38,384 copying build/lib/lm_eval/tasks/mmlu/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu 2024-03-18T14:20:38,387 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot 2024-03-18T14:20:38,389 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,391 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,393 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,395 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,397 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,398 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,400 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,402 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,404 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,406 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,407 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,409 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,411 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,413 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,415 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,417 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,419 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,421 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,423 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,425 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,427 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,429 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,431 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,433 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,435 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,437 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,440 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,442 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,444 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,446 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,449 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,451 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,453 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,455 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,458 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,460 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,462 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,464 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,466 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,467 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,469 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,471 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,473 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,476 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,478 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,480 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,482 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,484 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,486 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,488 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,491 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,493 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,495 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,497 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,500 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,502 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,505 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,507 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,508 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,511 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/loglikelihood 2024-03-18T14:20:38,514 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,515 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,517 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,519 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,521 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,523 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,525 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,527 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,528 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,530 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,532 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,534 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,536 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,538 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,540 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,542 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,544 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,546 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,547 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,549 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,551 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,554 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,556 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,558 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,560 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,562 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,564 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,566 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,568 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,570 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,572 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,574 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,576 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,577 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,579 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,581 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,583 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,585 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,586 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,588 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,590 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,592 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,594 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,595 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,598 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,600 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,602 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,604 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,607 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,609 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,611 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,614 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,616 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,618 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,620 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,622 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,624 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,625 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,627 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,629 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,631 copying build/lib/lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_n_shot/generative 2024-03-18T14:20:38,634 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,635 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,637 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,639 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,641 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,643 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,645 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,646 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,648 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,650 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,652 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,654 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,656 copying build/lib/lm_eval/tasks/mmlu/default/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,658 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,660 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,662 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,664 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,666 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,668 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,669 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,671 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,673 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,676 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,678 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,679 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,681 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,683 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,685 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,687 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,689 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,691 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,693 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,694 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,696 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,698 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,700 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,702 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,705 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,708 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,709 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,711 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,713 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,714 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,716 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,718 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,720 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,722 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,725 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,727 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,730 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,734 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,738 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,742 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,744 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,746 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,749 copying build/lib/lm_eval/tasks/mmlu/default/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,751 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,755 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,758 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,761 copying build/lib/lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/default 2024-03-18T14:20:38,765 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,767 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,770 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,772 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,776 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,779 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,781 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,785 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,789 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,792 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,794 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,798 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,800 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,803 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,808 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,811 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,814 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,817 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,821 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,823 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,825 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,828 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,832 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,834 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,837 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,839 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,843 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,846 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,848 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,850 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,852 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,854 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,862 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,864 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,866 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,869 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,872 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,874 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,876 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,878 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,880 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,881 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,883 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,885 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,887 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,889 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,891 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,893 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,895 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,898 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,900 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,903 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,906 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,908 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,910 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,912 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,914 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,917 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,919 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,921 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,924 copying build/lib/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_fewshot 2024-03-18T14:20:38,927 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,928 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,930 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,932 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,934 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,936 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,938 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,939 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,941 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,943 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,945 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,947 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,949 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,950 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,952 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,954 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,956 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,958 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,959 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,961 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,963 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,965 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,966 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,968 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,971 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,973 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,974 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,977 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,979 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,982 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,984 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,986 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,988 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,990 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,992 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,995 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,996 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:38,998 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,000 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,002 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,003 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,005 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,007 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,009 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,011 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,013 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,015 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,017 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,019 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,021 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,023 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,024 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,026 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,028 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,030 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,033 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,034 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,036 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,038 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,040 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,042 copying build/lib/lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/mmlu/flan_cot_zeroshot 2024-03-18T14:20:39,044 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-03-18T14:20:39,045 copying build/lib/lm_eval/tasks/logiqa/utils_logiqa.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-03-18T14:20:39,047 copying build/lib/lm_eval/tasks/logiqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-03-18T14:20:39,049 copying build/lib/lm_eval/tasks/logiqa/logiqa.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/logiqa 2024-03-18T14:20:39,051 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-03-18T14:20:39,052 copying build/lib/lm_eval/tasks/polemo2/polemo2_out.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-03-18T14:20:39,054 copying build/lib/lm_eval/tasks/polemo2/polemo2_in.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-03-18T14:20:39,056 copying build/lib/lm_eval/tasks/polemo2/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/polemo2 2024-03-18T14:20:39,061 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,061 copying build/lib/lm_eval/tasks/belebele/belebele_lin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,063 copying build/lib/lm_eval/tasks/belebele/belebele_mar_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,065 copying build/lib/lm_eval/tasks/belebele/belebele_fra_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,067 copying build/lib/lm_eval/tasks/belebele/belebele_swe_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,069 copying build/lib/lm_eval/tasks/belebele/belebele_pbt_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,071 copying build/lib/lm_eval/tasks/belebele/belebele_jav_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,073 copying build/lib/lm_eval/tasks/belebele/belebele_fuv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,075 copying build/lib/lm_eval/tasks/belebele/belebele_plt_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,077 copying build/lib/lm_eval/tasks/belebele/belebele_spa_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,078 copying build/lib/lm_eval/tasks/belebele/belebele_war_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,080 copying build/lib/lm_eval/tasks/belebele/belebele_ben_Beng.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,082 copying build/lib/lm_eval/tasks/belebele/belebele_ibo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,084 copying build/lib/lm_eval/tasks/belebele/belebele_tam_Taml.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,086 copying build/lib/lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,087 copying build/lib/lm_eval/tasks/belebele/belebele_eus_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,089 copying build/lib/lm_eval/tasks/belebele/belebele_bam_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,091 copying build/lib/lm_eval/tasks/belebele/belebele_arb_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,093 copying build/lib/lm_eval/tasks/belebele/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,095 copying build/lib/lm_eval/tasks/belebele/belebele_kan_Knda.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,097 copying build/lib/lm_eval/tasks/belebele/belebele_mya_Mymr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,099 copying build/lib/lm_eval/tasks/belebele/belebele_hin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,100 copying build/lib/lm_eval/tasks/belebele/belebele_dan_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,102 copying build/lib/lm_eval/tasks/belebele/belebele_ssw_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,104 copying build/lib/lm_eval/tasks/belebele/belebele_ita_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,106 copying build/lib/lm_eval/tasks/belebele/belebele_hin_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,108 copying build/lib/lm_eval/tasks/belebele/belebele_pan_Guru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,110 copying build/lib/lm_eval/tasks/belebele/belebele_shn_Mymr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,111 copying build/lib/lm_eval/tasks/belebele/belebele_tgl_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,113 copying build/lib/lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,115 copying build/lib/lm_eval/tasks/belebele/belebele_kea_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,117 copying build/lib/lm_eval/tasks/belebele/belebele_arz_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,118 copying build/lib/lm_eval/tasks/belebele/belebele_eng_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,120 copying build/lib/lm_eval/tasks/belebele/belebele_tel_Telu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,122 copying build/lib/lm_eval/tasks/belebele/belebele_tso_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,124 copying build/lib/lm_eval/tasks/belebele/belebele_luo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,131 copying build/lib/lm_eval/tasks/belebele/belebele_zho_Hant.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,133 copying build/lib/lm_eval/tasks/belebele/belebele_ces_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,134 copying build/lib/lm_eval/tasks/belebele/belebele_nya_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,136 copying build/lib/lm_eval/tasks/belebele/belebele_pol_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,138 copying build/lib/lm_eval/tasks/belebele/belebele_mal_Mlym.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,141 copying build/lib/lm_eval/tasks/belebele/belebele_nld_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,143 copying build/lib/lm_eval/tasks/belebele/belebele_npi_Deva.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,145 copying build/lib/lm_eval/tasks/belebele/belebele_pes_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,147 copying build/lib/lm_eval/tasks/belebele/belebele_kat_Geor.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,149 copying build/lib/lm_eval/tasks/belebele/belebele_ind_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,151 copying build/lib/lm_eval/tasks/belebele/belebele_hun_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,153 copying build/lib/lm_eval/tasks/belebele/belebele_ceb_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,155 copying build/lib/lm_eval/tasks/belebele/belebele_tir_Ethi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,157 copying build/lib/lm_eval/tasks/belebele/belebele_npi_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,159 copying build/lib/lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,161 copying build/lib/lm_eval/tasks/belebele/belebele_apc_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,163 copying build/lib/lm_eval/tasks/belebele/belebele_tsn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,165 copying build/lib/lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,166 copying build/lib/lm_eval/tasks/belebele/belebele_tha_Thai.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,169 copying build/lib/lm_eval/tasks/belebele/belebele_sun_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,171 copying build/lib/lm_eval/tasks/belebele/belebele_ron_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,173 copying build/lib/lm_eval/tasks/belebele/belebele_wol_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,175 copying build/lib/lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,177 copying build/lib/lm_eval/tasks/belebele/belebele_deu_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,179 copying build/lib/lm_eval/tasks/belebele/belebele_ell_Grek.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,181 copying build/lib/lm_eval/tasks/belebele/belebele_swh_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,183 copying build/lib/lm_eval/tasks/belebele/belebele_als_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,185 copying build/lib/lm_eval/tasks/belebele/belebele_acm_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,187 copying build/lib/lm_eval/tasks/belebele/belebele_nob_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,189 copying build/lib/lm_eval/tasks/belebele/belebele_ckb_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,190 copying build/lib/lm_eval/tasks/belebele/belebele_kin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,192 copying build/lib/lm_eval/tasks/belebele/belebele_sin_Sinh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,194 copying build/lib/lm_eval/tasks/belebele/belebele_xho_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,196 copying build/lib/lm_eval/tasks/belebele/belebele_fin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,198 copying build/lib/lm_eval/tasks/belebele/belebele_kac_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,200 copying build/lib/lm_eval/tasks/belebele/belebele_mlt_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,202 copying build/lib/lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,204 copying build/lib/lm_eval/tasks/belebele/belebele_uzn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,206 copying build/lib/lm_eval/tasks/belebele/belebele_azj_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,208 copying build/lib/lm_eval/tasks/belebele/belebele_lao_Laoo.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,210 copying build/lib/lm_eval/tasks/belebele/belebele_slv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,212 copying build/lib/lm_eval/tasks/belebele/belebele_est_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,213 copying build/lib/lm_eval/tasks/belebele/belebele_sna_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,215 copying build/lib/lm_eval/tasks/belebele/belebele_urd_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,217 copying build/lib/lm_eval/tasks/belebele/belebele_guj_Gujr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,219 copying build/lib/lm_eval/tasks/belebele/belebele_som_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,221 copying build/lib/lm_eval/tasks/belebele/belebele_tur_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,223 copying build/lib/lm_eval/tasks/belebele/belebele_amh_Ethi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,225 copying build/lib/lm_eval/tasks/belebele/belebele_cat_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,227 copying build/lib/lm_eval/tasks/belebele/belebele_khm_Khmr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,229 copying build/lib/lm_eval/tasks/belebele/belebele_mri_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,231 copying build/lib/lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,233 copying build/lib/lm_eval/tasks/belebele/belebele_urd_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,235 copying build/lib/lm_eval/tasks/belebele/belebele_ars_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,236 copying build/lib/lm_eval/tasks/belebele/belebele_heb_Hebr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,238 copying build/lib/lm_eval/tasks/belebele/belebele_lug_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,240 copying build/lib/lm_eval/tasks/belebele/belebele_sot_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,242 copying build/lib/lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,244 copying build/lib/lm_eval/tasks/belebele/belebele_slk_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,246 copying build/lib/lm_eval/tasks/belebele/belebele_por_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,248 copying build/lib/lm_eval/tasks/belebele/belebele_ary_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,250 copying build/lib/lm_eval/tasks/belebele/belebele_afr_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,252 copying build/lib/lm_eval/tasks/belebele/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,254 copying build/lib/lm_eval/tasks/belebele/belebele_hrv_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,255 copying build/lib/lm_eval/tasks/belebele/belebele_ilo_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,257 copying build/lib/lm_eval/tasks/belebele/belebele_zul_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,259 copying build/lib/lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,261 copying build/lib/lm_eval/tasks/belebele/belebele_snd_Arab.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,263 copying build/lib/lm_eval/tasks/belebele/belebele_isl_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,265 copying build/lib/lm_eval/tasks/belebele/belebele_arb_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,267 copying build/lib/lm_eval/tasks/belebele/belebele_sin_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,268 copying build/lib/lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,270 copying build/lib/lm_eval/tasks/belebele/belebele_hat_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,272 copying build/lib/lm_eval/tasks/belebele/belebele_lvs_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,274 copying build/lib/lm_eval/tasks/belebele/belebele_ory_Orya.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,276 copying build/lib/lm_eval/tasks/belebele/belebele_kor_Hang.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,278 copying build/lib/lm_eval/tasks/belebele/belebele_vie_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,280 copying build/lib/lm_eval/tasks/belebele/belebele_ben_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,282 copying build/lib/lm_eval/tasks/belebele/belebele_nso_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,284 copying build/lib/lm_eval/tasks/belebele/belebele_zho_Hans.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,286 copying build/lib/lm_eval/tasks/belebele/belebele_lit_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,289 copying build/lib/lm_eval/tasks/belebele/belebele_zsm_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,291 copying build/lib/lm_eval/tasks/belebele/belebele_yor_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,296 copying build/lib/lm_eval/tasks/belebele/belebele_bod_Tibt.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,298 copying build/lib/lm_eval/tasks/belebele/_default_template_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,300 copying build/lib/lm_eval/tasks/belebele/belebele_asm_Beng.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,302 copying build/lib/lm_eval/tasks/belebele/belebele_grn_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,304 copying build/lib/lm_eval/tasks/belebele/belebele_gaz_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,306 copying build/lib/lm_eval/tasks/belebele/belebele_hau_Latn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,308 copying build/lib/lm_eval/tasks/belebele/belebele_hye_Armn.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/belebele 2024-03-18T14:20:39,311 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,312 copying build/lib/lm_eval/tasks/xstorycloze/default_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,314 copying build/lib/lm_eval/tasks/xstorycloze/default_te.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,316 copying build/lib/lm_eval/tasks/xstorycloze/default_ru.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,317 copying build/lib/lm_eval/tasks/xstorycloze/default_my.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,319 copying build/lib/lm_eval/tasks/xstorycloze/default_sw.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,321 copying build/lib/lm_eval/tasks/xstorycloze/default_hi.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,323 copying build/lib/lm_eval/tasks/xstorycloze/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,325 copying build/lib/lm_eval/tasks/xstorycloze/default_eu.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,327 copying build/lib/lm_eval/tasks/xstorycloze/default_ar.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,329 copying build/lib/lm_eval/tasks/xstorycloze/default_id.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,331 copying build/lib/lm_eval/tasks/xstorycloze/default_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,332 copying build/lib/lm_eval/tasks/xstorycloze/default_zh.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/xstorycloze 2024-03-18T14:20:39,335 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-03-18T14:20:39,336 copying build/lib/lm_eval/tasks/headqa/headqa_es.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-03-18T14:20:39,338 copying build/lib/lm_eval/tasks/headqa/headqa_en.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-03-18T14:20:39,340 copying build/lib/lm_eval/tasks/headqa/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/headqa 2024-03-18T14:20:39,342 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-03-18T14:20:39,343 copying build/lib/lm_eval/tasks/drop/default.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-03-18T14:20:39,345 copying build/lib/lm_eval/tasks/drop/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-03-18T14:20:39,348 copying build/lib/lm_eval/tasks/drop/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/drop 2024-03-18T14:20:39,351 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-03-18T14:20:39,351 copying build/lib/lm_eval/tasks/hellaswag/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-03-18T14:20:39,353 copying build/lib/lm_eval/tasks/hellaswag/README.md -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-03-18T14:20:39,356 copying build/lib/lm_eval/tasks/hellaswag/hellaswag.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/hellaswag 2024-03-18T14:20:39,358 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,359 copying build/lib/lm_eval/tasks/csatqa/csatqa_rcs.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,362 copying build/lib/lm_eval/tasks/csatqa/csatqa_rcss.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,364 copying build/lib/lm_eval/tasks/csatqa/_generate_configs.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,366 copying build/lib/lm_eval/tasks/csatqa/_default_csatqa_yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,368 copying build/lib/lm_eval/tasks/csatqa/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,371 copying build/lib/lm_eval/tasks/csatqa/csatqa_wr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,373 copying build/lib/lm_eval/tasks/csatqa/csatqa_gr.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,375 copying build/lib/lm_eval/tasks/csatqa/csatqa_rch.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,377 copying build/lib/lm_eval/tasks/csatqa/csatqa_li.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/csatqa 2024-03-18T14:20:39,380 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue 2024-03-18T14:20:39,382 creating build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,383 copying build/lib/lm_eval/tasks/code_x_glue/code-text/ruby.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,385 copying build/lib/lm_eval/tasks/code_x_glue/code-text/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,387 copying build/lib/lm_eval/tasks/code_x_glue/code-text/go.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,388 copying build/lib/lm_eval/tasks/code_x_glue/code-text/bleu.py -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,391 copying build/lib/lm_eval/tasks/code_x_glue/code-text/javascript.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,393 copying build/lib/lm_eval/tasks/code_x_glue/code-text/php.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,395 copying build/lib/lm_eval/tasks/code_x_glue/code-text/java.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,397 copying build/lib/lm_eval/tasks/code_x_glue/code-text/python.yaml -> build/bdist.linux-armv7l/wheel/lm_eval/tasks/code_x_glue/code-text 2024-03-18T14:20:39,399 copying build/lib/lm_eval/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:39,401 copying build/lib/lm_eval/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:39,404 copying build/lib/lm_eval/evaluator_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:39,408 creating build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,409 copying build/lib/lm_eval/models/textsynth.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,411 copying build/lib/lm_eval/models/optimum_lm.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,414 copying build/lib/lm_eval/models/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,415 copying build/lib/lm_eval/models/utils.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,418 copying build/lib/lm_eval/models/neuron_optimum.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,421 copying build/lib/lm_eval/models/anthropic_llms.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,424 copying build/lib/lm_eval/models/openai_completions.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,427 copying build/lib/lm_eval/models/mamba_lm.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,429 copying build/lib/lm_eval/models/huggingface.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,434 copying build/lib/lm_eval/models/dummy.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,436 copying build/lib/lm_eval/models/gguf.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,438 copying build/lib/lm_eval/models/vllm_causallms.py -> build/bdist.linux-armv7l/wheel/lm_eval/models 2024-03-18T14:20:39,442 creating build/bdist.linux-armv7l/wheel/lm_eval/prompts 2024-03-18T14:20:39,443 copying build/lib/lm_eval/prompts/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/prompts 2024-03-18T14:20:39,445 copying build/lib/lm_eval/__main__.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:39,449 creating build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-03-18T14:20:39,450 copying build/lib/lm_eval/decontamination/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-03-18T14:20:39,452 copying build/lib/lm_eval/decontamination/janitor.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-03-18T14:20:39,455 copying build/lib/lm_eval/decontamination/archiver.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-03-18T14:20:39,457 copying build/lib/lm_eval/decontamination/decontaminate.py -> build/bdist.linux-armv7l/wheel/lm_eval/decontamination 2024-03-18T14:20:39,459 copying build/lib/lm_eval/logging_utils.py -> build/bdist.linux-armv7l/wheel/lm_eval 2024-03-18T14:20:39,463 creating build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,463 copying build/lib/lm_eval/filters/selection.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,466 copying build/lib/lm_eval/filters/transformation.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,468 copying build/lib/lm_eval/filters/decontamination.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,470 copying build/lib/lm_eval/filters/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,472 copying build/lib/lm_eval/filters/extraction.py -> build/bdist.linux-armv7l/wheel/lm_eval/filters 2024-03-18T14:20:39,476 creating build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,476 copying build/lib/lm_eval/api/samplers.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,479 copying build/lib/lm_eval/api/metrics.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,481 copying build/lib/lm_eval/api/__init__.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,483 copying build/lib/lm_eval/api/model.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,485 copying build/lib/lm_eval/api/registry.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,487 copying build/lib/lm_eval/api/filter.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,490 copying build/lib/lm_eval/api/task.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,494 copying build/lib/lm_eval/api/instance.py -> build/bdist.linux-armv7l/wheel/lm_eval/api 2024-03-18T14:20:39,497 creating build/bdist.linux-armv7l/wheel/lm_eval/caching 2024-03-18T14:20:39,498 copying build/lib/lm_eval/caching/cache.py -> build/bdist.linux-armv7l/wheel/lm_eval/caching 2024-03-18T14:20:39,501 running install_egg_info 2024-03-18T14:20:39,505 Copying lm_eval.egg-info to build/bdist.linux-armv7l/wheel/lm_eval-0.4.2-py3.11.egg-info 2024-03-18T14:20:39,525 running install_scripts 2024-03-18T14:20:39,567 creating build/bdist.linux-armv7l/wheel/lm_eval-0.4.2.dist-info/WHEEL 2024-03-18T14:20:39,571 creating '/tmp/pip-wheel-r61w3ndf/.tmp-kwxwi2y5/lm_eval-0.4.2-py3-none-any.whl' and adding 'build/bdist.linux-armv7l/wheel' to it 2024-03-18T14:20:39,574 adding 'lm_eval/__init__.py' 2024-03-18T14:20:39,577 adding 'lm_eval/__main__.py' 2024-03-18T14:20:39,582 adding 'lm_eval/evaluator.py' 2024-03-18T14:20:39,585 adding 'lm_eval/evaluator_utils.py' 2024-03-18T14:20:39,588 adding 'lm_eval/logging_utils.py' 2024-03-18T14:20:39,592 adding 'lm_eval/utils.py' 2024-03-18T14:20:39,594 adding 'lm_eval/api/__init__.py' 2024-03-18T14:20:39,596 adding 'lm_eval/api/filter.py' 2024-03-18T14:20:39,598 adding 'lm_eval/api/instance.py' 2024-03-18T14:20:39,601 adding 'lm_eval/api/metrics.py' 2024-03-18T14:20:39,605 adding 'lm_eval/api/model.py' 2024-03-18T14:20:39,607 adding 'lm_eval/api/registry.py' 2024-03-18T14:20:39,609 adding 'lm_eval/api/samplers.py' 2024-03-18T14:20:39,618 adding 'lm_eval/api/task.py' 2024-03-18T14:20:39,621 adding 'lm_eval/caching/cache.py' 2024-03-18T14:20:39,623 adding 'lm_eval/decontamination/__init__.py' 2024-03-18T14:20:39,625 adding 'lm_eval/decontamination/archiver.py' 2024-03-18T14:20:39,627 adding 'lm_eval/decontamination/decontaminate.py' 2024-03-18T14:20:39,630 adding 'lm_eval/decontamination/janitor.py' 2024-03-18T14:20:39,632 adding 'lm_eval/filters/__init__.py' 2024-03-18T14:20:39,634 adding 'lm_eval/filters/decontamination.py' 2024-03-18T14:20:39,636 adding 'lm_eval/filters/extraction.py' 2024-03-18T14:20:39,637 adding 'lm_eval/filters/selection.py' 2024-03-18T14:20:39,639 adding 'lm_eval/filters/transformation.py' 2024-03-18T14:20:39,641 adding 'lm_eval/models/__init__.py' 2024-03-18T14:20:39,643 adding 'lm_eval/models/anthropic_llms.py' 2024-03-18T14:20:39,645 adding 'lm_eval/models/dummy.py' 2024-03-18T14:20:39,647 adding 'lm_eval/models/gguf.py' 2024-03-18T14:20:39,653 adding 'lm_eval/models/huggingface.py' 2024-03-18T14:20:39,655 adding 'lm_eval/models/mamba_lm.py' 2024-03-18T14:20:39,658 adding 'lm_eval/models/neuron_optimum.py' 2024-03-18T14:20:39,661 adding 'lm_eval/models/openai_completions.py' 2024-03-18T14:20:39,662 adding 'lm_eval/models/optimum_lm.py' 2024-03-18T14:20:39,664 adding 'lm_eval/models/textsynth.py' 2024-03-18T14:20:39,667 adding 'lm_eval/models/utils.py' 2024-03-18T14:20:39,670 adding 'lm_eval/models/vllm_causallms.py' 2024-03-18T14:20:39,672 adding 'lm_eval/prompts/__init__.py' 2024-03-18T14:20:39,676 adding 'lm_eval/tasks/__init__.py' 2024-03-18T14:20:39,678 adding 'lm_eval/tasks/aexams/README.md' 2024-03-18T14:20:39,680 adding 'lm_eval/tasks/aexams/_default_template_yaml' 2024-03-18T14:20:39,681 adding 'lm_eval/tasks/aexams/aexams_Biology.yaml' 2024-03-18T14:20:39,682 adding 'lm_eval/tasks/aexams/aexams_IslamicStudies.yaml' 2024-03-18T14:20:39,683 adding 'lm_eval/tasks/aexams/aexams_Physics.yaml' 2024-03-18T14:20:39,684 adding 'lm_eval/tasks/aexams/aexams_Science.yaml' 2024-03-18T14:20:39,685 adding 'lm_eval/tasks/aexams/aexams_Social.yaml' 2024-03-18T14:20:39,688 adding 'lm_eval/tasks/agieval/README.md' 2024-03-18T14:20:39,689 adding 'lm_eval/tasks/agieval/aqua-rat.yaml' 2024-03-18T14:20:39,690 adding 'lm_eval/tasks/agieval/gaokao-biology.yaml' 2024-03-18T14:20:39,691 adding 'lm_eval/tasks/agieval/gaokao-chemistry.yaml' 2024-03-18T14:20:39,692 adding 'lm_eval/tasks/agieval/gaokao-chinese.yaml' 2024-03-18T14:20:39,694 adding 'lm_eval/tasks/agieval/gaokao-english.yaml' 2024-03-18T14:20:39,695 adding 'lm_eval/tasks/agieval/gaokao-geography.yaml' 2024-03-18T14:20:39,696 adding 'lm_eval/tasks/agieval/gaokao-history.yaml' 2024-03-18T14:20:39,697 adding 'lm_eval/tasks/agieval/gaokao-mathcloze.yaml' 2024-03-18T14:20:39,698 adding 'lm_eval/tasks/agieval/gaokao-mathqa.yaml' 2024-03-18T14:20:39,699 adding 'lm_eval/tasks/agieval/gaokao-physics.yaml' 2024-03-18T14:20:39,700 adding 'lm_eval/tasks/agieval/jec-qa-ca.yaml' 2024-03-18T14:20:39,702 adding 'lm_eval/tasks/agieval/jec-qa-kd.yaml' 2024-03-18T14:20:39,703 adding 'lm_eval/tasks/agieval/logiqa-en.yaml' 2024-03-18T14:20:39,705 adding 'lm_eval/tasks/agieval/logiqa-zh.yaml' 2024-03-18T14:20:39,706 adding 'lm_eval/tasks/agieval/lsat-ar.yaml' 2024-03-18T14:20:39,707 adding 'lm_eval/tasks/agieval/lsat-lr.yaml' 2024-03-18T14:20:39,708 adding 'lm_eval/tasks/agieval/lsat-rc.yaml' 2024-03-18T14:20:39,709 adding 'lm_eval/tasks/agieval/math.yaml' 2024-03-18T14:20:39,711 adding 'lm_eval/tasks/agieval/sat-en-without-passage.yaml' 2024-03-18T14:20:39,712 adding 'lm_eval/tasks/agieval/sat-en.yaml' 2024-03-18T14:20:39,713 adding 'lm_eval/tasks/agieval/sat-math.yaml' 2024-03-18T14:20:39,714 adding 'lm_eval/tasks/agieval/utils.py' 2024-03-18T14:20:39,718 adding 'lm_eval/tasks/ammlu/README.md' 2024-03-18T14:20:39,719 adding 'lm_eval/tasks/ammlu/_default_template_yaml' 2024-03-18T14:20:39,720 adding 'lm_eval/tasks/ammlu/_generate_configs.py' 2024-03-18T14:20:39,722 adding 'lm_eval/tasks/ammlu/ammlu_abstract_algebra.yaml' 2024-03-18T14:20:39,723 adding 'lm_eval/tasks/ammlu/ammlu_anatomy.yaml' 2024-03-18T14:20:39,724 adding 'lm_eval/tasks/ammlu/ammlu_astronomy.yaml' 2024-03-18T14:20:39,725 adding 'lm_eval/tasks/ammlu/ammlu_business_ethics.yaml' 2024-03-18T14:20:39,727 adding 'lm_eval/tasks/ammlu/ammlu_clinical_knowledge.yaml' 2024-03-18T14:20:39,728 adding 'lm_eval/tasks/ammlu/ammlu_college_biology.yaml' 2024-03-18T14:20:39,729 adding 'lm_eval/tasks/ammlu/ammlu_college_chemistry.yaml' 2024-03-18T14:20:39,730 adding 'lm_eval/tasks/ammlu/ammlu_college_computer_science.yaml' 2024-03-18T14:20:39,731 adding 'lm_eval/tasks/ammlu/ammlu_college_mathematics.yaml' 2024-03-18T14:20:39,732 adding 'lm_eval/tasks/ammlu/ammlu_college_medicine.yaml' 2024-03-18T14:20:39,734 adding 'lm_eval/tasks/ammlu/ammlu_college_physics.yaml' 2024-03-18T14:20:39,735 adding 'lm_eval/tasks/ammlu/ammlu_computer_security.yaml' 2024-03-18T14:20:39,736 adding 'lm_eval/tasks/ammlu/ammlu_conceptual_physics.yaml' 2024-03-18T14:20:39,737 adding 'lm_eval/tasks/ammlu/ammlu_econometrics.yaml' 2024-03-18T14:20:39,738 adding 'lm_eval/tasks/ammlu/ammlu_electrical_engineering.yaml' 2024-03-18T14:20:39,739 adding 'lm_eval/tasks/ammlu/ammlu_elementary_mathematics.yaml' 2024-03-18T14:20:39,741 adding 'lm_eval/tasks/ammlu/ammlu_formal_logic.yaml' 2024-03-18T14:20:39,742 adding 'lm_eval/tasks/ammlu/ammlu_global_facts.yaml' 2024-03-18T14:20:39,743 adding 'lm_eval/tasks/ammlu/ammlu_high_school_biology.yaml' 2024-03-18T14:20:39,744 adding 'lm_eval/tasks/ammlu/ammlu_high_school_chemistry.yaml' 2024-03-18T14:20:39,745 adding 'lm_eval/tasks/ammlu/ammlu_high_school_computer_science.yaml' 2024-03-18T14:20:39,746 adding 'lm_eval/tasks/ammlu/ammlu_high_school_european_history.yaml' 2024-03-18T14:20:39,748 adding 'lm_eval/tasks/ammlu/ammlu_high_school_geography.yaml' 2024-03-18T14:20:39,749 adding 'lm_eval/tasks/ammlu/ammlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:39,750 adding 'lm_eval/tasks/ammlu/ammlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:39,752 adding 'lm_eval/tasks/ammlu/ammlu_high_school_mathematics.yaml' 2024-03-18T14:20:39,753 adding 'lm_eval/tasks/ammlu/ammlu_high_school_microeconomics.yaml' 2024-03-18T14:20:39,754 adding 'lm_eval/tasks/ammlu/ammlu_high_school_physics.yaml' 2024-03-18T14:20:39,755 adding 'lm_eval/tasks/ammlu/ammlu_high_school_psychology.yaml' 2024-03-18T14:20:39,756 adding 'lm_eval/tasks/ammlu/ammlu_high_school_statistics.yaml' 2024-03-18T14:20:39,757 adding 'lm_eval/tasks/ammlu/ammlu_high_school_us_history.yaml' 2024-03-18T14:20:39,758 adding 'lm_eval/tasks/ammlu/ammlu_high_school_world_history.yaml' 2024-03-18T14:20:39,760 adding 'lm_eval/tasks/ammlu/ammlu_human_aging.yaml' 2024-03-18T14:20:39,761 adding 'lm_eval/tasks/ammlu/ammlu_human_sexuality.yaml' 2024-03-18T14:20:39,762 adding 'lm_eval/tasks/ammlu/ammlu_international_law.yaml' 2024-03-18T14:20:39,763 adding 'lm_eval/tasks/ammlu/ammlu_jurisprudence.yaml' 2024-03-18T14:20:39,764 adding 'lm_eval/tasks/ammlu/ammlu_logical_fallacies.yaml' 2024-03-18T14:20:39,765 adding 'lm_eval/tasks/ammlu/ammlu_machine_learning.yaml' 2024-03-18T14:20:39,766 adding 'lm_eval/tasks/ammlu/ammlu_management.yaml' 2024-03-18T14:20:39,768 adding 'lm_eval/tasks/ammlu/ammlu_marketing.yaml' 2024-03-18T14:20:39,769 adding 'lm_eval/tasks/ammlu/ammlu_medical_genetics.yaml' 2024-03-18T14:20:39,770 adding 'lm_eval/tasks/ammlu/ammlu_miscellaneous.yaml' 2024-03-18T14:20:39,771 adding 'lm_eval/tasks/ammlu/ammlu_moral_disputes.yaml' 2024-03-18T14:20:39,772 adding 'lm_eval/tasks/ammlu/ammlu_moral_scenarios.yaml' 2024-03-18T14:20:39,773 adding 'lm_eval/tasks/ammlu/ammlu_nutrition.yaml' 2024-03-18T14:20:39,774 adding 'lm_eval/tasks/ammlu/ammlu_philosophy.yaml' 2024-03-18T14:20:39,775 adding 'lm_eval/tasks/ammlu/ammlu_prehistory.yaml' 2024-03-18T14:20:39,777 adding 'lm_eval/tasks/ammlu/ammlu_professional_accounting.yaml' 2024-03-18T14:20:39,778 adding 'lm_eval/tasks/ammlu/ammlu_professional_law.yaml' 2024-03-18T14:20:39,779 adding 'lm_eval/tasks/ammlu/ammlu_professional_medicine.yaml' 2024-03-18T14:20:39,780 adding 'lm_eval/tasks/ammlu/ammlu_professional_psychology.yaml' 2024-03-18T14:20:39,782 adding 'lm_eval/tasks/ammlu/ammlu_public_relations.yaml' 2024-03-18T14:20:39,783 adding 'lm_eval/tasks/ammlu/ammlu_security_studies.yaml' 2024-03-18T14:20:39,784 adding 'lm_eval/tasks/ammlu/ammlu_sociology.yaml' 2024-03-18T14:20:39,785 adding 'lm_eval/tasks/ammlu/ammlu_us_foreign_policy.yaml' 2024-03-18T14:20:39,786 adding 'lm_eval/tasks/ammlu/ammlu_virology.yaml' 2024-03-18T14:20:39,787 adding 'lm_eval/tasks/ammlu/ammlu_world_religions.yaml' 2024-03-18T14:20:39,789 adding 'lm_eval/tasks/anli/README.md' 2024-03-18T14:20:39,790 adding 'lm_eval/tasks/anli/anli_r1.yaml' 2024-03-18T14:20:39,791 adding 'lm_eval/tasks/anli/anli_r2.yaml' 2024-03-18T14:20:39,792 adding 'lm_eval/tasks/anli/anli_r3.yaml' 2024-03-18T14:20:39,794 adding 'lm_eval/tasks/arc/README.md' 2024-03-18T14:20:39,795 adding 'lm_eval/tasks/arc/arc_challenge.yaml' 2024-03-18T14:20:39,796 adding 'lm_eval/tasks/arc/arc_easy.yaml' 2024-03-18T14:20:39,798 adding 'lm_eval/tasks/arithmetic/README.md' 2024-03-18T14:20:39,799 adding 'lm_eval/tasks/arithmetic/arithmetic_1dc.yaml' 2024-03-18T14:20:39,800 adding 'lm_eval/tasks/arithmetic/arithmetic_2da.yaml' 2024-03-18T14:20:39,802 adding 'lm_eval/tasks/arithmetic/arithmetic_2dm.yaml' 2024-03-18T14:20:39,803 adding 'lm_eval/tasks/arithmetic/arithmetic_2ds.yaml' 2024-03-18T14:20:39,803 adding 'lm_eval/tasks/arithmetic/arithmetic_3da.yaml' 2024-03-18T14:20:39,804 adding 'lm_eval/tasks/arithmetic/arithmetic_3ds.yaml' 2024-03-18T14:20:39,806 adding 'lm_eval/tasks/arithmetic/arithmetic_4da.yaml' 2024-03-18T14:20:39,807 adding 'lm_eval/tasks/arithmetic/arithmetic_4ds.yaml' 2024-03-18T14:20:39,808 adding 'lm_eval/tasks/arithmetic/arithmetic_5da.yaml' 2024-03-18T14:20:39,809 adding 'lm_eval/tasks/arithmetic/arithmetic_5ds.yaml' 2024-03-18T14:20:39,811 adding 'lm_eval/tasks/asdiv/README.md' 2024-03-18T14:20:39,812 adding 'lm_eval/tasks/asdiv/default.yaml' 2024-03-18T14:20:39,816 adding 'lm_eval/tasks/babi/README.md' 2024-03-18T14:20:39,817 adding 'lm_eval/tasks/babi/babi.yaml' 2024-03-18T14:20:39,819 adding 'lm_eval/tasks/bbh/README.md' 2024-03-18T14:20:39,821 adding 'lm_eval/tasks/bbh/_generate_configs.py' 2024-03-18T14:20:39,823 adding 'lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml' 2024-03-18T14:20:39,824 adding 'lm_eval/tasks/bbh/cot_fewshot/boolean_expressions.yaml' 2024-03-18T14:20:39,826 adding 'lm_eval/tasks/bbh/cot_fewshot/causal_judgement.yaml' 2024-03-18T14:20:39,827 adding 'lm_eval/tasks/bbh/cot_fewshot/date_understanding.yaml' 2024-03-18T14:20:39,828 adding 'lm_eval/tasks/bbh/cot_fewshot/disambiguation_qa.yaml' 2024-03-18T14:20:39,830 adding 'lm_eval/tasks/bbh/cot_fewshot/dyck_languages.yaml' 2024-03-18T14:20:39,831 adding 'lm_eval/tasks/bbh/cot_fewshot/formal_fallacies.yaml' 2024-03-18T14:20:39,833 adding 'lm_eval/tasks/bbh/cot_fewshot/geometric_shapes.yaml' 2024-03-18T14:20:39,834 adding 'lm_eval/tasks/bbh/cot_fewshot/hyperbaton.yaml' 2024-03-18T14:20:39,836 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_five_objects.yaml' 2024-03-18T14:20:39,837 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_seven_objects.yaml' 2024-03-18T14:20:39,838 adding 'lm_eval/tasks/bbh/cot_fewshot/logical_deduction_three_objects.yaml' 2024-03-18T14:20:39,840 adding 'lm_eval/tasks/bbh/cot_fewshot/movie_recommendation.yaml' 2024-03-18T14:20:39,841 adding 'lm_eval/tasks/bbh/cot_fewshot/multistep_arithmetic_two.yaml' 2024-03-18T14:20:39,843 adding 'lm_eval/tasks/bbh/cot_fewshot/navigate.yaml' 2024-03-18T14:20:39,844 adding 'lm_eval/tasks/bbh/cot_fewshot/object_counting.yaml' 2024-03-18T14:20:39,845 adding 'lm_eval/tasks/bbh/cot_fewshot/penguins_in_a_table.yaml' 2024-03-18T14:20:39,847 adding 'lm_eval/tasks/bbh/cot_fewshot/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:39,848 adding 'lm_eval/tasks/bbh/cot_fewshot/ruin_names.yaml' 2024-03-18T14:20:39,850 adding 'lm_eval/tasks/bbh/cot_fewshot/salient_translation_error_detection.yaml' 2024-03-18T14:20:39,852 adding 'lm_eval/tasks/bbh/cot_fewshot/snarks.yaml' 2024-03-18T14:20:39,853 adding 'lm_eval/tasks/bbh/cot_fewshot/sports_understanding.yaml' 2024-03-18T14:20:39,854 adding 'lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml' 2024-03-18T14:20:39,856 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_five_objects.yaml' 2024-03-18T14:20:39,857 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_seven_objects.yaml' 2024-03-18T14:20:39,859 adding 'lm_eval/tasks/bbh/cot_fewshot/tracking_shuffled_objects_three_objects.yaml' 2024-03-18T14:20:39,861 adding 'lm_eval/tasks/bbh/cot_fewshot/web_of_lies.yaml' 2024-03-18T14:20:39,862 adding 'lm_eval/tasks/bbh/cot_fewshot/word_sorting.yaml' 2024-03-18T14:20:39,865 adding 'lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml' 2024-03-18T14:20:39,866 adding 'lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml' 2024-03-18T14:20:39,867 adding 'lm_eval/tasks/bbh/cot_zeroshot/causal_judgement.yaml' 2024-03-18T14:20:39,868 adding 'lm_eval/tasks/bbh/cot_zeroshot/date_understanding.yaml' 2024-03-18T14:20:39,869 adding 'lm_eval/tasks/bbh/cot_zeroshot/disambiguation_qa.yaml' 2024-03-18T14:20:39,870 adding 'lm_eval/tasks/bbh/cot_zeroshot/dyck_languages.yaml' 2024-03-18T14:20:39,872 adding 'lm_eval/tasks/bbh/cot_zeroshot/formal_fallacies.yaml' 2024-03-18T14:20:39,873 adding 'lm_eval/tasks/bbh/cot_zeroshot/geometric_shapes.yaml' 2024-03-18T14:20:39,874 adding 'lm_eval/tasks/bbh/cot_zeroshot/hyperbaton.yaml' 2024-03-18T14:20:39,875 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_five_objects.yaml' 2024-03-18T14:20:39,877 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_seven_objects.yaml' 2024-03-18T14:20:39,878 adding 'lm_eval/tasks/bbh/cot_zeroshot/logical_deduction_three_objects.yaml' 2024-03-18T14:20:39,879 adding 'lm_eval/tasks/bbh/cot_zeroshot/movie_recommendation.yaml' 2024-03-18T14:20:39,880 adding 'lm_eval/tasks/bbh/cot_zeroshot/multistep_arithmetic_two.yaml' 2024-03-18T14:20:39,881 adding 'lm_eval/tasks/bbh/cot_zeroshot/navigate.yaml' 2024-03-18T14:20:39,882 adding 'lm_eval/tasks/bbh/cot_zeroshot/object_counting.yaml' 2024-03-18T14:20:39,884 adding 'lm_eval/tasks/bbh/cot_zeroshot/penguins_in_a_table.yaml' 2024-03-18T14:20:39,885 adding 'lm_eval/tasks/bbh/cot_zeroshot/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:39,886 adding 'lm_eval/tasks/bbh/cot_zeroshot/ruin_names.yaml' 2024-03-18T14:20:39,887 adding 'lm_eval/tasks/bbh/cot_zeroshot/salient_translation_error_detection.yaml' 2024-03-18T14:20:39,888 adding 'lm_eval/tasks/bbh/cot_zeroshot/snarks.yaml' 2024-03-18T14:20:39,890 adding 'lm_eval/tasks/bbh/cot_zeroshot/sports_understanding.yaml' 2024-03-18T14:20:39,891 adding 'lm_eval/tasks/bbh/cot_zeroshot/temporal_sequences.yaml' 2024-03-18T14:20:39,892 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_five_objects.yaml' 2024-03-18T14:20:39,894 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_seven_objects.yaml' 2024-03-18T14:20:39,895 adding 'lm_eval/tasks/bbh/cot_zeroshot/tracking_shuffled_objects_three_objects.yaml' 2024-03-18T14:20:39,897 adding 'lm_eval/tasks/bbh/cot_zeroshot/utils.py' 2024-03-18T14:20:39,898 adding 'lm_eval/tasks/bbh/cot_zeroshot/web_of_lies.yaml' 2024-03-18T14:20:39,900 adding 'lm_eval/tasks/bbh/cot_zeroshot/word_sorting.yaml' 2024-03-18T14:20:39,902 adding 'lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml' 2024-03-18T14:20:39,904 adding 'lm_eval/tasks/bbh/fewshot/boolean_expressions.yaml' 2024-03-18T14:20:39,905 adding 'lm_eval/tasks/bbh/fewshot/causal_judgement.yaml' 2024-03-18T14:20:39,907 adding 'lm_eval/tasks/bbh/fewshot/date_understanding.yaml' 2024-03-18T14:20:39,908 adding 'lm_eval/tasks/bbh/fewshot/disambiguation_qa.yaml' 2024-03-18T14:20:39,909 adding 'lm_eval/tasks/bbh/fewshot/dyck_languages.yaml' 2024-03-18T14:20:39,911 adding 'lm_eval/tasks/bbh/fewshot/formal_fallacies.yaml' 2024-03-18T14:20:39,912 adding 'lm_eval/tasks/bbh/fewshot/geometric_shapes.yaml' 2024-03-18T14:20:39,913 adding 'lm_eval/tasks/bbh/fewshot/hyperbaton.yaml' 2024-03-18T14:20:39,914 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_five_objects.yaml' 2024-03-18T14:20:39,915 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_seven_objects.yaml' 2024-03-18T14:20:39,917 adding 'lm_eval/tasks/bbh/fewshot/logical_deduction_three_objects.yaml' 2024-03-18T14:20:39,918 adding 'lm_eval/tasks/bbh/fewshot/movie_recommendation.yaml' 2024-03-18T14:20:39,919 adding 'lm_eval/tasks/bbh/fewshot/multistep_arithmetic_two.yaml' 2024-03-18T14:20:39,920 adding 'lm_eval/tasks/bbh/fewshot/navigate.yaml' 2024-03-18T14:20:39,921 adding 'lm_eval/tasks/bbh/fewshot/object_counting.yaml' 2024-03-18T14:20:39,922 adding 'lm_eval/tasks/bbh/fewshot/penguins_in_a_table.yaml' 2024-03-18T14:20:39,924 adding 'lm_eval/tasks/bbh/fewshot/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:39,925 adding 'lm_eval/tasks/bbh/fewshot/ruin_names.yaml' 2024-03-18T14:20:39,926 adding 'lm_eval/tasks/bbh/fewshot/salient_translation_error_detection.yaml' 2024-03-18T14:20:39,927 adding 'lm_eval/tasks/bbh/fewshot/snarks.yaml' 2024-03-18T14:20:39,929 adding 'lm_eval/tasks/bbh/fewshot/sports_understanding.yaml' 2024-03-18T14:20:39,930 adding 'lm_eval/tasks/bbh/fewshot/temporal_sequences.yaml' 2024-03-18T14:20:39,931 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_five_objects.yaml' 2024-03-18T14:20:39,932 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_seven_objects.yaml' 2024-03-18T14:20:39,934 adding 'lm_eval/tasks/bbh/fewshot/tracking_shuffled_objects_three_objects.yaml' 2024-03-18T14:20:39,935 adding 'lm_eval/tasks/bbh/fewshot/web_of_lies.yaml' 2024-03-18T14:20:39,936 adding 'lm_eval/tasks/bbh/fewshot/word_sorting.yaml' 2024-03-18T14:20:39,938 adding 'lm_eval/tasks/bbh/zeroshot/_zeroshot_template_yaml' 2024-03-18T14:20:39,939 adding 'lm_eval/tasks/bbh/zeroshot/boolean_expressions.yaml' 2024-03-18T14:20:39,940 adding 'lm_eval/tasks/bbh/zeroshot/causal_judgement.yaml' 2024-03-18T14:20:39,942 adding 'lm_eval/tasks/bbh/zeroshot/date_understanding.yaml' 2024-03-18T14:20:39,943 adding 'lm_eval/tasks/bbh/zeroshot/disambiguation_qa.yaml' 2024-03-18T14:20:39,944 adding 'lm_eval/tasks/bbh/zeroshot/dyck_languages.yaml' 2024-03-18T14:20:39,945 adding 'lm_eval/tasks/bbh/zeroshot/formal_fallacies.yaml' 2024-03-18T14:20:39,946 adding 'lm_eval/tasks/bbh/zeroshot/geometric_shapes.yaml' 2024-03-18T14:20:39,947 adding 'lm_eval/tasks/bbh/zeroshot/hyperbaton.yaml' 2024-03-18T14:20:39,948 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_five_objects.yaml' 2024-03-18T14:20:39,950 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_seven_objects.yaml' 2024-03-18T14:20:39,951 adding 'lm_eval/tasks/bbh/zeroshot/logical_deduction_three_objects.yaml' 2024-03-18T14:20:39,952 adding 'lm_eval/tasks/bbh/zeroshot/movie_recommendation.yaml' 2024-03-18T14:20:39,954 adding 'lm_eval/tasks/bbh/zeroshot/multistep_arithmetic_two.yaml' 2024-03-18T14:20:39,955 adding 'lm_eval/tasks/bbh/zeroshot/navigate.yaml' 2024-03-18T14:20:39,956 adding 'lm_eval/tasks/bbh/zeroshot/object_counting.yaml' 2024-03-18T14:20:39,957 adding 'lm_eval/tasks/bbh/zeroshot/penguins_in_a_table.yaml' 2024-03-18T14:20:39,958 adding 'lm_eval/tasks/bbh/zeroshot/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:39,960 adding 'lm_eval/tasks/bbh/zeroshot/ruin_names.yaml' 2024-03-18T14:20:39,961 adding 'lm_eval/tasks/bbh/zeroshot/salient_translation_error_detection.yaml' 2024-03-18T14:20:39,962 adding 'lm_eval/tasks/bbh/zeroshot/snarks.yaml' 2024-03-18T14:20:39,963 adding 'lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml' 2024-03-18T14:20:39,965 adding 'lm_eval/tasks/bbh/zeroshot/temporal_sequences.yaml' 2024-03-18T14:20:39,966 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_five_objects.yaml' 2024-03-18T14:20:39,967 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_seven_objects.yaml' 2024-03-18T14:20:39,968 adding 'lm_eval/tasks/bbh/zeroshot/tracking_shuffled_objects_three_objects.yaml' 2024-03-18T14:20:39,970 adding 'lm_eval/tasks/bbh/zeroshot/utils.py' 2024-03-18T14:20:39,971 adding 'lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml' 2024-03-18T14:20:39,973 adding 'lm_eval/tasks/bbh/zeroshot/word_sorting.yaml' 2024-03-18T14:20:39,977 adding 'lm_eval/tasks/belebele/README.md' 2024-03-18T14:20:39,978 adding 'lm_eval/tasks/belebele/_default_template_yaml' 2024-03-18T14:20:39,979 adding 'lm_eval/tasks/belebele/_generate_configs.py' 2024-03-18T14:20:39,980 adding 'lm_eval/tasks/belebele/belebele_acm_Arab.yaml' 2024-03-18T14:20:39,982 adding 'lm_eval/tasks/belebele/belebele_afr_Latn.yaml' 2024-03-18T14:20:39,983 adding 'lm_eval/tasks/belebele/belebele_als_Latn.yaml' 2024-03-18T14:20:39,984 adding 'lm_eval/tasks/belebele/belebele_amh_Ethi.yaml' 2024-03-18T14:20:39,985 adding 'lm_eval/tasks/belebele/belebele_apc_Arab.yaml' 2024-03-18T14:20:39,986 adding 'lm_eval/tasks/belebele/belebele_arb_Arab.yaml' 2024-03-18T14:20:39,987 adding 'lm_eval/tasks/belebele/belebele_arb_Latn.yaml' 2024-03-18T14:20:39,988 adding 'lm_eval/tasks/belebele/belebele_ars_Arab.yaml' 2024-03-18T14:20:39,989 adding 'lm_eval/tasks/belebele/belebele_ary_Arab.yaml' 2024-03-18T14:20:39,990 adding 'lm_eval/tasks/belebele/belebele_arz_Arab.yaml' 2024-03-18T14:20:39,991 adding 'lm_eval/tasks/belebele/belebele_asm_Beng.yaml' 2024-03-18T14:20:39,992 adding 'lm_eval/tasks/belebele/belebele_azj_Latn.yaml' 2024-03-18T14:20:39,993 adding 'lm_eval/tasks/belebele/belebele_bam_Latn.yaml' 2024-03-18T14:20:39,995 adding 'lm_eval/tasks/belebele/belebele_ben_Beng.yaml' 2024-03-18T14:20:39,996 adding 'lm_eval/tasks/belebele/belebele_ben_Latn.yaml' 2024-03-18T14:20:39,997 adding 'lm_eval/tasks/belebele/belebele_bod_Tibt.yaml' 2024-03-18T14:20:39,998 adding 'lm_eval/tasks/belebele/belebele_bul_Cyrl.yaml' 2024-03-18T14:20:39,999 adding 'lm_eval/tasks/belebele/belebele_cat_Latn.yaml' 2024-03-18T14:20:40,000 adding 'lm_eval/tasks/belebele/belebele_ceb_Latn.yaml' 2024-03-18T14:20:40,001 adding 'lm_eval/tasks/belebele/belebele_ces_Latn.yaml' 2024-03-18T14:20:40,002 adding 'lm_eval/tasks/belebele/belebele_ckb_Arab.yaml' 2024-03-18T14:20:40,003 adding 'lm_eval/tasks/belebele/belebele_dan_Latn.yaml' 2024-03-18T14:20:40,005 adding 'lm_eval/tasks/belebele/belebele_deu_Latn.yaml' 2024-03-18T14:20:40,006 adding 'lm_eval/tasks/belebele/belebele_ell_Grek.yaml' 2024-03-18T14:20:40,007 adding 'lm_eval/tasks/belebele/belebele_eng_Latn.yaml' 2024-03-18T14:20:40,008 adding 'lm_eval/tasks/belebele/belebele_est_Latn.yaml' 2024-03-18T14:20:40,009 adding 'lm_eval/tasks/belebele/belebele_eus_Latn.yaml' 2024-03-18T14:20:40,011 adding 'lm_eval/tasks/belebele/belebele_fin_Latn.yaml' 2024-03-18T14:20:40,012 adding 'lm_eval/tasks/belebele/belebele_fra_Latn.yaml' 2024-03-18T14:20:40,013 adding 'lm_eval/tasks/belebele/belebele_fuv_Latn.yaml' 2024-03-18T14:20:40,014 adding 'lm_eval/tasks/belebele/belebele_gaz_Latn.yaml' 2024-03-18T14:20:40,015 adding 'lm_eval/tasks/belebele/belebele_grn_Latn.yaml' 2024-03-18T14:20:40,017 adding 'lm_eval/tasks/belebele/belebele_guj_Gujr.yaml' 2024-03-18T14:20:40,018 adding 'lm_eval/tasks/belebele/belebele_hat_Latn.yaml' 2024-03-18T14:20:40,019 adding 'lm_eval/tasks/belebele/belebele_hau_Latn.yaml' 2024-03-18T14:20:40,020 adding 'lm_eval/tasks/belebele/belebele_heb_Hebr.yaml' 2024-03-18T14:20:40,021 adding 'lm_eval/tasks/belebele/belebele_hin_Deva.yaml' 2024-03-18T14:20:40,022 adding 'lm_eval/tasks/belebele/belebele_hin_Latn.yaml' 2024-03-18T14:20:40,023 adding 'lm_eval/tasks/belebele/belebele_hrv_Latn.yaml' 2024-03-18T14:20:40,025 adding 'lm_eval/tasks/belebele/belebele_hun_Latn.yaml' 2024-03-18T14:20:40,026 adding 'lm_eval/tasks/belebele/belebele_hye_Armn.yaml' 2024-03-18T14:20:40,027 adding 'lm_eval/tasks/belebele/belebele_ibo_Latn.yaml' 2024-03-18T14:20:40,028 adding 'lm_eval/tasks/belebele/belebele_ilo_Latn.yaml' 2024-03-18T14:20:40,029 adding 'lm_eval/tasks/belebele/belebele_ind_Latn.yaml' 2024-03-18T14:20:40,030 adding 'lm_eval/tasks/belebele/belebele_isl_Latn.yaml' 2024-03-18T14:20:40,031 adding 'lm_eval/tasks/belebele/belebele_ita_Latn.yaml' 2024-03-18T14:20:40,033 adding 'lm_eval/tasks/belebele/belebele_jav_Latn.yaml' 2024-03-18T14:20:40,034 adding 'lm_eval/tasks/belebele/belebele_jpn_Jpan.yaml' 2024-03-18T14:20:40,035 adding 'lm_eval/tasks/belebele/belebele_kac_Latn.yaml' 2024-03-18T14:20:40,036 adding 'lm_eval/tasks/belebele/belebele_kan_Knda.yaml' 2024-03-18T14:20:40,037 adding 'lm_eval/tasks/belebele/belebele_kat_Geor.yaml' 2024-03-18T14:20:40,038 adding 'lm_eval/tasks/belebele/belebele_kaz_Cyrl.yaml' 2024-03-18T14:20:40,039 adding 'lm_eval/tasks/belebele/belebele_kea_Latn.yaml' 2024-03-18T14:20:40,040 adding 'lm_eval/tasks/belebele/belebele_khk_Cyrl.yaml' 2024-03-18T14:20:40,041 adding 'lm_eval/tasks/belebele/belebele_khm_Khmr.yaml' 2024-03-18T14:20:40,042 adding 'lm_eval/tasks/belebele/belebele_kin_Latn.yaml' 2024-03-18T14:20:40,043 adding 'lm_eval/tasks/belebele/belebele_kir_Cyrl.yaml' 2024-03-18T14:20:40,044 adding 'lm_eval/tasks/belebele/belebele_kor_Hang.yaml' 2024-03-18T14:20:40,045 adding 'lm_eval/tasks/belebele/belebele_lao_Laoo.yaml' 2024-03-18T14:20:40,046 adding 'lm_eval/tasks/belebele/belebele_lin_Latn.yaml' 2024-03-18T14:20:40,047 adding 'lm_eval/tasks/belebele/belebele_lit_Latn.yaml' 2024-03-18T14:20:40,049 adding 'lm_eval/tasks/belebele/belebele_lug_Latn.yaml' 2024-03-18T14:20:40,050 adding 'lm_eval/tasks/belebele/belebele_luo_Latn.yaml' 2024-03-18T14:20:40,051 adding 'lm_eval/tasks/belebele/belebele_lvs_Latn.yaml' 2024-03-18T14:20:40,052 adding 'lm_eval/tasks/belebele/belebele_mal_Mlym.yaml' 2024-03-18T14:20:40,053 adding 'lm_eval/tasks/belebele/belebele_mar_Deva.yaml' 2024-03-18T14:20:40,054 adding 'lm_eval/tasks/belebele/belebele_mkd_Cyrl.yaml' 2024-03-18T14:20:40,055 adding 'lm_eval/tasks/belebele/belebele_mlt_Latn.yaml' 2024-03-18T14:20:40,056 adding 'lm_eval/tasks/belebele/belebele_mri_Latn.yaml' 2024-03-18T14:20:40,057 adding 'lm_eval/tasks/belebele/belebele_mya_Mymr.yaml' 2024-03-18T14:20:40,058 adding 'lm_eval/tasks/belebele/belebele_nld_Latn.yaml' 2024-03-18T14:20:40,059 adding 'lm_eval/tasks/belebele/belebele_nob_Latn.yaml' 2024-03-18T14:20:40,060 adding 'lm_eval/tasks/belebele/belebele_npi_Deva.yaml' 2024-03-18T14:20:40,062 adding 'lm_eval/tasks/belebele/belebele_npi_Latn.yaml' 2024-03-18T14:20:40,063 adding 'lm_eval/tasks/belebele/belebele_nso_Latn.yaml' 2024-03-18T14:20:40,064 adding 'lm_eval/tasks/belebele/belebele_nya_Latn.yaml' 2024-03-18T14:20:40,065 adding 'lm_eval/tasks/belebele/belebele_ory_Orya.yaml' 2024-03-18T14:20:40,066 adding 'lm_eval/tasks/belebele/belebele_pan_Guru.yaml' 2024-03-18T14:20:40,067 adding 'lm_eval/tasks/belebele/belebele_pbt_Arab.yaml' 2024-03-18T14:20:40,068 adding 'lm_eval/tasks/belebele/belebele_pes_Arab.yaml' 2024-03-18T14:20:40,069 adding 'lm_eval/tasks/belebele/belebele_plt_Latn.yaml' 2024-03-18T14:20:40,070 adding 'lm_eval/tasks/belebele/belebele_pol_Latn.yaml' 2024-03-18T14:20:40,072 adding 'lm_eval/tasks/belebele/belebele_por_Latn.yaml' 2024-03-18T14:20:40,073 adding 'lm_eval/tasks/belebele/belebele_ron_Latn.yaml' 2024-03-18T14:20:40,074 adding 'lm_eval/tasks/belebele/belebele_rus_Cyrl.yaml' 2024-03-18T14:20:40,075 adding 'lm_eval/tasks/belebele/belebele_shn_Mymr.yaml' 2024-03-18T14:20:40,076 adding 'lm_eval/tasks/belebele/belebele_sin_Latn.yaml' 2024-03-18T14:20:40,077 adding 'lm_eval/tasks/belebele/belebele_sin_Sinh.yaml' 2024-03-18T14:20:40,078 adding 'lm_eval/tasks/belebele/belebele_slk_Latn.yaml' 2024-03-18T14:20:40,080 adding 'lm_eval/tasks/belebele/belebele_slv_Latn.yaml' 2024-03-18T14:20:40,081 adding 'lm_eval/tasks/belebele/belebele_sna_Latn.yaml' 2024-03-18T14:20:40,082 adding 'lm_eval/tasks/belebele/belebele_snd_Arab.yaml' 2024-03-18T14:20:40,083 adding 'lm_eval/tasks/belebele/belebele_som_Latn.yaml' 2024-03-18T14:20:40,084 adding 'lm_eval/tasks/belebele/belebele_sot_Latn.yaml' 2024-03-18T14:20:40,085 adding 'lm_eval/tasks/belebele/belebele_spa_Latn.yaml' 2024-03-18T14:20:40,086 adding 'lm_eval/tasks/belebele/belebele_srp_Cyrl.yaml' 2024-03-18T14:20:40,087 adding 'lm_eval/tasks/belebele/belebele_ssw_Latn.yaml' 2024-03-18T14:20:40,088 adding 'lm_eval/tasks/belebele/belebele_sun_Latn.yaml' 2024-03-18T14:20:40,090 adding 'lm_eval/tasks/belebele/belebele_swe_Latn.yaml' 2024-03-18T14:20:40,091 adding 'lm_eval/tasks/belebele/belebele_swh_Latn.yaml' 2024-03-18T14:20:40,092 adding 'lm_eval/tasks/belebele/belebele_tam_Taml.yaml' 2024-03-18T14:20:40,093 adding 'lm_eval/tasks/belebele/belebele_tel_Telu.yaml' 2024-03-18T14:20:40,094 adding 'lm_eval/tasks/belebele/belebele_tgk_Cyrl.yaml' 2024-03-18T14:20:40,095 adding 'lm_eval/tasks/belebele/belebele_tgl_Latn.yaml' 2024-03-18T14:20:40,096 adding 'lm_eval/tasks/belebele/belebele_tha_Thai.yaml' 2024-03-18T14:20:40,097 adding 'lm_eval/tasks/belebele/belebele_tir_Ethi.yaml' 2024-03-18T14:20:40,098 adding 'lm_eval/tasks/belebele/belebele_tsn_Latn.yaml' 2024-03-18T14:20:40,099 adding 'lm_eval/tasks/belebele/belebele_tso_Latn.yaml' 2024-03-18T14:20:40,100 adding 'lm_eval/tasks/belebele/belebele_tur_Latn.yaml' 2024-03-18T14:20:40,101 adding 'lm_eval/tasks/belebele/belebele_ukr_Cyrl.yaml' 2024-03-18T14:20:40,102 adding 'lm_eval/tasks/belebele/belebele_urd_Arab.yaml' 2024-03-18T14:20:40,103 adding 'lm_eval/tasks/belebele/belebele_urd_Latn.yaml' 2024-03-18T14:20:40,104 adding 'lm_eval/tasks/belebele/belebele_uzn_Latn.yaml' 2024-03-18T14:20:40,106 adding 'lm_eval/tasks/belebele/belebele_vie_Latn.yaml' 2024-03-18T14:20:40,107 adding 'lm_eval/tasks/belebele/belebele_war_Latn.yaml' 2024-03-18T14:20:40,108 adding 'lm_eval/tasks/belebele/belebele_wol_Latn.yaml' 2024-03-18T14:20:40,109 adding 'lm_eval/tasks/belebele/belebele_xho_Latn.yaml' 2024-03-18T14:20:40,110 adding 'lm_eval/tasks/belebele/belebele_yor_Latn.yaml' 2024-03-18T14:20:40,111 adding 'lm_eval/tasks/belebele/belebele_zho_Hans.yaml' 2024-03-18T14:20:40,112 adding 'lm_eval/tasks/belebele/belebele_zho_Hant.yaml' 2024-03-18T14:20:40,113 adding 'lm_eval/tasks/belebele/belebele_zsm_Latn.yaml' 2024-03-18T14:20:40,114 adding 'lm_eval/tasks/belebele/belebele_zul_Latn.yaml' 2024-03-18T14:20:40,115 adding 'lm_eval/tasks/benchmarks/minerva_math.yaml' 2024-03-18T14:20:40,117 adding 'lm_eval/tasks/benchmarks/openllm.yaml' 2024-03-18T14:20:40,118 adding 'lm_eval/tasks/benchmarks/pythia.yaml' 2024-03-18T14:20:40,119 adding 'lm_eval/tasks/benchmarks/t0_eval.yaml' 2024-03-18T14:20:40,121 adding 'lm_eval/tasks/benchmarks/flan/_held_in_template_yaml' 2024-03-18T14:20:40,123 adding 'lm_eval/tasks/benchmarks/flan/flan_held_in.yaml' 2024-03-18T14:20:40,124 adding 'lm_eval/tasks/benchmarks/flan/flan_held_out.yaml' 2024-03-18T14:20:40,125 adding 'lm_eval/tasks/benchmarks/multimedqa/README.md' 2024-03-18T14:20:40,127 adding 'lm_eval/tasks/benchmarks/multimedqa/multimedqa.yaml' 2024-03-18T14:20:40,130 adding 'lm_eval/tasks/bigbench/README.md' 2024-03-18T14:20:40,131 adding 'lm_eval/tasks/bigbench/generate_tasks.py' 2024-03-18T14:20:40,132 adding 'lm_eval/tasks/bigbench/generate_until_template_yaml' 2024-03-18T14:20:40,134 adding 'lm_eval/tasks/bigbench/multiple_choice_template_yaml' 2024-03-18T14:20:40,135 adding 'lm_eval/tasks/bigbench/push_bigbench_dataset.py' 2024-03-18T14:20:40,141 adding 'lm_eval/tasks/bigbench/generate_until/abstract_narrative_understanding.yaml' 2024-03-18T14:20:40,142 adding 'lm_eval/tasks/bigbench/generate_until/anachronisms.yaml' 2024-03-18T14:20:40,144 adding 'lm_eval/tasks/bigbench/generate_until/analogical_similarity.yaml' 2024-03-18T14:20:40,145 adding 'lm_eval/tasks/bigbench/generate_until/analytic_entailment.yaml' 2024-03-18T14:20:40,146 adding 'lm_eval/tasks/bigbench/generate_until/arithmetic.yaml' 2024-03-18T14:20:40,147 adding 'lm_eval/tasks/bigbench/generate_until/ascii_word_recognition.yaml' 2024-03-18T14:20:40,149 adding 'lm_eval/tasks/bigbench/generate_until/authorship_verification.yaml' 2024-03-18T14:20:40,150 adding 'lm_eval/tasks/bigbench/generate_until/auto_categorization.yaml' 2024-03-18T14:20:40,151 adding 'lm_eval/tasks/bigbench/generate_until/auto_debugging.yaml' 2024-03-18T14:20:40,152 adding 'lm_eval/tasks/bigbench/generate_until/bbq_lite_json.yaml' 2024-03-18T14:20:40,153 adding 'lm_eval/tasks/bigbench/generate_until/bridging_anaphora_resolution_barqa.yaml' 2024-03-18T14:20:40,155 adding 'lm_eval/tasks/bigbench/generate_until/causal_judgment.yaml' 2024-03-18T14:20:40,156 adding 'lm_eval/tasks/bigbench/generate_until/cause_and_effect.yaml' 2024-03-18T14:20:40,157 adding 'lm_eval/tasks/bigbench/generate_until/checkmate_in_one.yaml' 2024-03-18T14:20:40,158 adding 'lm_eval/tasks/bigbench/generate_until/chess_state_tracking.yaml' 2024-03-18T14:20:40,159 adding 'lm_eval/tasks/bigbench/generate_until/chinese_remainder_theorem.yaml' 2024-03-18T14:20:40,161 adding 'lm_eval/tasks/bigbench/generate_until/cifar10_classification.yaml' 2024-03-18T14:20:40,162 adding 'lm_eval/tasks/bigbench/generate_until/code_line_description.yaml' 2024-03-18T14:20:40,163 adding 'lm_eval/tasks/bigbench/generate_until/codenames.yaml' 2024-03-18T14:20:40,164 adding 'lm_eval/tasks/bigbench/generate_until/color.yaml' 2024-03-18T14:20:40,165 adding 'lm_eval/tasks/bigbench/generate_until/common_morpheme.yaml' 2024-03-18T14:20:40,166 adding 'lm_eval/tasks/bigbench/generate_until/conceptual_combinations.yaml' 2024-03-18T14:20:40,167 adding 'lm_eval/tasks/bigbench/generate_until/conlang_translation.yaml' 2024-03-18T14:20:40,168 adding 'lm_eval/tasks/bigbench/generate_until/contextual_parametric_knowledge_conflicts.yaml' 2024-03-18T14:20:40,170 adding 'lm_eval/tasks/bigbench/generate_until/crash_blossom.yaml' 2024-03-18T14:20:40,171 adding 'lm_eval/tasks/bigbench/generate_until/crass_ai.yaml' 2024-03-18T14:20:40,172 adding 'lm_eval/tasks/bigbench/generate_until/cryobiology_spanish.yaml' 2024-03-18T14:20:40,173 adding 'lm_eval/tasks/bigbench/generate_until/cryptonite.yaml' 2024-03-18T14:20:40,175 adding 'lm_eval/tasks/bigbench/generate_until/cs_algorithms.yaml' 2024-03-18T14:20:40,176 adding 'lm_eval/tasks/bigbench/generate_until/dark_humor_detection.yaml' 2024-03-18T14:20:40,177 adding 'lm_eval/tasks/bigbench/generate_until/date_understanding.yaml' 2024-03-18T14:20:40,178 adding 'lm_eval/tasks/bigbench/generate_until/disambiguation_qa.yaml' 2024-03-18T14:20:40,179 adding 'lm_eval/tasks/bigbench/generate_until/discourse_marker_prediction.yaml' 2024-03-18T14:20:40,181 adding 'lm_eval/tasks/bigbench/generate_until/disfl_qa.yaml' 2024-03-18T14:20:40,182 adding 'lm_eval/tasks/bigbench/generate_until/dyck_languages.yaml' 2024-03-18T14:20:40,183 adding 'lm_eval/tasks/bigbench/generate_until/elementary_math_qa.yaml' 2024-03-18T14:20:40,184 adding 'lm_eval/tasks/bigbench/generate_until/emoji_movie.yaml' 2024-03-18T14:20:40,185 adding 'lm_eval/tasks/bigbench/generate_until/emojis_emotion_prediction.yaml' 2024-03-18T14:20:40,187 adding 'lm_eval/tasks/bigbench/generate_until/empirical_judgments.yaml' 2024-03-18T14:20:40,188 adding 'lm_eval/tasks/bigbench/generate_until/english_proverbs.yaml' 2024-03-18T14:20:40,189 adding 'lm_eval/tasks/bigbench/generate_until/english_russian_proverbs.yaml' 2024-03-18T14:20:40,190 adding 'lm_eval/tasks/bigbench/generate_until/entailed_polarity.yaml' 2024-03-18T14:20:40,191 adding 'lm_eval/tasks/bigbench/generate_until/entailed_polarity_hindi.yaml' 2024-03-18T14:20:40,192 adding 'lm_eval/tasks/bigbench/generate_until/epistemic_reasoning.yaml' 2024-03-18T14:20:40,194 adding 'lm_eval/tasks/bigbench/generate_until/evaluating_information_essentiality.yaml' 2024-03-18T14:20:40,195 adding 'lm_eval/tasks/bigbench/generate_until/fact_checker.yaml' 2024-03-18T14:20:40,196 adding 'lm_eval/tasks/bigbench/generate_until/fantasy_reasoning.yaml' 2024-03-18T14:20:40,197 adding 'lm_eval/tasks/bigbench/generate_until/few_shot_nlg.yaml' 2024-03-18T14:20:40,199 adding 'lm_eval/tasks/bigbench/generate_until/figure_of_speech_detection.yaml' 2024-03-18T14:20:40,200 adding 'lm_eval/tasks/bigbench/generate_until/formal_fallacies_syllogisms_negation.yaml' 2024-03-18T14:20:40,201 adding 'lm_eval/tasks/bigbench/generate_until/gem.yaml' 2024-03-18T14:20:40,202 adding 'lm_eval/tasks/bigbench/generate_until/gender_inclusive_sentences_german.yaml' 2024-03-18T14:20:40,204 adding 'lm_eval/tasks/bigbench/generate_until/general_knowledge.yaml' 2024-03-18T14:20:40,205 adding 'lm_eval/tasks/bigbench/generate_until/geometric_shapes.yaml' 2024-03-18T14:20:40,206 adding 'lm_eval/tasks/bigbench/generate_until/goal_step_wikihow.yaml' 2024-03-18T14:20:40,207 adding 'lm_eval/tasks/bigbench/generate_until/gre_reading_comprehension.yaml' 2024-03-18T14:20:40,208 adding 'lm_eval/tasks/bigbench/generate_until/hhh_alignment.yaml' 2024-03-18T14:20:40,210 adding 'lm_eval/tasks/bigbench/generate_until/hindi_question_answering.yaml' 2024-03-18T14:20:40,211 adding 'lm_eval/tasks/bigbench/generate_until/hindu_knowledge.yaml' 2024-03-18T14:20:40,212 adding 'lm_eval/tasks/bigbench/generate_until/hinglish_toxicity.yaml' 2024-03-18T14:20:40,213 adding 'lm_eval/tasks/bigbench/generate_until/human_organs_senses.yaml' 2024-03-18T14:20:40,214 adding 'lm_eval/tasks/bigbench/generate_until/hyperbaton.yaml' 2024-03-18T14:20:40,216 adding 'lm_eval/tasks/bigbench/generate_until/identify_math_theorems.yaml' 2024-03-18T14:20:40,217 adding 'lm_eval/tasks/bigbench/generate_until/identify_odd_metaphor.yaml' 2024-03-18T14:20:40,218 adding 'lm_eval/tasks/bigbench/generate_until/implicatures.yaml' 2024-03-18T14:20:40,219 adding 'lm_eval/tasks/bigbench/generate_until/implicit_relations.yaml' 2024-03-18T14:20:40,220 adding 'lm_eval/tasks/bigbench/generate_until/intent_recognition.yaml' 2024-03-18T14:20:40,222 adding 'lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_nli.yaml' 2024-03-18T14:20:40,223 adding 'lm_eval/tasks/bigbench/generate_until/international_phonetic_alphabet_transliterate.yaml' 2024-03-18T14:20:40,224 adding 'lm_eval/tasks/bigbench/generate_until/intersect_geometry.yaml' 2024-03-18T14:20:40,225 adding 'lm_eval/tasks/bigbench/generate_until/irony_identification.yaml' 2024-03-18T14:20:40,227 adding 'lm_eval/tasks/bigbench/generate_until/kanji_ascii.yaml' 2024-03-18T14:20:40,228 adding 'lm_eval/tasks/bigbench/generate_until/kannada.yaml' 2024-03-18T14:20:40,229 adding 'lm_eval/tasks/bigbench/generate_until/key_value_maps.yaml' 2024-03-18T14:20:40,231 adding 'lm_eval/tasks/bigbench/generate_until/known_unknowns.yaml' 2024-03-18T14:20:40,232 adding 'lm_eval/tasks/bigbench/generate_until/language_games.yaml' 2024-03-18T14:20:40,233 adding 'lm_eval/tasks/bigbench/generate_until/language_identification.yaml' 2024-03-18T14:20:40,234 adding 'lm_eval/tasks/bigbench/generate_until/linguistic_mappings.yaml' 2024-03-18T14:20:40,235 adding 'lm_eval/tasks/bigbench/generate_until/linguistics_puzzles.yaml' 2024-03-18T14:20:40,237 adding 'lm_eval/tasks/bigbench/generate_until/list_functions.yaml' 2024-03-18T14:20:40,238 adding 'lm_eval/tasks/bigbench/generate_until/logic_grid_puzzle.yaml' 2024-03-18T14:20:40,239 adding 'lm_eval/tasks/bigbench/generate_until/logical_args.yaml' 2024-03-18T14:20:40,240 adding 'lm_eval/tasks/bigbench/generate_until/logical_deduction.yaml' 2024-03-18T14:20:40,242 adding 'lm_eval/tasks/bigbench/generate_until/logical_fallacy_detection.yaml' 2024-03-18T14:20:40,243 adding 'lm_eval/tasks/bigbench/generate_until/logical_sequence.yaml' 2024-03-18T14:20:40,244 adding 'lm_eval/tasks/bigbench/generate_until/mathematical_induction.yaml' 2024-03-18T14:20:40,246 adding 'lm_eval/tasks/bigbench/generate_until/matrixshapes.yaml' 2024-03-18T14:20:40,247 adding 'lm_eval/tasks/bigbench/generate_until/metaphor_boolean.yaml' 2024-03-18T14:20:40,248 adding 'lm_eval/tasks/bigbench/generate_until/metaphor_understanding.yaml' 2024-03-18T14:20:40,249 adding 'lm_eval/tasks/bigbench/generate_until/minute_mysteries_qa.yaml' 2024-03-18T14:20:40,251 adding 'lm_eval/tasks/bigbench/generate_until/misconceptions.yaml' 2024-03-18T14:20:40,252 adding 'lm_eval/tasks/bigbench/generate_until/misconceptions_russian.yaml' 2024-03-18T14:20:40,253 adding 'lm_eval/tasks/bigbench/generate_until/mnist_ascii.yaml' 2024-03-18T14:20:40,254 adding 'lm_eval/tasks/bigbench/generate_until/modified_arithmetic.yaml' 2024-03-18T14:20:40,255 adding 'lm_eval/tasks/bigbench/generate_until/moral_permissibility.yaml' 2024-03-18T14:20:40,257 adding 'lm_eval/tasks/bigbench/generate_until/movie_dialog_same_or_different.yaml' 2024-03-18T14:20:40,258 adding 'lm_eval/tasks/bigbench/generate_until/movie_recommendation.yaml' 2024-03-18T14:20:40,259 adding 'lm_eval/tasks/bigbench/generate_until/mult_data_wrangling.yaml' 2024-03-18T14:20:40,260 adding 'lm_eval/tasks/bigbench/generate_until/multiemo.yaml' 2024-03-18T14:20:40,262 adding 'lm_eval/tasks/bigbench/generate_until/natural_instructions.yaml' 2024-03-18T14:20:40,263 adding 'lm_eval/tasks/bigbench/generate_until/navigate.yaml' 2024-03-18T14:20:40,264 adding 'lm_eval/tasks/bigbench/generate_until/nonsense_words_grammar.yaml' 2024-03-18T14:20:40,265 adding 'lm_eval/tasks/bigbench/generate_until/novel_concepts.yaml' 2024-03-18T14:20:40,266 adding 'lm_eval/tasks/bigbench/generate_until/object_counting.yaml' 2024-03-18T14:20:40,268 adding 'lm_eval/tasks/bigbench/generate_until/odd_one_out.yaml' 2024-03-18T14:20:40,269 adding 'lm_eval/tasks/bigbench/generate_until/operators.yaml' 2024-03-18T14:20:40,270 adding 'lm_eval/tasks/bigbench/generate_until/paragraph_segmentation.yaml' 2024-03-18T14:20:40,271 adding 'lm_eval/tasks/bigbench/generate_until/parsinlu_qa.yaml' 2024-03-18T14:20:40,272 adding 'lm_eval/tasks/bigbench/generate_until/parsinlu_reading_comprehension.yaml' 2024-03-18T14:20:40,274 adding 'lm_eval/tasks/bigbench/generate_until/penguins_in_a_table.yaml' 2024-03-18T14:20:40,275 adding 'lm_eval/tasks/bigbench/generate_until/periodic_elements.yaml' 2024-03-18T14:20:40,276 adding 'lm_eval/tasks/bigbench/generate_until/persian_idioms.yaml' 2024-03-18T14:20:40,278 adding 'lm_eval/tasks/bigbench/generate_until/phrase_relatedness.yaml' 2024-03-18T14:20:40,279 adding 'lm_eval/tasks/bigbench/generate_until/physical_intuition.yaml' 2024-03-18T14:20:40,280 adding 'lm_eval/tasks/bigbench/generate_until/physics.yaml' 2024-03-18T14:20:40,281 adding 'lm_eval/tasks/bigbench/generate_until/physics_questions.yaml' 2024-03-18T14:20:40,283 adding 'lm_eval/tasks/bigbench/generate_until/play_dialog_same_or_different.yaml' 2024-03-18T14:20:40,284 adding 'lm_eval/tasks/bigbench/generate_until/polish_sequence_labeling.yaml' 2024-03-18T14:20:40,285 adding 'lm_eval/tasks/bigbench/generate_until/presuppositions_as_nli.yaml' 2024-03-18T14:20:40,286 adding 'lm_eval/tasks/bigbench/generate_until/qa_wikidata.yaml' 2024-03-18T14:20:40,287 adding 'lm_eval/tasks/bigbench/generate_until/question_selection.yaml' 2024-03-18T14:20:40,289 adding 'lm_eval/tasks/bigbench/generate_until/real_or_fake_text.yaml' 2024-03-18T14:20:40,290 adding 'lm_eval/tasks/bigbench/generate_until/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:40,291 adding 'lm_eval/tasks/bigbench/generate_until/repeat_copy_logic.yaml' 2024-03-18T14:20:40,292 adding 'lm_eval/tasks/bigbench/generate_until/rephrase.yaml' 2024-03-18T14:20:40,293 adding 'lm_eval/tasks/bigbench/generate_until/riddle_sense.yaml' 2024-03-18T14:20:40,294 adding 'lm_eval/tasks/bigbench/generate_until/ruin_names.yaml' 2024-03-18T14:20:40,295 adding 'lm_eval/tasks/bigbench/generate_until/salient_translation_error_detection.yaml' 2024-03-18T14:20:40,297 adding 'lm_eval/tasks/bigbench/generate_until/scientific_press_release.yaml' 2024-03-18T14:20:40,298 adding 'lm_eval/tasks/bigbench/generate_until/semantic_parsing_in_context_sparc.yaml' 2024-03-18T14:20:40,299 adding 'lm_eval/tasks/bigbench/generate_until/semantic_parsing_spider.yaml' 2024-03-18T14:20:40,301 adding 'lm_eval/tasks/bigbench/generate_until/sentence_ambiguity.yaml' 2024-03-18T14:20:40,302 adding 'lm_eval/tasks/bigbench/generate_until/similarities_abstraction.yaml' 2024-03-18T14:20:40,303 adding 'lm_eval/tasks/bigbench/generate_until/simp_turing_concept.yaml' 2024-03-18T14:20:40,304 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json.yaml' 2024-03-18T14:20:40,306 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_multiple_choice.yaml' 2024-03-18T14:20:40,307 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_json_subtasks.yaml' 2024-03-18T14:20:40,308 adding 'lm_eval/tasks/bigbench/generate_until/simple_arithmetic_multiple_targets_json.yaml' 2024-03-18T14:20:40,309 adding 'lm_eval/tasks/bigbench/generate_until/simple_ethical_questions.yaml' 2024-03-18T14:20:40,310 adding 'lm_eval/tasks/bigbench/generate_until/simple_text_editing.yaml' 2024-03-18T14:20:40,311 adding 'lm_eval/tasks/bigbench/generate_until/snarks.yaml' 2024-03-18T14:20:40,313 adding 'lm_eval/tasks/bigbench/generate_until/social_iqa.yaml' 2024-03-18T14:20:40,314 adding 'lm_eval/tasks/bigbench/generate_until/social_support.yaml' 2024-03-18T14:20:40,315 adding 'lm_eval/tasks/bigbench/generate_until/sports_understanding.yaml' 2024-03-18T14:20:40,316 adding 'lm_eval/tasks/bigbench/generate_until/strange_stories.yaml' 2024-03-18T14:20:40,317 adding 'lm_eval/tasks/bigbench/generate_until/strategyqa.yaml' 2024-03-18T14:20:40,318 adding 'lm_eval/tasks/bigbench/generate_until/sufficient_information.yaml' 2024-03-18T14:20:40,319 adding 'lm_eval/tasks/bigbench/generate_until/suicide_risk.yaml' 2024-03-18T14:20:40,320 adding 'lm_eval/tasks/bigbench/generate_until/swahili_english_proverbs.yaml' 2024-03-18T14:20:40,321 adding 'lm_eval/tasks/bigbench/generate_until/swedish_to_german_proverbs.yaml' 2024-03-18T14:20:40,322 adding 'lm_eval/tasks/bigbench/generate_until/symbol_interpretation.yaml' 2024-03-18T14:20:40,323 adding 'lm_eval/tasks/bigbench/generate_until/temporal_sequences.yaml' 2024-03-18T14:20:40,324 adding 'lm_eval/tasks/bigbench/generate_until/tense.yaml' 2024-03-18T14:20:40,325 adding 'lm_eval/tasks/bigbench/generate_until/timedial.yaml' 2024-03-18T14:20:40,326 adding 'lm_eval/tasks/bigbench/generate_until/topical_chat.yaml' 2024-03-18T14:20:40,328 adding 'lm_eval/tasks/bigbench/generate_until/tracking_shuffled_objects.yaml' 2024-03-18T14:20:40,329 adding 'lm_eval/tasks/bigbench/generate_until/understanding_fables.yaml' 2024-03-18T14:20:40,330 adding 'lm_eval/tasks/bigbench/generate_until/undo_permutation.yaml' 2024-03-18T14:20:40,331 adding 'lm_eval/tasks/bigbench/generate_until/unit_conversion.yaml' 2024-03-18T14:20:40,332 adding 'lm_eval/tasks/bigbench/generate_until/unit_interpretation.yaml' 2024-03-18T14:20:40,334 adding 'lm_eval/tasks/bigbench/generate_until/unnatural_in_context_learning.yaml' 2024-03-18T14:20:40,335 adding 'lm_eval/tasks/bigbench/generate_until/vitaminc_fact_verification.yaml' 2024-03-18T14:20:40,336 adding 'lm_eval/tasks/bigbench/generate_until/what_is_the_tao.yaml' 2024-03-18T14:20:40,337 adding 'lm_eval/tasks/bigbench/generate_until/which_wiki_edit.yaml' 2024-03-18T14:20:40,338 adding 'lm_eval/tasks/bigbench/generate_until/winowhy.yaml' 2024-03-18T14:20:40,339 adding 'lm_eval/tasks/bigbench/generate_until/word_sorting.yaml' 2024-03-18T14:20:40,340 adding 'lm_eval/tasks/bigbench/generate_until/word_unscrambling.yaml' 2024-03-18T14:20:40,346 adding 'lm_eval/tasks/bigbench/multiple_choice/abstract_narrative_understanding.yaml' 2024-03-18T14:20:40,347 adding 'lm_eval/tasks/bigbench/multiple_choice/anachronisms.yaml' 2024-03-18T14:20:40,348 adding 'lm_eval/tasks/bigbench/multiple_choice/analogical_similarity.yaml' 2024-03-18T14:20:40,349 adding 'lm_eval/tasks/bigbench/multiple_choice/analytic_entailment.yaml' 2024-03-18T14:20:40,351 adding 'lm_eval/tasks/bigbench/multiple_choice/arithmetic.yaml' 2024-03-18T14:20:40,352 adding 'lm_eval/tasks/bigbench/multiple_choice/ascii_word_recognition.yaml' 2024-03-18T14:20:40,353 adding 'lm_eval/tasks/bigbench/multiple_choice/authorship_verification.yaml' 2024-03-18T14:20:40,354 adding 'lm_eval/tasks/bigbench/multiple_choice/auto_categorization.yaml' 2024-03-18T14:20:40,355 adding 'lm_eval/tasks/bigbench/multiple_choice/auto_debugging.yaml' 2024-03-18T14:20:40,357 adding 'lm_eval/tasks/bigbench/multiple_choice/bbq_lite_json.yaml' 2024-03-18T14:20:40,358 adding 'lm_eval/tasks/bigbench/multiple_choice/bridging_anaphora_resolution_barqa.yaml' 2024-03-18T14:20:40,359 adding 'lm_eval/tasks/bigbench/multiple_choice/causal_judgement.yaml' 2024-03-18T14:20:40,360 adding 'lm_eval/tasks/bigbench/multiple_choice/causal_judgment.yaml' 2024-03-18T14:20:40,361 adding 'lm_eval/tasks/bigbench/multiple_choice/cause_and_effect.yaml' 2024-03-18T14:20:40,362 adding 'lm_eval/tasks/bigbench/multiple_choice/checkmate_in_one.yaml' 2024-03-18T14:20:40,363 adding 'lm_eval/tasks/bigbench/multiple_choice/chess_state_tracking.yaml' 2024-03-18T14:20:40,364 adding 'lm_eval/tasks/bigbench/multiple_choice/chinese_remainder_theorem.yaml' 2024-03-18T14:20:40,365 adding 'lm_eval/tasks/bigbench/multiple_choice/cifar10_classification.yaml' 2024-03-18T14:20:40,367 adding 'lm_eval/tasks/bigbench/multiple_choice/code_line_description.yaml' 2024-03-18T14:20:40,368 adding 'lm_eval/tasks/bigbench/multiple_choice/codenames.yaml' 2024-03-18T14:20:40,369 adding 'lm_eval/tasks/bigbench/multiple_choice/color.yaml' 2024-03-18T14:20:40,370 adding 'lm_eval/tasks/bigbench/multiple_choice/common_morpheme.yaml' 2024-03-18T14:20:40,371 adding 'lm_eval/tasks/bigbench/multiple_choice/conceptual_combinations.yaml' 2024-03-18T14:20:40,372 adding 'lm_eval/tasks/bigbench/multiple_choice/conlang_translation.yaml' 2024-03-18T14:20:40,373 adding 'lm_eval/tasks/bigbench/multiple_choice/contextual_parametric_knowledge_conflicts.yaml' 2024-03-18T14:20:40,374 adding 'lm_eval/tasks/bigbench/multiple_choice/crash_blossom.yaml' 2024-03-18T14:20:40,375 adding 'lm_eval/tasks/bigbench/multiple_choice/crass_ai.yaml' 2024-03-18T14:20:40,376 adding 'lm_eval/tasks/bigbench/multiple_choice/cryobiology_spanish.yaml' 2024-03-18T14:20:40,377 adding 'lm_eval/tasks/bigbench/multiple_choice/cryptonite.yaml' 2024-03-18T14:20:40,379 adding 'lm_eval/tasks/bigbench/multiple_choice/cs_algorithms.yaml' 2024-03-18T14:20:40,380 adding 'lm_eval/tasks/bigbench/multiple_choice/dark_humor_detection.yaml' 2024-03-18T14:20:40,381 adding 'lm_eval/tasks/bigbench/multiple_choice/date_understanding.yaml' 2024-03-18T14:20:40,382 adding 'lm_eval/tasks/bigbench/multiple_choice/disambiguation_qa.yaml' 2024-03-18T14:20:40,383 adding 'lm_eval/tasks/bigbench/multiple_choice/discourse_marker_prediction.yaml' 2024-03-18T14:20:40,385 adding 'lm_eval/tasks/bigbench/multiple_choice/disfl_qa.yaml' 2024-03-18T14:20:40,386 adding 'lm_eval/tasks/bigbench/multiple_choice/dyck_languages.yaml' 2024-03-18T14:20:40,387 adding 'lm_eval/tasks/bigbench/multiple_choice/elementary_math_qa.yaml' 2024-03-18T14:20:40,388 adding 'lm_eval/tasks/bigbench/multiple_choice/emoji_movie.yaml' 2024-03-18T14:20:40,389 adding 'lm_eval/tasks/bigbench/multiple_choice/emojis_emotion_prediction.yaml' 2024-03-18T14:20:40,390 adding 'lm_eval/tasks/bigbench/multiple_choice/empirical_judgments.yaml' 2024-03-18T14:20:40,392 adding 'lm_eval/tasks/bigbench/multiple_choice/english_proverbs.yaml' 2024-03-18T14:20:40,393 adding 'lm_eval/tasks/bigbench/multiple_choice/english_russian_proverbs.yaml' 2024-03-18T14:20:40,394 adding 'lm_eval/tasks/bigbench/multiple_choice/entailed_polarity.yaml' 2024-03-18T14:20:40,396 adding 'lm_eval/tasks/bigbench/multiple_choice/entailed_polarity_hindi.yaml' 2024-03-18T14:20:40,397 adding 'lm_eval/tasks/bigbench/multiple_choice/epistemic_reasoning.yaml' 2024-03-18T14:20:40,398 adding 'lm_eval/tasks/bigbench/multiple_choice/evaluating_information_essentiality.yaml' 2024-03-18T14:20:40,399 adding 'lm_eval/tasks/bigbench/multiple_choice/fact_checker.yaml' 2024-03-18T14:20:40,401 adding 'lm_eval/tasks/bigbench/multiple_choice/fantasy_reasoning.yaml' 2024-03-18T14:20:40,402 adding 'lm_eval/tasks/bigbench/multiple_choice/few_shot_nlg.yaml' 2024-03-18T14:20:40,403 adding 'lm_eval/tasks/bigbench/multiple_choice/figure_of_speech_detection.yaml' 2024-03-18T14:20:40,404 adding 'lm_eval/tasks/bigbench/multiple_choice/formal_fallacies_syllogisms_negation.yaml' 2024-03-18T14:20:40,405 adding 'lm_eval/tasks/bigbench/multiple_choice/gem.yaml' 2024-03-18T14:20:40,407 adding 'lm_eval/tasks/bigbench/multiple_choice/gender_inclusive_sentences_german.yaml' 2024-03-18T14:20:40,408 adding 'lm_eval/tasks/bigbench/multiple_choice/general_knowledge.yaml' 2024-03-18T14:20:40,409 adding 'lm_eval/tasks/bigbench/multiple_choice/geometric_shapes.yaml' 2024-03-18T14:20:40,411 adding 'lm_eval/tasks/bigbench/multiple_choice/goal_step_wikihow.yaml' 2024-03-18T14:20:40,412 adding 'lm_eval/tasks/bigbench/multiple_choice/gre_reading_comprehension.yaml' 2024-03-18T14:20:40,413 adding 'lm_eval/tasks/bigbench/multiple_choice/hhh_alignment.yaml' 2024-03-18T14:20:40,414 adding 'lm_eval/tasks/bigbench/multiple_choice/hindi_question_answering.yaml' 2024-03-18T14:20:40,415 adding 'lm_eval/tasks/bigbench/multiple_choice/hindu_knowledge.yaml' 2024-03-18T14:20:40,416 adding 'lm_eval/tasks/bigbench/multiple_choice/hinglish_toxicity.yaml' 2024-03-18T14:20:40,417 adding 'lm_eval/tasks/bigbench/multiple_choice/human_organs_senses.yaml' 2024-03-18T14:20:40,418 adding 'lm_eval/tasks/bigbench/multiple_choice/hyperbaton.yaml' 2024-03-18T14:20:40,419 adding 'lm_eval/tasks/bigbench/multiple_choice/identify_math_theorems.yaml' 2024-03-18T14:20:40,420 adding 'lm_eval/tasks/bigbench/multiple_choice/identify_odd_metaphor.yaml' 2024-03-18T14:20:40,421 adding 'lm_eval/tasks/bigbench/multiple_choice/implicatures.yaml' 2024-03-18T14:20:40,422 adding 'lm_eval/tasks/bigbench/multiple_choice/implicit_relations.yaml' 2024-03-18T14:20:40,424 adding 'lm_eval/tasks/bigbench/multiple_choice/intent_recognition.yaml' 2024-03-18T14:20:40,425 adding 'lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_nli.yaml' 2024-03-18T14:20:40,426 adding 'lm_eval/tasks/bigbench/multiple_choice/international_phonetic_alphabet_transliterate.yaml' 2024-03-18T14:20:40,427 adding 'lm_eval/tasks/bigbench/multiple_choice/intersect_geometry.yaml' 2024-03-18T14:20:40,428 adding 'lm_eval/tasks/bigbench/multiple_choice/irony_identification.yaml' 2024-03-18T14:20:40,429 adding 'lm_eval/tasks/bigbench/multiple_choice/kanji_ascii.yaml' 2024-03-18T14:20:40,431 adding 'lm_eval/tasks/bigbench/multiple_choice/kannada.yaml' 2024-03-18T14:20:40,432 adding 'lm_eval/tasks/bigbench/multiple_choice/key_value_maps.yaml' 2024-03-18T14:20:40,433 adding 'lm_eval/tasks/bigbench/multiple_choice/known_unknowns.yaml' 2024-03-18T14:20:40,434 adding 'lm_eval/tasks/bigbench/multiple_choice/language_games.yaml' 2024-03-18T14:20:40,435 adding 'lm_eval/tasks/bigbench/multiple_choice/language_identification.yaml' 2024-03-18T14:20:40,436 adding 'lm_eval/tasks/bigbench/multiple_choice/linguistic_mappings.yaml' 2024-03-18T14:20:40,437 adding 'lm_eval/tasks/bigbench/multiple_choice/linguistics_puzzles.yaml' 2024-03-18T14:20:40,438 adding 'lm_eval/tasks/bigbench/multiple_choice/list_functions.yaml' 2024-03-18T14:20:40,440 adding 'lm_eval/tasks/bigbench/multiple_choice/logic_grid_puzzle.yaml' 2024-03-18T14:20:40,441 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_args.yaml' 2024-03-18T14:20:40,442 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_deduction.yaml' 2024-03-18T14:20:40,443 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_fallacy_detection.yaml' 2024-03-18T14:20:40,444 adding 'lm_eval/tasks/bigbench/multiple_choice/logical_sequence.yaml' 2024-03-18T14:20:40,445 adding 'lm_eval/tasks/bigbench/multiple_choice/mathematical_induction.yaml' 2024-03-18T14:20:40,447 adding 'lm_eval/tasks/bigbench/multiple_choice/matrixshapes.yaml' 2024-03-18T14:20:40,448 adding 'lm_eval/tasks/bigbench/multiple_choice/metaphor_boolean.yaml' 2024-03-18T14:20:40,449 adding 'lm_eval/tasks/bigbench/multiple_choice/metaphor_understanding.yaml' 2024-03-18T14:20:40,450 adding 'lm_eval/tasks/bigbench/multiple_choice/minute_mysteries_qa.yaml' 2024-03-18T14:20:40,451 adding 'lm_eval/tasks/bigbench/multiple_choice/misconceptions.yaml' 2024-03-18T14:20:40,452 adding 'lm_eval/tasks/bigbench/multiple_choice/misconceptions_russian.yaml' 2024-03-18T14:20:40,454 adding 'lm_eval/tasks/bigbench/multiple_choice/mnist_ascii.yaml' 2024-03-18T14:20:40,455 adding 'lm_eval/tasks/bigbench/multiple_choice/modified_arithmetic.yaml' 2024-03-18T14:20:40,456 adding 'lm_eval/tasks/bigbench/multiple_choice/moral_permissibility.yaml' 2024-03-18T14:20:40,457 adding 'lm_eval/tasks/bigbench/multiple_choice/movie_dialog_same_or_different.yaml' 2024-03-18T14:20:40,458 adding 'lm_eval/tasks/bigbench/multiple_choice/movie_recommendation.yaml' 2024-03-18T14:20:40,459 adding 'lm_eval/tasks/bigbench/multiple_choice/mult_data_wrangling.yaml' 2024-03-18T14:20:40,460 adding 'lm_eval/tasks/bigbench/multiple_choice/multiemo.yaml' 2024-03-18T14:20:40,461 adding 'lm_eval/tasks/bigbench/multiple_choice/natural_instructions.yaml' 2024-03-18T14:20:40,462 adding 'lm_eval/tasks/bigbench/multiple_choice/navigate.yaml' 2024-03-18T14:20:40,463 adding 'lm_eval/tasks/bigbench/multiple_choice/nonsense_words_grammar.yaml' 2024-03-18T14:20:40,464 adding 'lm_eval/tasks/bigbench/multiple_choice/novel_concepts.yaml' 2024-03-18T14:20:40,465 adding 'lm_eval/tasks/bigbench/multiple_choice/object_counting.yaml' 2024-03-18T14:20:40,466 adding 'lm_eval/tasks/bigbench/multiple_choice/odd_one_out.yaml' 2024-03-18T14:20:40,467 adding 'lm_eval/tasks/bigbench/multiple_choice/operators.yaml' 2024-03-18T14:20:40,469 adding 'lm_eval/tasks/bigbench/multiple_choice/paragraph_segmentation.yaml' 2024-03-18T14:20:40,470 adding 'lm_eval/tasks/bigbench/multiple_choice/parsinlu_qa.yaml' 2024-03-18T14:20:40,471 adding 'lm_eval/tasks/bigbench/multiple_choice/parsinlu_reading_comprehension.yaml' 2024-03-18T14:20:40,472 adding 'lm_eval/tasks/bigbench/multiple_choice/penguins_in_a_table.yaml' 2024-03-18T14:20:40,473 adding 'lm_eval/tasks/bigbench/multiple_choice/periodic_elements.yaml' 2024-03-18T14:20:40,474 adding 'lm_eval/tasks/bigbench/multiple_choice/persian_idioms.yaml' 2024-03-18T14:20:40,475 adding 'lm_eval/tasks/bigbench/multiple_choice/phrase_relatedness.yaml' 2024-03-18T14:20:40,476 adding 'lm_eval/tasks/bigbench/multiple_choice/physical_intuition.yaml' 2024-03-18T14:20:40,478 adding 'lm_eval/tasks/bigbench/multiple_choice/physics.yaml' 2024-03-18T14:20:40,479 adding 'lm_eval/tasks/bigbench/multiple_choice/physics_questions.yaml' 2024-03-18T14:20:40,480 adding 'lm_eval/tasks/bigbench/multiple_choice/play_dialog_same_or_different.yaml' 2024-03-18T14:20:40,481 adding 'lm_eval/tasks/bigbench/multiple_choice/polish_sequence_labeling.yaml' 2024-03-18T14:20:40,482 adding 'lm_eval/tasks/bigbench/multiple_choice/presuppositions_as_nli.yaml' 2024-03-18T14:20:40,483 adding 'lm_eval/tasks/bigbench/multiple_choice/qa_wikidata.yaml' 2024-03-18T14:20:40,485 adding 'lm_eval/tasks/bigbench/multiple_choice/question_selection.yaml' 2024-03-18T14:20:40,486 adding 'lm_eval/tasks/bigbench/multiple_choice/real_or_fake_text.yaml' 2024-03-18T14:20:40,487 adding 'lm_eval/tasks/bigbench/multiple_choice/reasoning_about_colored_objects.yaml' 2024-03-18T14:20:40,488 adding 'lm_eval/tasks/bigbench/multiple_choice/repeat_copy_logic.yaml' 2024-03-18T14:20:40,489 adding 'lm_eval/tasks/bigbench/multiple_choice/rephrase.yaml' 2024-03-18T14:20:40,490 adding 'lm_eval/tasks/bigbench/multiple_choice/riddle_sense.yaml' 2024-03-18T14:20:40,492 adding 'lm_eval/tasks/bigbench/multiple_choice/ruin_names.yaml' 2024-03-18T14:20:40,493 adding 'lm_eval/tasks/bigbench/multiple_choice/salient_translation_error_detection.yaml' 2024-03-18T14:20:40,494 adding 'lm_eval/tasks/bigbench/multiple_choice/scientific_press_release.yaml' 2024-03-18T14:20:40,495 adding 'lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_in_context_sparc.yaml' 2024-03-18T14:20:40,496 adding 'lm_eval/tasks/bigbench/multiple_choice/semantic_parsing_spider.yaml' 2024-03-18T14:20:40,497 adding 'lm_eval/tasks/bigbench/multiple_choice/sentence_ambiguity.yaml' 2024-03-18T14:20:40,498 adding 'lm_eval/tasks/bigbench/multiple_choice/similarities_abstraction.yaml' 2024-03-18T14:20:40,500 adding 'lm_eval/tasks/bigbench/multiple_choice/simp_turing_concept.yaml' 2024-03-18T14:20:40,501 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json.yaml' 2024-03-18T14:20:40,502 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_multiple_choice.yaml' 2024-03-18T14:20:40,503 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_json_subtasks.yaml' 2024-03-18T14:20:40,504 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_arithmetic_multiple_targets_json.yaml' 2024-03-18T14:20:40,505 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_ethical_questions.yaml' 2024-03-18T14:20:40,506 adding 'lm_eval/tasks/bigbench/multiple_choice/simple_text_editing.yaml' 2024-03-18T14:20:40,507 adding 'lm_eval/tasks/bigbench/multiple_choice/snarks.yaml' 2024-03-18T14:20:40,509 adding 'lm_eval/tasks/bigbench/multiple_choice/social_iqa.yaml' 2024-03-18T14:20:40,510 adding 'lm_eval/tasks/bigbench/multiple_choice/social_support.yaml' 2024-03-18T14:20:40,511 adding 'lm_eval/tasks/bigbench/multiple_choice/sports_understanding.yaml' 2024-03-18T14:20:40,512 adding 'lm_eval/tasks/bigbench/multiple_choice/strange_stories.yaml' 2024-03-18T14:20:40,513 adding 'lm_eval/tasks/bigbench/multiple_choice/strategyqa.yaml' 2024-03-18T14:20:40,514 adding 'lm_eval/tasks/bigbench/multiple_choice/sufficient_information.yaml' 2024-03-18T14:20:40,515 adding 'lm_eval/tasks/bigbench/multiple_choice/suicide_risk.yaml' 2024-03-18T14:20:40,516 adding 'lm_eval/tasks/bigbench/multiple_choice/swahili_english_proverbs.yaml' 2024-03-18T14:20:40,517 adding 'lm_eval/tasks/bigbench/multiple_choice/swedish_to_german_proverbs.yaml' 2024-03-18T14:20:40,519 adding 'lm_eval/tasks/bigbench/multiple_choice/symbol_interpretation.yaml' 2024-03-18T14:20:40,520 adding 'lm_eval/tasks/bigbench/multiple_choice/temporal_sequences.yaml' 2024-03-18T14:20:40,521 adding 'lm_eval/tasks/bigbench/multiple_choice/tense.yaml' 2024-03-18T14:20:40,522 adding 'lm_eval/tasks/bigbench/multiple_choice/timedial.yaml' 2024-03-18T14:20:40,523 adding 'lm_eval/tasks/bigbench/multiple_choice/topical_chat.yaml' 2024-03-18T14:20:40,524 adding 'lm_eval/tasks/bigbench/multiple_choice/tracking_shuffled_objects.yaml' 2024-03-18T14:20:40,525 adding 'lm_eval/tasks/bigbench/multiple_choice/understanding_fables.yaml' 2024-03-18T14:20:40,526 adding 'lm_eval/tasks/bigbench/multiple_choice/undo_permutation.yaml' 2024-03-18T14:20:40,527 adding 'lm_eval/tasks/bigbench/multiple_choice/unit_conversion.yaml' 2024-03-18T14:20:40,529 adding 'lm_eval/tasks/bigbench/multiple_choice/unit_interpretation.yaml' 2024-03-18T14:20:40,530 adding 'lm_eval/tasks/bigbench/multiple_choice/unnatural_in_context_learning.yaml' 2024-03-18T14:20:40,531 adding 'lm_eval/tasks/bigbench/multiple_choice/vitaminc_fact_verification.yaml' 2024-03-18T14:20:40,532 adding 'lm_eval/tasks/bigbench/multiple_choice/what_is_the_tao.yaml' 2024-03-18T14:20:40,534 adding 'lm_eval/tasks/bigbench/multiple_choice/which_wiki_edit.yaml' 2024-03-18T14:20:40,535 adding 'lm_eval/tasks/bigbench/multiple_choice/winowhy.yaml' 2024-03-18T14:20:40,536 adding 'lm_eval/tasks/bigbench/multiple_choice/word_sorting.yaml' 2024-03-18T14:20:40,537 adding 'lm_eval/tasks/bigbench/multiple_choice/word_unscrambling.yaml' 2024-03-18T14:20:40,540 adding 'lm_eval/tasks/blimp/README.md' 2024-03-18T14:20:40,541 adding 'lm_eval/tasks/blimp/_template_yaml' 2024-03-18T14:20:40,543 adding 'lm_eval/tasks/blimp/adjunct_island.yaml' 2024-03-18T14:20:40,544 adding 'lm_eval/tasks/blimp/anaphor_gender_agreement.yaml' 2024-03-18T14:20:40,545 adding 'lm_eval/tasks/blimp/anaphor_number_agreement.yaml' 2024-03-18T14:20:40,546 adding 'lm_eval/tasks/blimp/animate_subject_passive.yaml' 2024-03-18T14:20:40,547 adding 'lm_eval/tasks/blimp/animate_subject_trans.yaml' 2024-03-18T14:20:40,548 adding 'lm_eval/tasks/blimp/causative.yaml' 2024-03-18T14:20:40,550 adding 'lm_eval/tasks/blimp/complex_NP_island.yaml' 2024-03-18T14:20:40,551 adding 'lm_eval/tasks/blimp/coordinate_structure_constraint_complex_left_branch.yaml' 2024-03-18T14:20:40,552 adding 'lm_eval/tasks/blimp/coordinate_structure_constraint_object_extraction.yaml' 2024-03-18T14:20:40,553 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_1.yaml' 2024-03-18T14:20:40,554 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_2.yaml' 2024-03-18T14:20:40,555 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_irregular_1.yaml' 2024-03-18T14:20:40,556 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_irregular_2.yaml' 2024-03-18T14:20:40,557 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_2.yaml' 2024-03-18T14:20:40,559 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_1.yaml' 2024-03-18T14:20:40,560 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adj_irregular_2.yaml' 2024-03-18T14:20:40,561 adding 'lm_eval/tasks/blimp/determiner_noun_agreement_with_adjective_1.yaml' 2024-03-18T14:20:40,562 adding 'lm_eval/tasks/blimp/distractor_agreement_relational_noun.yaml' 2024-03-18T14:20:40,563 adding 'lm_eval/tasks/blimp/distractor_agreement_relative_clause.yaml' 2024-03-18T14:20:40,564 adding 'lm_eval/tasks/blimp/drop_argument.yaml' 2024-03-18T14:20:40,565 adding 'lm_eval/tasks/blimp/ellipsis_n_bar_1.yaml' 2024-03-18T14:20:40,566 adding 'lm_eval/tasks/blimp/ellipsis_n_bar_2.yaml' 2024-03-18T14:20:40,567 adding 'lm_eval/tasks/blimp/existential_there_object_raising.yaml' 2024-03-18T14:20:40,568 adding 'lm_eval/tasks/blimp/existential_there_quantifiers_1.yaml' 2024-03-18T14:20:40,570 adding 'lm_eval/tasks/blimp/existential_there_quantifiers_2.yaml' 2024-03-18T14:20:40,571 adding 'lm_eval/tasks/blimp/existential_there_subject_raising.yaml' 2024-03-18T14:20:40,572 adding 'lm_eval/tasks/blimp/expletive_it_object_raising.yaml' 2024-03-18T14:20:40,573 adding 'lm_eval/tasks/blimp/generate_configs.py' 2024-03-18T14:20:40,574 adding 'lm_eval/tasks/blimp/inchoative.yaml' 2024-03-18T14:20:40,575 adding 'lm_eval/tasks/blimp/intransitive.yaml' 2024-03-18T14:20:40,576 adding 'lm_eval/tasks/blimp/irregular_past_participle_adjectives.yaml' 2024-03-18T14:20:40,577 adding 'lm_eval/tasks/blimp/irregular_past_participle_verbs.yaml' 2024-03-18T14:20:40,578 adding 'lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_1.yaml' 2024-03-18T14:20:40,580 adding 'lm_eval/tasks/blimp/irregular_plural_subject_verb_agreement_2.yaml' 2024-03-18T14:20:40,581 adding 'lm_eval/tasks/blimp/left_branch_island_echo_question.yaml' 2024-03-18T14:20:40,582 adding 'lm_eval/tasks/blimp/left_branch_island_simple_question.yaml' 2024-03-18T14:20:40,583 adding 'lm_eval/tasks/blimp/matrix_question_npi_licensor_present.yaml' 2024-03-18T14:20:40,584 adding 'lm_eval/tasks/blimp/npi_present_1.yaml' 2024-03-18T14:20:40,585 adding 'lm_eval/tasks/blimp/npi_present_2.yaml' 2024-03-18T14:20:40,586 adding 'lm_eval/tasks/blimp/only_npi_licensor_present.yaml' 2024-03-18T14:20:40,587 adding 'lm_eval/tasks/blimp/only_npi_scope.yaml' 2024-03-18T14:20:40,589 adding 'lm_eval/tasks/blimp/passive_1.yaml' 2024-03-18T14:20:40,590 adding 'lm_eval/tasks/blimp/passive_2.yaml' 2024-03-18T14:20:40,591 adding 'lm_eval/tasks/blimp/principle_A_c_command.yaml' 2024-03-18T14:20:40,592 adding 'lm_eval/tasks/blimp/principle_A_case_1.yaml' 2024-03-18T14:20:40,593 adding 'lm_eval/tasks/blimp/principle_A_case_2.yaml' 2024-03-18T14:20:40,594 adding 'lm_eval/tasks/blimp/principle_A_domain_1.yaml' 2024-03-18T14:20:40,595 adding 'lm_eval/tasks/blimp/principle_A_domain_2.yaml' 2024-03-18T14:20:40,597 adding 'lm_eval/tasks/blimp/principle_A_domain_3.yaml' 2024-03-18T14:20:40,598 adding 'lm_eval/tasks/blimp/principle_A_reconstruction.yaml' 2024-03-18T14:20:40,600 adding 'lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_1.yaml' 2024-03-18T14:20:40,601 adding 'lm_eval/tasks/blimp/regular_plural_subject_verb_agreement_2.yaml' 2024-03-18T14:20:40,603 adding 'lm_eval/tasks/blimp/sentential_negation_npi_licensor_present.yaml' 2024-03-18T14:20:40,604 adding 'lm_eval/tasks/blimp/sentential_negation_npi_scope.yaml' 2024-03-18T14:20:40,605 adding 'lm_eval/tasks/blimp/sentential_subject_island.yaml' 2024-03-18T14:20:40,606 adding 'lm_eval/tasks/blimp/superlative_quantifiers_1.yaml' 2024-03-18T14:20:40,608 adding 'lm_eval/tasks/blimp/superlative_quantifiers_2.yaml' 2024-03-18T14:20:40,609 adding 'lm_eval/tasks/blimp/tough_vs_raising_1.yaml' 2024-03-18T14:20:40,610 adding 'lm_eval/tasks/blimp/tough_vs_raising_2.yaml' 2024-03-18T14:20:40,611 adding 'lm_eval/tasks/blimp/transitive.yaml' 2024-03-18T14:20:40,612 adding 'lm_eval/tasks/blimp/wh_island.yaml' 2024-03-18T14:20:40,613 adding 'lm_eval/tasks/blimp/wh_questions_object_gap.yaml' 2024-03-18T14:20:40,614 adding 'lm_eval/tasks/blimp/wh_questions_subject_gap.yaml' 2024-03-18T14:20:40,615 adding 'lm_eval/tasks/blimp/wh_questions_subject_gap_long_distance.yaml' 2024-03-18T14:20:40,616 adding 'lm_eval/tasks/blimp/wh_vs_that_no_gap.yaml' 2024-03-18T14:20:40,617 adding 'lm_eval/tasks/blimp/wh_vs_that_no_gap_long_distance.yaml' 2024-03-18T14:20:40,618 adding 'lm_eval/tasks/blimp/wh_vs_that_with_gap.yaml' 2024-03-18T14:20:40,620 adding 'lm_eval/tasks/blimp/wh_vs_that_with_gap_long_distance.yaml' 2024-03-18T14:20:40,623 adding 'lm_eval/tasks/ceval/README.md' 2024-03-18T14:20:40,624 adding 'lm_eval/tasks/ceval/_default_ceval_yaml' 2024-03-18T14:20:40,625 adding 'lm_eval/tasks/ceval/_generate_configs.py' 2024-03-18T14:20:40,627 adding 'lm_eval/tasks/ceval/ceval-valid_accountant.yaml' 2024-03-18T14:20:40,628 adding 'lm_eval/tasks/ceval/ceval-valid_advanced_mathematics.yaml' 2024-03-18T14:20:40,629 adding 'lm_eval/tasks/ceval/ceval-valid_art_studies.yaml' 2024-03-18T14:20:40,630 adding 'lm_eval/tasks/ceval/ceval-valid_basic_medicine.yaml' 2024-03-18T14:20:40,631 adding 'lm_eval/tasks/ceval/ceval-valid_business_administration.yaml' 2024-03-18T14:20:40,632 adding 'lm_eval/tasks/ceval/ceval-valid_chinese_language_and_literature.yaml' 2024-03-18T14:20:40,633 adding 'lm_eval/tasks/ceval/ceval-valid_civil_servant.yaml' 2024-03-18T14:20:40,634 adding 'lm_eval/tasks/ceval/ceval-valid_clinical_medicine.yaml' 2024-03-18T14:20:40,636 adding 'lm_eval/tasks/ceval/ceval-valid_college_chemistry.yaml' 2024-03-18T14:20:40,637 adding 'lm_eval/tasks/ceval/ceval-valid_college_economics.yaml' 2024-03-18T14:20:40,638 adding 'lm_eval/tasks/ceval/ceval-valid_college_physics.yaml' 2024-03-18T14:20:40,640 adding 'lm_eval/tasks/ceval/ceval-valid_college_programming.yaml' 2024-03-18T14:20:40,641 adding 'lm_eval/tasks/ceval/ceval-valid_computer_architecture.yaml' 2024-03-18T14:20:40,642 adding 'lm_eval/tasks/ceval/ceval-valid_computer_network.yaml' 2024-03-18T14:20:40,644 adding 'lm_eval/tasks/ceval/ceval-valid_discrete_mathematics.yaml' 2024-03-18T14:20:40,645 adding 'lm_eval/tasks/ceval/ceval-valid_education_science.yaml' 2024-03-18T14:20:40,646 adding 'lm_eval/tasks/ceval/ceval-valid_electrical_engineer.yaml' 2024-03-18T14:20:40,647 adding 'lm_eval/tasks/ceval/ceval-valid_environmental_impact_assessment_engineer.yaml' 2024-03-18T14:20:40,649 adding 'lm_eval/tasks/ceval/ceval-valid_fire_engineer.yaml' 2024-03-18T14:20:40,650 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_biology.yaml' 2024-03-18T14:20:40,651 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_chemistry.yaml' 2024-03-18T14:20:40,653 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_chinese.yaml' 2024-03-18T14:20:40,654 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_geography.yaml' 2024-03-18T14:20:40,655 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_history.yaml' 2024-03-18T14:20:40,656 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_mathematics.yaml' 2024-03-18T14:20:40,658 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_physics.yaml' 2024-03-18T14:20:40,659 adding 'lm_eval/tasks/ceval/ceval-valid_high_school_politics.yaml' 2024-03-18T14:20:40,660 adding 'lm_eval/tasks/ceval/ceval-valid_ideological_and_moral_cultivation.yaml' 2024-03-18T14:20:40,661 adding 'lm_eval/tasks/ceval/ceval-valid_law.yaml' 2024-03-18T14:20:40,662 adding 'lm_eval/tasks/ceval/ceval-valid_legal_professional.yaml' 2024-03-18T14:20:40,664 adding 'lm_eval/tasks/ceval/ceval-valid_logic.yaml' 2024-03-18T14:20:40,665 adding 'lm_eval/tasks/ceval/ceval-valid_mao_zedong_thought.yaml' 2024-03-18T14:20:40,667 adding 'lm_eval/tasks/ceval/ceval-valid_marxism.yaml' 2024-03-18T14:20:40,668 adding 'lm_eval/tasks/ceval/ceval-valid_metrology_engineer.yaml' 2024-03-18T14:20:40,669 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_biology.yaml' 2024-03-18T14:20:40,670 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_chemistry.yaml' 2024-03-18T14:20:40,671 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_geography.yaml' 2024-03-18T14:20:40,673 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_history.yaml' 2024-03-18T14:20:40,674 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_mathematics.yaml' 2024-03-18T14:20:40,675 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_physics.yaml' 2024-03-18T14:20:40,677 adding 'lm_eval/tasks/ceval/ceval-valid_middle_school_politics.yaml' 2024-03-18T14:20:40,678 adding 'lm_eval/tasks/ceval/ceval-valid_modern_chinese_history.yaml' 2024-03-18T14:20:40,679 adding 'lm_eval/tasks/ceval/ceval-valid_operating_system.yaml' 2024-03-18T14:20:40,680 adding 'lm_eval/tasks/ceval/ceval-valid_physician.yaml' 2024-03-18T14:20:40,682 adding 'lm_eval/tasks/ceval/ceval-valid_plant_protection.yaml' 2024-03-18T14:20:40,683 adding 'lm_eval/tasks/ceval/ceval-valid_probability_and_statistics.yaml' 2024-03-18T14:20:40,684 adding 'lm_eval/tasks/ceval/ceval-valid_professional_tour_guide.yaml' 2024-03-18T14:20:40,685 adding 'lm_eval/tasks/ceval/ceval-valid_sports_science.yaml' 2024-03-18T14:20:40,686 adding 'lm_eval/tasks/ceval/ceval-valid_tax_accountant.yaml' 2024-03-18T14:20:40,688 adding 'lm_eval/tasks/ceval/ceval-valid_teacher_qualification.yaml' 2024-03-18T14:20:40,689 adding 'lm_eval/tasks/ceval/ceval-valid_urban_and_rural_planner.yaml' 2024-03-18T14:20:40,690 adding 'lm_eval/tasks/ceval/ceval-valid_veterinary_medicine.yaml' 2024-03-18T14:20:40,694 adding 'lm_eval/tasks/cmmlu/README.md' 2024-03-18T14:20:40,695 adding 'lm_eval/tasks/cmmlu/_default_template_yaml' 2024-03-18T14:20:40,697 adding 'lm_eval/tasks/cmmlu/_generate_configs.py' 2024-03-18T14:20:40,698 adding 'lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml' 2024-03-18T14:20:40,699 adding 'lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml' 2024-03-18T14:20:40,701 adding 'lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml' 2024-03-18T14:20:40,702 adding 'lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml' 2024-03-18T14:20:40,703 adding 'lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml' 2024-03-18T14:20:40,704 adding 'lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml' 2024-03-18T14:20:40,706 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml' 2024-03-18T14:20:40,707 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml' 2024-03-18T14:20:40,708 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml' 2024-03-18T14:20:40,710 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml' 2024-03-18T14:20:40,711 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml' 2024-03-18T14:20:40,712 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml' 2024-03-18T14:20:40,714 adding 'lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml' 2024-03-18T14:20:40,715 adding 'lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml' 2024-03-18T14:20:40,717 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml' 2024-03-18T14:20:40,718 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml' 2024-03-18T14:20:40,719 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_engineering_hydrology.yaml' 2024-03-18T14:20:40,720 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_law.yaml' 2024-03-18T14:20:40,722 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_mathematics.yaml' 2024-03-18T14:20:40,723 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_medical_statistics.yaml' 2024-03-18T14:20:40,724 adding 'lm_eval/tasks/cmmlu/cmmlu_default_college_medicine.yaml' 2024-03-18T14:20:40,725 adding 'lm_eval/tasks/cmmlu/cmmlu_default_computer_science.yaml' 2024-03-18T14:20:40,727 adding 'lm_eval/tasks/cmmlu/cmmlu_default_computer_security.yaml' 2024-03-18T14:20:40,728 adding 'lm_eval/tasks/cmmlu/cmmlu_default_conceptual_physics.yaml' 2024-03-18T14:20:40,729 adding 'lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml' 2024-03-18T14:20:40,730 adding 'lm_eval/tasks/cmmlu/cmmlu_default_economics.yaml' 2024-03-18T14:20:40,731 adding 'lm_eval/tasks/cmmlu/cmmlu_default_education.yaml' 2024-03-18T14:20:40,732 adding 'lm_eval/tasks/cmmlu/cmmlu_default_electrical_engineering.yaml' 2024-03-18T14:20:40,734 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_chinese.yaml' 2024-03-18T14:20:40,735 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_commonsense.yaml' 2024-03-18T14:20:40,736 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_information_and_technology.yaml' 2024-03-18T14:20:40,738 adding 'lm_eval/tasks/cmmlu/cmmlu_default_elementary_mathematics.yaml' 2024-03-18T14:20:40,739 adding 'lm_eval/tasks/cmmlu/cmmlu_default_ethnology.yaml' 2024-03-18T14:20:40,740 adding 'lm_eval/tasks/cmmlu/cmmlu_default_food_science.yaml' 2024-03-18T14:20:40,741 adding 'lm_eval/tasks/cmmlu/cmmlu_default_genetics.yaml' 2024-03-18T14:20:40,742 adding 'lm_eval/tasks/cmmlu/cmmlu_default_global_facts.yaml' 2024-03-18T14:20:40,744 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_biology.yaml' 2024-03-18T14:20:40,746 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_chemistry.yaml' 2024-03-18T14:20:40,747 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml' 2024-03-18T14:20:40,748 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_mathematics.yaml' 2024-03-18T14:20:40,749 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_physics.yaml' 2024-03-18T14:20:40,751 adding 'lm_eval/tasks/cmmlu/cmmlu_default_high_school_politics.yaml' 2024-03-18T14:20:40,752 adding 'lm_eval/tasks/cmmlu/cmmlu_default_human_sexuality.yaml' 2024-03-18T14:20:40,753 adding 'lm_eval/tasks/cmmlu/cmmlu_default_international_law.yaml' 2024-03-18T14:20:40,754 adding 'lm_eval/tasks/cmmlu/cmmlu_default_journalism.yaml' 2024-03-18T14:20:40,755 adding 'lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml' 2024-03-18T14:20:40,757 adding 'lm_eval/tasks/cmmlu/cmmlu_default_legal_and_moral_basis.yaml' 2024-03-18T14:20:40,758 adding 'lm_eval/tasks/cmmlu/cmmlu_default_logical.yaml' 2024-03-18T14:20:40,759 adding 'lm_eval/tasks/cmmlu/cmmlu_default_machine_learning.yaml' 2024-03-18T14:20:40,760 adding 'lm_eval/tasks/cmmlu/cmmlu_default_management.yaml' 2024-03-18T14:20:40,762 adding 'lm_eval/tasks/cmmlu/cmmlu_default_marketing.yaml' 2024-03-18T14:20:40,763 adding 'lm_eval/tasks/cmmlu/cmmlu_default_marxist_theory.yaml' 2024-03-18T14:20:40,764 adding 'lm_eval/tasks/cmmlu/cmmlu_default_modern_chinese.yaml' 2024-03-18T14:20:40,765 adding 'lm_eval/tasks/cmmlu/cmmlu_default_nutrition.yaml' 2024-03-18T14:20:40,767 adding 'lm_eval/tasks/cmmlu/cmmlu_default_philosophy.yaml' 2024-03-18T14:20:40,768 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_accounting.yaml' 2024-03-18T14:20:40,769 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_law.yaml' 2024-03-18T14:20:40,770 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_medicine.yaml' 2024-03-18T14:20:40,772 adding 'lm_eval/tasks/cmmlu/cmmlu_default_professional_psychology.yaml' 2024-03-18T14:20:40,773 adding 'lm_eval/tasks/cmmlu/cmmlu_default_public_relations.yaml' 2024-03-18T14:20:40,774 adding 'lm_eval/tasks/cmmlu/cmmlu_default_security_study.yaml' 2024-03-18T14:20:40,775 adding 'lm_eval/tasks/cmmlu/cmmlu_default_sociology.yaml' 2024-03-18T14:20:40,777 adding 'lm_eval/tasks/cmmlu/cmmlu_default_sports_science.yaml' 2024-03-18T14:20:40,778 adding 'lm_eval/tasks/cmmlu/cmmlu_default_traditional_chinese_medicine.yaml' 2024-03-18T14:20:40,779 adding 'lm_eval/tasks/cmmlu/cmmlu_default_virology.yaml' 2024-03-18T14:20:40,780 adding 'lm_eval/tasks/cmmlu/cmmlu_default_world_history.yaml' 2024-03-18T14:20:40,781 adding 'lm_eval/tasks/cmmlu/cmmlu_default_world_religions.yaml' 2024-03-18T14:20:40,784 adding 'lm_eval/tasks/code_x_glue/code-text/bleu.py' 2024-03-18T14:20:40,785 adding 'lm_eval/tasks/code_x_glue/code-text/go.yaml' 2024-03-18T14:20:40,786 adding 'lm_eval/tasks/code_x_glue/code-text/java.yaml' 2024-03-18T14:20:40,787 adding 'lm_eval/tasks/code_x_glue/code-text/javascript.yaml' 2024-03-18T14:20:40,788 adding 'lm_eval/tasks/code_x_glue/code-text/php.yaml' 2024-03-18T14:20:40,790 adding 'lm_eval/tasks/code_x_glue/code-text/python.yaml' 2024-03-18T14:20:40,791 adding 'lm_eval/tasks/code_x_glue/code-text/ruby.yaml' 2024-03-18T14:20:40,792 adding 'lm_eval/tasks/code_x_glue/code-text/utils.py' 2024-03-18T14:20:40,794 adding 'lm_eval/tasks/coqa/README.md' 2024-03-18T14:20:40,795 adding 'lm_eval/tasks/coqa/default.yaml' 2024-03-18T14:20:40,796 adding 'lm_eval/tasks/coqa/utils.py' 2024-03-18T14:20:40,799 adding 'lm_eval/tasks/crows_pairs/README.md' 2024-03-18T14:20:40,800 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english.yaml' 2024-03-18T14:20:40,801 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_age.yaml' 2024-03-18T14:20:40,802 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_autre.yaml' 2024-03-18T14:20:40,803 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_disability.yaml' 2024-03-18T14:20:40,804 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_gender.yaml' 2024-03-18T14:20:40,806 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_nationality.yaml' 2024-03-18T14:20:40,807 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_physical_appearance.yaml' 2024-03-18T14:20:40,808 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_race_color.yaml' 2024-03-18T14:20:40,809 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_religion.yaml' 2024-03-18T14:20:40,810 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_sexual_orientation.yaml' 2024-03-18T14:20:40,811 adding 'lm_eval/tasks/crows_pairs/crows_pairs_english_socioeconomic.yaml' 2024-03-18T14:20:40,812 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french.yaml' 2024-03-18T14:20:40,813 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_age.yaml' 2024-03-18T14:20:40,814 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_autre.yaml' 2024-03-18T14:20:40,815 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_disability.yaml' 2024-03-18T14:20:40,817 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_gender.yaml' 2024-03-18T14:20:40,818 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_nationality.yaml' 2024-03-18T14:20:40,819 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_physical_appearance.yaml' 2024-03-18T14:20:40,820 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_race_color.yaml' 2024-03-18T14:20:40,821 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_religion.yaml' 2024-03-18T14:20:40,822 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_sexual_orientation.yaml' 2024-03-18T14:20:40,823 adding 'lm_eval/tasks/crows_pairs/crows_pairs_french_socioeconomic.yaml' 2024-03-18T14:20:40,825 adding 'lm_eval/tasks/crows_pairs/utils.py' 2024-03-18T14:20:40,827 adding 'lm_eval/tasks/csatqa/_default_csatqa_yaml' 2024-03-18T14:20:40,828 adding 'lm_eval/tasks/csatqa/_generate_configs.py' 2024-03-18T14:20:40,829 adding 'lm_eval/tasks/csatqa/csatqa_gr.yaml' 2024-03-18T14:20:40,830 adding 'lm_eval/tasks/csatqa/csatqa_li.yaml' 2024-03-18T14:20:40,831 adding 'lm_eval/tasks/csatqa/csatqa_rch.yaml' 2024-03-18T14:20:40,832 adding 'lm_eval/tasks/csatqa/csatqa_rcs.yaml' 2024-03-18T14:20:40,834 adding 'lm_eval/tasks/csatqa/csatqa_rcss.yaml' 2024-03-18T14:20:40,835 adding 'lm_eval/tasks/csatqa/csatqa_wr.yaml' 2024-03-18T14:20:40,836 adding 'lm_eval/tasks/csatqa/utils.py' 2024-03-18T14:20:40,838 adding 'lm_eval/tasks/drop/README.md' 2024-03-18T14:20:40,839 adding 'lm_eval/tasks/drop/default.yaml' 2024-03-18T14:20:40,841 adding 'lm_eval/tasks/drop/utils.py' 2024-03-18T14:20:40,843 adding 'lm_eval/tasks/eq_bench/README.md' 2024-03-18T14:20:40,844 adding 'lm_eval/tasks/eq_bench/default.yaml' 2024-03-18T14:20:40,846 adding 'lm_eval/tasks/eq_bench/utils.py' 2024-03-18T14:20:40,848 adding 'lm_eval/tasks/fld/README.md' 2024-03-18T14:20:40,849 adding 'lm_eval/tasks/fld/fld_default.yaml' 2024-03-18T14:20:40,850 adding 'lm_eval/tasks/fld/fld_star.yaml' 2024-03-18T14:20:40,852 adding 'lm_eval/tasks/french_bench/README.md' 2024-03-18T14:20:40,853 adding 'lm_eval/tasks/french_bench/_default_template_yaml' 2024-03-18T14:20:40,854 adding 'lm_eval/tasks/french_bench/french_bench_arc_challenge.yaml' 2024-03-18T14:20:40,856 adding 'lm_eval/tasks/french_bench/french_bench_boolqa.yaml' 2024-03-18T14:20:40,857 adding 'lm_eval/tasks/french_bench/french_bench_fquadv2.yaml' 2024-03-18T14:20:40,858 adding 'lm_eval/tasks/french_bench/french_bench_fquadv2_bool.yaml' 2024-03-18T14:20:40,859 adding 'lm_eval/tasks/french_bench/french_bench_fquadv2_genq.yaml' 2024-03-18T14:20:40,860 adding 'lm_eval/tasks/french_bench/french_bench_fquadv2_hasAns.yaml' 2024-03-18T14:20:40,861 adding 'lm_eval/tasks/french_bench/french_bench_grammar.yaml' 2024-03-18T14:20:40,863 adding 'lm_eval/tasks/french_bench/french_bench_hellaswag.yaml' 2024-03-18T14:20:40,864 adding 'lm_eval/tasks/french_bench/french_bench_multifquad.yaml' 2024-03-18T14:20:40,865 adding 'lm_eval/tasks/french_bench/french_bench_opus_perplexity.yaml' 2024-03-18T14:20:40,866 adding 'lm_eval/tasks/french_bench/french_bench_orangesum_abstract.yaml' 2024-03-18T14:20:40,867 adding 'lm_eval/tasks/french_bench/french_bench_orangesum_title.yaml' 2024-03-18T14:20:40,868 adding 'lm_eval/tasks/french_bench/french_bench_reading_comp.yaml' 2024-03-18T14:20:40,870 adding 'lm_eval/tasks/french_bench/french_bench_topic_based_nli.yaml' 2024-03-18T14:20:40,871 adding 'lm_eval/tasks/french_bench/french_bench_trivia.yaml' 2024-03-18T14:20:40,872 adding 'lm_eval/tasks/french_bench/french_bench_vocab.yaml' 2024-03-18T14:20:40,874 adding 'lm_eval/tasks/french_bench/french_bench_wikitext_fr.yaml' 2024-03-18T14:20:40,875 adding 'lm_eval/tasks/french_bench/french_bench_xnli.yaml' 2024-03-18T14:20:40,876 adding 'lm_eval/tasks/french_bench/preprocess_wikitext.py' 2024-03-18T14:20:40,877 adding 'lm_eval/tasks/french_bench/utils.py' 2024-03-18T14:20:40,880 adding 'lm_eval/tasks/glue/README.md' 2024-03-18T14:20:40,881 adding 'lm_eval/tasks/glue/cola/default.yaml' 2024-03-18T14:20:40,883 adding 'lm_eval/tasks/glue/mnli/default.yaml' 2024-03-18T14:20:40,884 adding 'lm_eval/tasks/glue/mnli/mismatch.yaml' 2024-03-18T14:20:40,886 adding 'lm_eval/tasks/glue/mnli/utils.py' 2024-03-18T14:20:40,887 adding 'lm_eval/tasks/glue/mrpc/default.yaml' 2024-03-18T14:20:40,889 adding 'lm_eval/tasks/glue/qnli/default.yaml' 2024-03-18T14:20:40,890 adding 'lm_eval/tasks/glue/qqp/default.yaml' 2024-03-18T14:20:40,892 adding 'lm_eval/tasks/glue/rte/default.yaml' 2024-03-18T14:20:40,894 adding 'lm_eval/tasks/glue/sst2/default.yaml' 2024-03-18T14:20:40,896 adding 'lm_eval/tasks/glue/wnli/default.yaml' 2024-03-18T14:20:40,898 adding 'lm_eval/tasks/gpqa/README.md' 2024-03-18T14:20:40,900 adding 'lm_eval/tasks/gpqa/cot_n_shot/_generate_configs.py' 2024-03-18T14:20:40,901 adding 'lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml' 2024-03-18T14:20:40,902 adding 'lm_eval/tasks/gpqa/cot_n_shot/gpqa_diamond_cot_n_shot.yaml' 2024-03-18T14:20:40,903 adding 'lm_eval/tasks/gpqa/cot_n_shot/gpqa_extended_cot_n_shot.yaml' 2024-03-18T14:20:40,904 adding 'lm_eval/tasks/gpqa/cot_n_shot/gpqa_main_cot_n_shot.yaml' 2024-03-18T14:20:40,906 adding 'lm_eval/tasks/gpqa/cot_n_shot/utils.py' 2024-03-18T14:20:40,907 adding 'lm_eval/tasks/gpqa/cot_zeroshot/_generate_configs.py' 2024-03-18T14:20:40,909 adding 'lm_eval/tasks/gpqa/cot_zeroshot/_gpqa_cot_zeroshot_yaml' 2024-03-18T14:20:40,910 adding 'lm_eval/tasks/gpqa/cot_zeroshot/gpqa_diamond_cot_zeroshot.yaml' 2024-03-18T14:20:40,911 adding 'lm_eval/tasks/gpqa/cot_zeroshot/gpqa_extended_cot_zeroshot.yaml' 2024-03-18T14:20:40,912 adding 'lm_eval/tasks/gpqa/cot_zeroshot/gpqa_main_cot_zeroshot.yaml' 2024-03-18T14:20:40,913 adding 'lm_eval/tasks/gpqa/cot_zeroshot/utils.py' 2024-03-18T14:20:40,915 adding 'lm_eval/tasks/gpqa/generative/_generate_configs.py' 2024-03-18T14:20:40,917 adding 'lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml' 2024-03-18T14:20:40,918 adding 'lm_eval/tasks/gpqa/generative/gpqa_diamond_generative_n_shot.yaml' 2024-03-18T14:20:40,919 adding 'lm_eval/tasks/gpqa/generative/gpqa_extended_generative_n_shot.yaml' 2024-03-18T14:20:40,920 adding 'lm_eval/tasks/gpqa/generative/gpqa_main_generative_n_shot.yaml' 2024-03-18T14:20:40,921 adding 'lm_eval/tasks/gpqa/generative/utils.py' 2024-03-18T14:20:40,922 adding 'lm_eval/tasks/gpqa/n_shot/_generate_configs.py' 2024-03-18T14:20:40,924 adding 'lm_eval/tasks/gpqa/n_shot/_gpqa_n_shot_yaml' 2024-03-18T14:20:40,925 adding 'lm_eval/tasks/gpqa/n_shot/gpqa_diamond_n_shot.yaml' 2024-03-18T14:20:40,926 adding 'lm_eval/tasks/gpqa/n_shot/gpqa_extended_n_shot.yaml' 2024-03-18T14:20:40,927 adding 'lm_eval/tasks/gpqa/n_shot/gpqa_main_n_shot.yaml' 2024-03-18T14:20:40,928 adding 'lm_eval/tasks/gpqa/n_shot/utils.py' 2024-03-18T14:20:40,930 adding 'lm_eval/tasks/gpqa/zeroshot/_generate_configs.py' 2024-03-18T14:20:40,931 adding 'lm_eval/tasks/gpqa/zeroshot/_gpqa_zeroshot_yaml' 2024-03-18T14:20:40,933 adding 'lm_eval/tasks/gpqa/zeroshot/gpqa_diamond_zeroshot.yaml' 2024-03-18T14:20:40,934 adding 'lm_eval/tasks/gpqa/zeroshot/gpqa_extended_zeroshot.yaml' 2024-03-18T14:20:40,935 adding 'lm_eval/tasks/gpqa/zeroshot/gpqa_main_zeroshot.yaml' 2024-03-18T14:20:40,936 adding 'lm_eval/tasks/gpqa/zeroshot/utils.py' 2024-03-18T14:20:40,937 adding 'lm_eval/tasks/gsm8k/README.md' 2024-03-18T14:20:40,939 adding 'lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml' 2024-03-18T14:20:40,940 adding 'lm_eval/tasks/gsm8k/gsm8k-cot-zeroshot.yaml' 2024-03-18T14:20:40,941 adding 'lm_eval/tasks/gsm8k/gsm8k-cot.yaml' 2024-03-18T14:20:40,943 adding 'lm_eval/tasks/gsm8k/gsm8k.yaml' 2024-03-18T14:20:40,945 adding 'lm_eval/tasks/haerae/README.md' 2024-03-18T14:20:40,946 adding 'lm_eval/tasks/haerae/_default_haerae_yaml' 2024-03-18T14:20:40,947 adding 'lm_eval/tasks/haerae/haerae_gk.yaml' 2024-03-18T14:20:40,948 adding 'lm_eval/tasks/haerae/haerae_hi.yaml' 2024-03-18T14:20:40,949 adding 'lm_eval/tasks/haerae/haerae_lw.yaml' 2024-03-18T14:20:40,950 adding 'lm_eval/tasks/haerae/haerae_rw.yaml' 2024-03-18T14:20:40,951 adding 'lm_eval/tasks/haerae/haerae_sn.yaml' 2024-03-18T14:20:40,953 adding 'lm_eval/tasks/headqa/README.md' 2024-03-18T14:20:40,954 adding 'lm_eval/tasks/headqa/headqa_en.yaml' 2024-03-18T14:20:40,955 adding 'lm_eval/tasks/headqa/headqa_es.yaml' 2024-03-18T14:20:40,957 adding 'lm_eval/tasks/hellaswag/README.md' 2024-03-18T14:20:40,959 adding 'lm_eval/tasks/hellaswag/hellaswag.yaml' 2024-03-18T14:20:40,960 adding 'lm_eval/tasks/hellaswag/utils.py' 2024-03-18T14:20:40,962 adding 'lm_eval/tasks/hendrycks_ethics/README.md' 2024-03-18T14:20:40,963 adding 'lm_eval/tasks/hendrycks_ethics/commonsense.yaml' 2024-03-18T14:20:40,964 adding 'lm_eval/tasks/hendrycks_ethics/deontology.yaml' 2024-03-18T14:20:40,965 adding 'lm_eval/tasks/hendrycks_ethics/justice.yaml' 2024-03-18T14:20:40,966 adding 'lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml' 2024-03-18T14:20:40,968 adding 'lm_eval/tasks/hendrycks_ethics/utilitarianism_original_yaml' 2024-03-18T14:20:40,969 adding 'lm_eval/tasks/hendrycks_ethics/utils.py' 2024-03-18T14:20:40,970 adding 'lm_eval/tasks/hendrycks_ethics/virtue.yaml' 2024-03-18T14:20:40,972 adding 'lm_eval/tasks/ifeval/README.md' 2024-03-18T14:20:40,973 adding 'lm_eval/tasks/ifeval/ifeval.yaml' 2024-03-18T14:20:40,978 adding 'lm_eval/tasks/ifeval/instructions.py' 2024-03-18T14:20:40,981 adding 'lm_eval/tasks/ifeval/instructions_registry.py' 2024-03-18T14:20:40,984 adding 'lm_eval/tasks/ifeval/instructions_util.py' 2024-03-18T14:20:40,986 adding 'lm_eval/tasks/ifeval/utils.py' 2024-03-18T14:20:40,988 adding 'lm_eval/tasks/kmmlu/README.md' 2024-03-18T14:20:40,990 adding 'lm_eval/tasks/kmmlu/cot_hard/_cot_kmmlu_yaml' 2024-03-18T14:20:40,992 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_accounting.yaml' 2024-03-18T14:20:40,994 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_agricultural_sciences.yaml' 2024-03-18T14:20:40,996 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_aviation_engineering_and_maintenance.yaml' 2024-03-18T14:20:40,998 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_biology.yaml' 2024-03-18T14:20:41,000 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemical_engineering.yaml' 2024-03-18T14:20:41,002 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_chemistry.yaml' 2024-03-18T14:20:41,004 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_civil_engineering.yaml' 2024-03-18T14:20:41,006 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_computer_science.yaml' 2024-03-18T14:20:41,008 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_construction.yaml' 2024-03-18T14:20:41,010 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_criminal_law.yaml' 2024-03-18T14:20:41,013 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_ecology.yaml' 2024-03-18T14:20:41,015 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_economics.yaml' 2024-03-18T14:20:41,017 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_education.yaml' 2024-03-18T14:20:41,020 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electrical_engineering.yaml' 2024-03-18T14:20:41,021 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_electronics_engineering.yaml' 2024-03-18T14:20:41,023 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_energy_management.yaml' 2024-03-18T14:20:41,025 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_environmental_science.yaml' 2024-03-18T14:20:41,027 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_fashion.yaml' 2024-03-18T14:20:41,029 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_food_processing.yaml' 2024-03-18T14:20:41,031 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_gas_technology_and_engineering.yaml' 2024-03-18T14:20:41,033 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_geomatics.yaml' 2024-03-18T14:20:41,035 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_health.yaml' 2024-03-18T14:20:41,036 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_industrial_engineer.yaml' 2024-03-18T14:20:41,038 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_information_technology.yaml' 2024-03-18T14:20:41,040 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_interior_architecture_and_design.yaml' 2024-03-18T14:20:41,043 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_korean_history.yaml' 2024-03-18T14:20:41,045 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_law.yaml' 2024-03-18T14:20:41,047 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_machine_design_and_manufacturing.yaml' 2024-03-18T14:20:41,049 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_management.yaml' 2024-03-18T14:20:41,051 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_maritime_engineering.yaml' 2024-03-18T14:20:41,054 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_marketing.yaml' 2024-03-18T14:20:41,056 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_materials_engineering.yaml' 2024-03-18T14:20:41,058 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_math.yaml' 2024-03-18T14:20:41,061 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_mechanical_engineering.yaml' 2024-03-18T14:20:41,063 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_nondestructive_testing.yaml' 2024-03-18T14:20:41,065 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_patent.yaml' 2024-03-18T14:20:41,068 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_political_science_and_sociology.yaml' 2024-03-18T14:20:41,070 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_psychology.yaml' 2024-03-18T14:20:41,072 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_public_safety.yaml' 2024-03-18T14:20:41,074 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_railway_and_automotive_engineering.yaml' 2024-03-18T14:20:41,076 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_real_estate.yaml' 2024-03-18T14:20:41,078 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_refrigerating_machinery.yaml' 2024-03-18T14:20:41,080 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_social_welfare.yaml' 2024-03-18T14:20:41,082 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_taxation.yaml' 2024-03-18T14:20:41,084 adding 'lm_eval/tasks/kmmlu/cot_hard/kmmlu_cot_hard_telecommunications_and_wireless_technology.yaml' 2024-03-18T14:20:41,086 adding 'lm_eval/tasks/kmmlu/direct/_direct_kmmlu_yaml' 2024-03-18T14:20:41,087 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_accounting.yaml' 2024-03-18T14:20:41,088 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_agricultural_sciences.yaml' 2024-03-18T14:20:41,089 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_aviation_engineering_and_maintenance.yaml' 2024-03-18T14:20:41,091 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_biology.yaml' 2024-03-18T14:20:41,092 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemical_engineering.yaml' 2024-03-18T14:20:41,093 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_chemistry.yaml' 2024-03-18T14:20:41,094 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_civil_engineering.yaml' 2024-03-18T14:20:41,095 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_computer_science.yaml' 2024-03-18T14:20:41,096 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_construction.yaml' 2024-03-18T14:20:41,097 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_criminal_law.yaml' 2024-03-18T14:20:41,099 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_ecology.yaml' 2024-03-18T14:20:41,100 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_economics.yaml' 2024-03-18T14:20:41,101 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_education.yaml' 2024-03-18T14:20:41,102 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_electrical_engineering.yaml' 2024-03-18T14:20:41,103 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_electronics_engineering.yaml' 2024-03-18T14:20:41,105 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_energy_management.yaml' 2024-03-18T14:20:41,106 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_environmental_science.yaml' 2024-03-18T14:20:41,108 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_fashion.yaml' 2024-03-18T14:20:41,109 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_food_processing.yaml' 2024-03-18T14:20:41,110 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_gas_technology_and_engineering.yaml' 2024-03-18T14:20:41,112 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_geomatics.yaml' 2024-03-18T14:20:41,113 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_health.yaml' 2024-03-18T14:20:41,114 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_industrial_engineer.yaml' 2024-03-18T14:20:41,115 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_information_technology.yaml' 2024-03-18T14:20:41,116 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_interior_architecture_and_design.yaml' 2024-03-18T14:20:41,117 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_korean_history.yaml' 2024-03-18T14:20:41,119 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_law.yaml' 2024-03-18T14:20:41,120 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_machine_design_and_manufacturing.yaml' 2024-03-18T14:20:41,121 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_management.yaml' 2024-03-18T14:20:41,122 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_maritime_engineering.yaml' 2024-03-18T14:20:41,123 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_marketing.yaml' 2024-03-18T14:20:41,125 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_materials_engineering.yaml' 2024-03-18T14:20:41,126 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_math.yaml' 2024-03-18T14:20:41,127 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_mechanical_engineering.yaml' 2024-03-18T14:20:41,129 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_nondestructive_testing.yaml' 2024-03-18T14:20:41,130 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_patent.yaml' 2024-03-18T14:20:41,131 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_political_science_and_sociology.yaml' 2024-03-18T14:20:41,132 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_psychology.yaml' 2024-03-18T14:20:41,133 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_public_safety.yaml' 2024-03-18T14:20:41,135 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_railway_and_automotive_engineering.yaml' 2024-03-18T14:20:41,136 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_real_estate.yaml' 2024-03-18T14:20:41,137 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_refrigerating_machinery.yaml' 2024-03-18T14:20:41,138 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_social_welfare.yaml' 2024-03-18T14:20:41,139 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_taxation.yaml' 2024-03-18T14:20:41,141 adding 'lm_eval/tasks/kmmlu/direct/kmmlu_direct_telecommunications_and_wireless_technology.yaml' 2024-03-18T14:20:41,148 adding 'lm_eval/tasks/kmmlu/direct_hard/_direct_hard_kmmlu_yaml' 2024-03-18T14:20:41,149 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_accounting.yaml' 2024-03-18T14:20:41,151 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_agricultural_sciences.yaml' 2024-03-18T14:20:41,152 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_aviation_engineering_and_maintenance.yaml' 2024-03-18T14:20:41,153 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_biology.yaml' 2024-03-18T14:20:41,154 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemical_engineering.yaml' 2024-03-18T14:20:41,156 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_chemistry.yaml' 2024-03-18T14:20:41,157 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_civil_engineering.yaml' 2024-03-18T14:20:41,158 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_computer_science.yaml' 2024-03-18T14:20:41,159 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_construction.yaml' 2024-03-18T14:20:41,160 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_criminal_law.yaml' 2024-03-18T14:20:41,161 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_ecology.yaml' 2024-03-18T14:20:41,162 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_economics.yaml' 2024-03-18T14:20:41,163 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_education.yaml' 2024-03-18T14:20:41,164 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electrical_engineering.yaml' 2024-03-18T14:20:41,166 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_electronics_engineering.yaml' 2024-03-18T14:20:41,167 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_energy_management.yaml' 2024-03-18T14:20:41,168 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_environmental_science.yaml' 2024-03-18T14:20:41,169 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_fashion.yaml' 2024-03-18T14:20:41,170 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_food_processing.yaml' 2024-03-18T14:20:41,171 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_gas_technology_and_engineering.yaml' 2024-03-18T14:20:41,172 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_geomatics.yaml' 2024-03-18T14:20:41,174 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_health.yaml' 2024-03-18T14:20:41,175 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_industrial_engineer.yaml' 2024-03-18T14:20:41,176 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_information_technology.yaml' 2024-03-18T14:20:41,177 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_interior_architecture_and_design.yaml' 2024-03-18T14:20:41,178 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_korean_history.yaml' 2024-03-18T14:20:41,179 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_law.yaml' 2024-03-18T14:20:41,180 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_machine_design_and_manufacturing.yaml' 2024-03-18T14:20:41,181 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_management.yaml' 2024-03-18T14:20:41,183 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_maritime_engineering.yaml' 2024-03-18T14:20:41,184 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_marketing.yaml' 2024-03-18T14:20:41,185 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_materials_engineering.yaml' 2024-03-18T14:20:41,186 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_math.yaml' 2024-03-18T14:20:41,187 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_mechanical_engineering.yaml' 2024-03-18T14:20:41,188 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_nondestructive_testing.yaml' 2024-03-18T14:20:41,189 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_patent.yaml' 2024-03-18T14:20:41,190 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_political_science_and_sociology.yaml' 2024-03-18T14:20:41,191 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_psychology.yaml' 2024-03-18T14:20:41,192 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_public_safety.yaml' 2024-03-18T14:20:41,193 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_railway_and_automotive_engineering.yaml' 2024-03-18T14:20:41,194 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_real_estate.yaml' 2024-03-18T14:20:41,196 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_refrigerating_machinery.yaml' 2024-03-18T14:20:41,197 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_social_welfare.yaml' 2024-03-18T14:20:41,198 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_taxation.yaml' 2024-03-18T14:20:41,199 adding 'lm_eval/tasks/kmmlu/direct_hard/kmmlu_direct_hard_telecommunications_and_wireless_technology.yaml' 2024-03-18T14:20:41,202 adding 'lm_eval/tasks/kmmlu/hard/_hard_kmmlu_yaml' 2024-03-18T14:20:41,203 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_accounting.yaml' 2024-03-18T14:20:41,204 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_agricultural_sciences.yaml' 2024-03-18T14:20:41,205 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_aviation_engineering_and_maintenance.yaml' 2024-03-18T14:20:41,207 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_biology.yaml' 2024-03-18T14:20:41,208 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemical_engineering.yaml' 2024-03-18T14:20:41,209 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_chemistry.yaml' 2024-03-18T14:20:41,210 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_civil_engineering.yaml' 2024-03-18T14:20:41,212 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_computer_science.yaml' 2024-03-18T14:20:41,213 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_construction.yaml' 2024-03-18T14:20:41,214 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_criminal_law.yaml' 2024-03-18T14:20:41,215 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_ecology.yaml' 2024-03-18T14:20:41,217 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_economics.yaml' 2024-03-18T14:20:41,218 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_education.yaml' 2024-03-18T14:20:41,219 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_electrical_engineering.yaml' 2024-03-18T14:20:41,220 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_electronics_engineering.yaml' 2024-03-18T14:20:41,222 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_energy_management.yaml' 2024-03-18T14:20:41,223 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_environmental_science.yaml' 2024-03-18T14:20:41,224 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_fashion.yaml' 2024-03-18T14:20:41,225 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_food_processing.yaml' 2024-03-18T14:20:41,226 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_gas_technology_and_engineering.yaml' 2024-03-18T14:20:41,227 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_geomatics.yaml' 2024-03-18T14:20:41,229 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_health.yaml' 2024-03-18T14:20:41,230 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_industrial_engineer.yaml' 2024-03-18T14:20:41,231 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_information_technology.yaml' 2024-03-18T14:20:41,232 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_interior_architecture_and_design.yaml' 2024-03-18T14:20:41,234 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_korean_history.yaml' 2024-03-18T14:20:41,235 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_law.yaml' 2024-03-18T14:20:41,236 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_machine_design_and_manufacturing.yaml' 2024-03-18T14:20:41,237 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_management.yaml' 2024-03-18T14:20:41,238 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_maritime_engineering.yaml' 2024-03-18T14:20:41,239 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_marketing.yaml' 2024-03-18T14:20:41,240 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_materials_engineering.yaml' 2024-03-18T14:20:41,242 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_math.yaml' 2024-03-18T14:20:41,243 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_mechanical_engineering.yaml' 2024-03-18T14:20:41,244 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_nondestructive_testing.yaml' 2024-03-18T14:20:41,245 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_patent.yaml' 2024-03-18T14:20:41,246 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_political_science_and_sociology.yaml' 2024-03-18T14:20:41,247 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_psychology.yaml' 2024-03-18T14:20:41,249 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_public_safety.yaml' 2024-03-18T14:20:41,250 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_railway_and_automotive_engineering.yaml' 2024-03-18T14:20:41,251 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_real_estate.yaml' 2024-03-18T14:20:41,252 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_refrigerating_machinery.yaml' 2024-03-18T14:20:41,253 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_social_welfare.yaml' 2024-03-18T14:20:41,254 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_taxation.yaml' 2024-03-18T14:20:41,255 adding 'lm_eval/tasks/kmmlu/hard/kmmlu_hard_telecommunications_and_wireless_technology.yaml' 2024-03-18T14:20:41,257 adding 'lm_eval/tasks/kobest/README.md' 2024-03-18T14:20:41,258 adding 'lm_eval/tasks/kobest/kobest_boolq.yaml' 2024-03-18T14:20:41,260 adding 'lm_eval/tasks/kobest/kobest_copa.yaml' 2024-03-18T14:20:41,261 adding 'lm_eval/tasks/kobest/kobest_hellaswag.yaml' 2024-03-18T14:20:41,263 adding 'lm_eval/tasks/kobest/kobest_sentineg.yaml' 2024-03-18T14:20:41,264 adding 'lm_eval/tasks/kobest/kobest_wic.yaml' 2024-03-18T14:20:41,265 adding 'lm_eval/tasks/kobest/utils.py' 2024-03-18T14:20:41,267 adding 'lm_eval/tasks/kormedmcqa/README.md' 2024-03-18T14:20:41,269 adding 'lm_eval/tasks/kormedmcqa/kormedmcqa_doctor.yaml' 2024-03-18T14:20:41,270 adding 'lm_eval/tasks/kormedmcqa/kormedmcqa_nurse.yaml' 2024-03-18T14:20:41,272 adding 'lm_eval/tasks/kormedmcqa/kormedmcqa_pharm.yaml' 2024-03-18T14:20:41,274 adding 'lm_eval/tasks/lambada/README.md' 2024-03-18T14:20:41,275 adding 'lm_eval/tasks/lambada/lambada_openai.yaml' 2024-03-18T14:20:41,277 adding 'lm_eval/tasks/lambada/lambada_standard.yaml' 2024-03-18T14:20:41,279 adding 'lm_eval/tasks/lambada_cloze/README.md' 2024-03-18T14:20:41,280 adding 'lm_eval/tasks/lambada_cloze/lambada_openai_cloze.yaml' 2024-03-18T14:20:41,281 adding 'lm_eval/tasks/lambada_cloze/lambada_standard_cloze.yaml' 2024-03-18T14:20:41,283 adding 'lm_eval/tasks/lambada_multilingual/README.md' 2024-03-18T14:20:41,285 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml' 2024-03-18T14:20:41,286 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml' 2024-03-18T14:20:41,287 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_es.yaml' 2024-03-18T14:20:41,288 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_fr.yaml' 2024-03-18T14:20:41,289 adding 'lm_eval/tasks/lambada_multilingual/lambada_mt_it.yaml' 2024-03-18T14:20:41,291 adding 'lm_eval/tasks/logiqa/README.md' 2024-03-18T14:20:41,293 adding 'lm_eval/tasks/logiqa/logiqa.yaml' 2024-03-18T14:20:41,294 adding 'lm_eval/tasks/logiqa/utils_logiqa.py' 2024-03-18T14:20:41,296 adding 'lm_eval/tasks/logiqa2/README.md' 2024-03-18T14:20:41,297 adding 'lm_eval/tasks/logiqa2/logieval.yaml' 2024-03-18T14:20:41,298 adding 'lm_eval/tasks/logiqa2/logiqa2.yaml' 2024-03-18T14:20:41,300 adding 'lm_eval/tasks/logiqa2/utils_logiqa2.py' 2024-03-18T14:20:41,301 adding 'lm_eval/tasks/mathqa/README.md' 2024-03-18T14:20:41,303 adding 'lm_eval/tasks/mathqa/mathqa.yaml' 2024-03-18T14:20:41,304 adding 'lm_eval/tasks/mathqa/utils.py' 2024-03-18T14:20:41,306 adding 'lm_eval/tasks/mc_taco/README.md' 2024-03-18T14:20:41,307 adding 'lm_eval/tasks/mc_taco/default.yaml' 2024-03-18T14:20:41,308 adding 'lm_eval/tasks/medmcqa/medmcqa.yaml' 2024-03-18T14:20:41,310 adding 'lm_eval/tasks/medmcqa/utils_medmcqa.py' 2024-03-18T14:20:41,311 adding 'lm_eval/tasks/medqa/medqa.yaml' 2024-03-18T14:20:41,312 adding 'lm_eval/tasks/medqa/preprocess_medqa.py' 2024-03-18T14:20:41,314 adding 'lm_eval/tasks/mgsm/README.md' 2024-03-18T14:20:41,315 adding 'lm_eval/tasks/mgsm/gen_yaml.sh' 2024-03-18T14:20:41,317 adding 'lm_eval/tasks/mgsm/utils.py' 2024-03-18T14:20:41,319 adding 'lm_eval/tasks/mgsm/direct/direct_yaml' 2024-03-18T14:20:41,321 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_bn.yaml' 2024-03-18T14:20:41,322 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_de.yaml' 2024-03-18T14:20:41,323 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_en.yaml' 2024-03-18T14:20:41,325 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_es.yaml' 2024-03-18T14:20:41,326 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_fr.yaml' 2024-03-18T14:20:41,327 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_ja.yaml' 2024-03-18T14:20:41,328 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_ru.yaml' 2024-03-18T14:20:41,330 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_sw.yaml' 2024-03-18T14:20:41,331 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_te.yaml' 2024-03-18T14:20:41,332 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_th.yaml' 2024-03-18T14:20:41,334 adding 'lm_eval/tasks/mgsm/direct/mgsm_direct_zh.yaml' 2024-03-18T14:20:41,335 adding 'lm_eval/tasks/mgsm/en_cot/cot_yaml' 2024-03-18T14:20:41,337 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_bn.yaml' 2024-03-18T14:20:41,338 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_de.yaml' 2024-03-18T14:20:41,339 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_en.yaml' 2024-03-18T14:20:41,340 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_es.yaml' 2024-03-18T14:20:41,342 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_fr.yaml' 2024-03-18T14:20:41,343 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ja.yaml' 2024-03-18T14:20:41,344 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_ru.yaml' 2024-03-18T14:20:41,346 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_sw.yaml' 2024-03-18T14:20:41,347 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_te.yaml' 2024-03-18T14:20:41,348 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_th.yaml' 2024-03-18T14:20:41,350 adding 'lm_eval/tasks/mgsm/en_cot/mgsm_en_cot_zh.yaml' 2024-03-18T14:20:41,352 adding 'lm_eval/tasks/mgsm/native_cot/cot_yaml' 2024-03-18T14:20:41,353 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_bn.yaml' 2024-03-18T14:20:41,354 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_de.yaml' 2024-03-18T14:20:41,355 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_en.yaml' 2024-03-18T14:20:41,357 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_es.yaml' 2024-03-18T14:20:41,358 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_fr.yaml' 2024-03-18T14:20:41,360 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ja.yaml' 2024-03-18T14:20:41,361 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_ru.yaml' 2024-03-18T14:20:41,362 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_sw.yaml' 2024-03-18T14:20:41,364 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_te.yaml' 2024-03-18T14:20:41,365 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_th.yaml' 2024-03-18T14:20:41,367 adding 'lm_eval/tasks/mgsm/native_cot/mgsm_native_cot_zh.yaml' 2024-03-18T14:20:41,369 adding 'lm_eval/tasks/minerva_math/README.md' 2024-03-18T14:20:41,370 adding 'lm_eval/tasks/minerva_math/minerva_math_algebra.yaml' 2024-03-18T14:20:41,371 adding 'lm_eval/tasks/minerva_math/minerva_math_counting_and_prob.yaml' 2024-03-18T14:20:41,372 adding 'lm_eval/tasks/minerva_math/minerva_math_geometry.yaml' 2024-03-18T14:20:41,373 adding 'lm_eval/tasks/minerva_math/minerva_math_intermediate_algebra.yaml' 2024-03-18T14:20:41,375 adding 'lm_eval/tasks/minerva_math/minerva_math_num_theory.yaml' 2024-03-18T14:20:41,376 adding 'lm_eval/tasks/minerva_math/minerva_math_prealgebra.yaml' 2024-03-18T14:20:41,377 adding 'lm_eval/tasks/minerva_math/minerva_math_precalc.yaml' 2024-03-18T14:20:41,379 adding 'lm_eval/tasks/minerva_math/utils.py' 2024-03-18T14:20:41,381 adding 'lm_eval/tasks/mmlu/_generate_configs.py' 2024-03-18T14:20:41,384 adding 'lm_eval/tasks/mmlu/default/_default_template_yaml' 2024-03-18T14:20:41,385 adding 'lm_eval/tasks/mmlu/default/_mmlu.yaml' 2024-03-18T14:20:41,386 adding 'lm_eval/tasks/mmlu/default/mmlu_abstract_algebra.yaml' 2024-03-18T14:20:41,387 adding 'lm_eval/tasks/mmlu/default/mmlu_anatomy.yaml' 2024-03-18T14:20:41,388 adding 'lm_eval/tasks/mmlu/default/mmlu_astronomy.yaml' 2024-03-18T14:20:41,389 adding 'lm_eval/tasks/mmlu/default/mmlu_business_ethics.yaml' 2024-03-18T14:20:41,390 adding 'lm_eval/tasks/mmlu/default/mmlu_clinical_knowledge.yaml' 2024-03-18T14:20:41,391 adding 'lm_eval/tasks/mmlu/default/mmlu_college_biology.yaml' 2024-03-18T14:20:41,393 adding 'lm_eval/tasks/mmlu/default/mmlu_college_chemistry.yaml' 2024-03-18T14:20:41,394 adding 'lm_eval/tasks/mmlu/default/mmlu_college_computer_science.yaml' 2024-03-18T14:20:41,395 adding 'lm_eval/tasks/mmlu/default/mmlu_college_mathematics.yaml' 2024-03-18T14:20:41,396 adding 'lm_eval/tasks/mmlu/default/mmlu_college_medicine.yaml' 2024-03-18T14:20:41,397 adding 'lm_eval/tasks/mmlu/default/mmlu_college_physics.yaml' 2024-03-18T14:20:41,399 adding 'lm_eval/tasks/mmlu/default/mmlu_computer_security.yaml' 2024-03-18T14:20:41,400 adding 'lm_eval/tasks/mmlu/default/mmlu_conceptual_physics.yaml' 2024-03-18T14:20:41,401 adding 'lm_eval/tasks/mmlu/default/mmlu_econometrics.yaml' 2024-03-18T14:20:41,402 adding 'lm_eval/tasks/mmlu/default/mmlu_electrical_engineering.yaml' 2024-03-18T14:20:41,403 adding 'lm_eval/tasks/mmlu/default/mmlu_elementary_mathematics.yaml' 2024-03-18T14:20:41,404 adding 'lm_eval/tasks/mmlu/default/mmlu_formal_logic.yaml' 2024-03-18T14:20:41,406 adding 'lm_eval/tasks/mmlu/default/mmlu_global_facts.yaml' 2024-03-18T14:20:41,407 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_biology.yaml' 2024-03-18T14:20:41,408 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_chemistry.yaml' 2024-03-18T14:20:41,409 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_computer_science.yaml' 2024-03-18T14:20:41,411 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_european_history.yaml' 2024-03-18T14:20:41,412 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_geography.yaml' 2024-03-18T14:20:41,413 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:41,414 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:41,415 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_mathematics.yaml' 2024-03-18T14:20:41,416 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_microeconomics.yaml' 2024-03-18T14:20:41,418 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_physics.yaml' 2024-03-18T14:20:41,419 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_psychology.yaml' 2024-03-18T14:20:41,420 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_statistics.yaml' 2024-03-18T14:20:41,421 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml' 2024-03-18T14:20:41,422 adding 'lm_eval/tasks/mmlu/default/mmlu_high_school_world_history.yaml' 2024-03-18T14:20:41,424 adding 'lm_eval/tasks/mmlu/default/mmlu_human_aging.yaml' 2024-03-18T14:20:41,425 adding 'lm_eval/tasks/mmlu/default/mmlu_human_sexuality.yaml' 2024-03-18T14:20:41,426 adding 'lm_eval/tasks/mmlu/default/mmlu_international_law.yaml' 2024-03-18T14:20:41,427 adding 'lm_eval/tasks/mmlu/default/mmlu_jurisprudence.yaml' 2024-03-18T14:20:41,428 adding 'lm_eval/tasks/mmlu/default/mmlu_logical_fallacies.yaml' 2024-03-18T14:20:41,429 adding 'lm_eval/tasks/mmlu/default/mmlu_machine_learning.yaml' 2024-03-18T14:20:41,430 adding 'lm_eval/tasks/mmlu/default/mmlu_management.yaml' 2024-03-18T14:20:41,431 adding 'lm_eval/tasks/mmlu/default/mmlu_marketing.yaml' 2024-03-18T14:20:41,433 adding 'lm_eval/tasks/mmlu/default/mmlu_medical_genetics.yaml' 2024-03-18T14:20:41,434 adding 'lm_eval/tasks/mmlu/default/mmlu_miscellaneous.yaml' 2024-03-18T14:20:41,435 adding 'lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml' 2024-03-18T14:20:41,436 adding 'lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml' 2024-03-18T14:20:41,437 adding 'lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml' 2024-03-18T14:20:41,438 adding 'lm_eval/tasks/mmlu/default/mmlu_philosophy.yaml' 2024-03-18T14:20:41,439 adding 'lm_eval/tasks/mmlu/default/mmlu_prehistory.yaml' 2024-03-18T14:20:41,441 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_accounting.yaml' 2024-03-18T14:20:41,442 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_law.yaml' 2024-03-18T14:20:41,443 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_medicine.yaml' 2024-03-18T14:20:41,444 adding 'lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml' 2024-03-18T14:20:41,445 adding 'lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml' 2024-03-18T14:20:41,446 adding 'lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml' 2024-03-18T14:20:41,447 adding 'lm_eval/tasks/mmlu/default/mmlu_sociology.yaml' 2024-03-18T14:20:41,448 adding 'lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml' 2024-03-18T14:20:41,450 adding 'lm_eval/tasks/mmlu/default/mmlu_virology.yaml' 2024-03-18T14:20:41,451 adding 'lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml' 2024-03-18T14:20:41,491 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_cot_prompts.json' 2024-03-18T14:20:41,494 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu.yaml' 2024-03-18T14:20:41,495 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/_mmlu_flan_cot_fewshot_template_yaml' 2024-03-18T14:20:41,497 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml' 2024-03-18T14:20:41,498 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml' 2024-03-18T14:20:41,500 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml' 2024-03-18T14:20:41,501 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml' 2024-03-18T14:20:41,502 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml' 2024-03-18T14:20:41,504 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml' 2024-03-18T14:20:41,505 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml' 2024-03-18T14:20:41,507 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml' 2024-03-18T14:20:41,509 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml' 2024-03-18T14:20:41,510 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml' 2024-03-18T14:20:41,511 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml' 2024-03-18T14:20:41,513 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml' 2024-03-18T14:20:41,514 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml' 2024-03-18T14:20:41,516 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml' 2024-03-18T14:20:41,517 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml' 2024-03-18T14:20:41,519 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml' 2024-03-18T14:20:41,520 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml' 2024-03-18T14:20:41,522 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml' 2024-03-18T14:20:41,523 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml' 2024-03-18T14:20:41,525 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml' 2024-03-18T14:20:41,526 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml' 2024-03-18T14:20:41,529 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml' 2024-03-18T14:20:41,531 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml' 2024-03-18T14:20:41,532 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:41,533 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:41,534 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml' 2024-03-18T14:20:41,536 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml' 2024-03-18T14:20:41,537 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml' 2024-03-18T14:20:41,539 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml' 2024-03-18T14:20:41,540 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml' 2024-03-18T14:20:41,543 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml' 2024-03-18T14:20:41,544 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml' 2024-03-18T14:20:41,546 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml' 2024-03-18T14:20:41,547 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml' 2024-03-18T14:20:41,548 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml' 2024-03-18T14:20:41,550 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml' 2024-03-18T14:20:41,551 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml' 2024-03-18T14:20:41,553 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml' 2024-03-18T14:20:41,554 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml' 2024-03-18T14:20:41,555 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml' 2024-03-18T14:20:41,557 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml' 2024-03-18T14:20:41,558 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml' 2024-03-18T14:20:41,560 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml' 2024-03-18T14:20:41,561 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml' 2024-03-18T14:20:41,563 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml' 2024-03-18T14:20:41,564 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml' 2024-03-18T14:20:41,565 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml' 2024-03-18T14:20:41,567 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml' 2024-03-18T14:20:41,569 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml' 2024-03-18T14:20:41,571 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml' 2024-03-18T14:20:41,572 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml' 2024-03-18T14:20:41,574 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml' 2024-03-18T14:20:41,575 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml' 2024-03-18T14:20:41,577 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml' 2024-03-18T14:20:41,578 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml' 2024-03-18T14:20:41,580 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml' 2024-03-18T14:20:41,582 adding 'lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml' 2024-03-18T14:20:41,584 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml' 2024-03-18T14:20:41,585 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml' 2024-03-18T14:20:41,587 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml' 2024-03-18T14:20:41,588 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml' 2024-03-18T14:20:41,589 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml' 2024-03-18T14:20:41,590 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml' 2024-03-18T14:20:41,592 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml' 2024-03-18T14:20:41,593 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml' 2024-03-18T14:20:41,594 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml' 2024-03-18T14:20:41,595 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml' 2024-03-18T14:20:41,596 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml' 2024-03-18T14:20:41,597 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml' 2024-03-18T14:20:41,599 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_physics.yaml' 2024-03-18T14:20:41,600 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_computer_security.yaml' 2024-03-18T14:20:41,601 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_conceptual_physics.yaml' 2024-03-18T14:20:41,602 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_econometrics.yaml' 2024-03-18T14:20:41,603 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_electrical_engineering.yaml' 2024-03-18T14:20:41,604 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_elementary_mathematics.yaml' 2024-03-18T14:20:41,605 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_formal_logic.yaml' 2024-03-18T14:20:41,607 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_global_facts.yaml' 2024-03-18T14:20:41,608 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_biology.yaml' 2024-03-18T14:20:41,609 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_chemistry.yaml' 2024-03-18T14:20:41,610 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_computer_science.yaml' 2024-03-18T14:20:41,611 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_european_history.yaml' 2024-03-18T14:20:41,612 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_geography.yaml' 2024-03-18T14:20:41,614 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:41,615 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:41,616 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_mathematics.yaml' 2024-03-18T14:20:41,617 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_microeconomics.yaml' 2024-03-18T14:20:41,618 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_physics.yaml' 2024-03-18T14:20:41,620 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_psychology.yaml' 2024-03-18T14:20:41,621 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_statistics.yaml' 2024-03-18T14:20:41,622 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_us_history.yaml' 2024-03-18T14:20:41,624 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_high_school_world_history.yaml' 2024-03-18T14:20:41,625 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_aging.yaml' 2024-03-18T14:20:41,626 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_human_sexuality.yaml' 2024-03-18T14:20:41,627 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_international_law.yaml' 2024-03-18T14:20:41,628 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_jurisprudence.yaml' 2024-03-18T14:20:41,630 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_logical_fallacies.yaml' 2024-03-18T14:20:41,631 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_machine_learning.yaml' 2024-03-18T14:20:41,632 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_management.yaml' 2024-03-18T14:20:41,633 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_marketing.yaml' 2024-03-18T14:20:41,634 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_medical_genetics.yaml' 2024-03-18T14:20:41,636 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_miscellaneous.yaml' 2024-03-18T14:20:41,637 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_disputes.yaml' 2024-03-18T14:20:41,638 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_moral_scenarios.yaml' 2024-03-18T14:20:41,640 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_nutrition.yaml' 2024-03-18T14:20:41,641 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_philosophy.yaml' 2024-03-18T14:20:41,642 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_prehistory.yaml' 2024-03-18T14:20:41,644 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_accounting.yaml' 2024-03-18T14:20:41,645 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_law.yaml' 2024-03-18T14:20:41,646 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_medicine.yaml' 2024-03-18T14:20:41,648 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_professional_psychology.yaml' 2024-03-18T14:20:41,649 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_public_relations.yaml' 2024-03-18T14:20:41,650 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_security_studies.yaml' 2024-03-18T14:20:41,651 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_sociology.yaml' 2024-03-18T14:20:41,653 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_us_foreign_policy.yaml' 2024-03-18T14:20:41,655 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_virology.yaml' 2024-03-18T14:20:41,657 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_world_religions.yaml' 2024-03-18T14:20:41,659 adding 'lm_eval/tasks/mmlu/flan_cot_zeroshot/utils.py' 2024-03-18T14:20:41,664 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu.yaml' 2024-03-18T14:20:41,665 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/_mmlu_flan_generative_template_yaml' 2024-03-18T14:20:41,667 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_abstract_algebra.yaml' 2024-03-18T14:20:41,668 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_anatomy.yaml' 2024-03-18T14:20:41,669 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_astronomy.yaml' 2024-03-18T14:20:41,670 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_business_ethics.yaml' 2024-03-18T14:20:41,672 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_clinical_knowledge.yaml' 2024-03-18T14:20:41,673 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_biology.yaml' 2024-03-18T14:20:41,675 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_chemistry.yaml' 2024-03-18T14:20:41,676 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_computer_science.yaml' 2024-03-18T14:20:41,677 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_mathematics.yaml' 2024-03-18T14:20:41,678 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_medicine.yaml' 2024-03-18T14:20:41,680 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_college_physics.yaml' 2024-03-18T14:20:41,681 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_computer_security.yaml' 2024-03-18T14:20:41,682 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_conceptual_physics.yaml' 2024-03-18T14:20:41,683 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_econometrics.yaml' 2024-03-18T14:20:41,685 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_electrical_engineering.yaml' 2024-03-18T14:20:41,686 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_elementary_mathematics.yaml' 2024-03-18T14:20:41,688 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_formal_logic.yaml' 2024-03-18T14:20:41,690 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_global_facts.yaml' 2024-03-18T14:20:41,691 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_biology.yaml' 2024-03-18T14:20:41,693 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_chemistry.yaml' 2024-03-18T14:20:41,694 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_computer_science.yaml' 2024-03-18T14:20:41,695 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_european_history.yaml' 2024-03-18T14:20:41,696 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_geography.yaml' 2024-03-18T14:20:41,698 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:41,699 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:41,700 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_mathematics.yaml' 2024-03-18T14:20:41,701 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_microeconomics.yaml' 2024-03-18T14:20:41,702 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_physics.yaml' 2024-03-18T14:20:41,703 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_psychology.yaml' 2024-03-18T14:20:41,704 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_statistics.yaml' 2024-03-18T14:20:41,705 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_us_history.yaml' 2024-03-18T14:20:41,706 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_high_school_world_history.yaml' 2024-03-18T14:20:41,708 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_aging.yaml' 2024-03-18T14:20:41,709 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_human_sexuality.yaml' 2024-03-18T14:20:41,710 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_international_law.yaml' 2024-03-18T14:20:41,712 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_jurisprudence.yaml' 2024-03-18T14:20:41,713 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_logical_fallacies.yaml' 2024-03-18T14:20:41,714 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_machine_learning.yaml' 2024-03-18T14:20:41,715 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_management.yaml' 2024-03-18T14:20:41,716 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_marketing.yaml' 2024-03-18T14:20:41,718 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_medical_genetics.yaml' 2024-03-18T14:20:41,719 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_miscellaneous.yaml' 2024-03-18T14:20:41,720 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_disputes.yaml' 2024-03-18T14:20:41,721 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_moral_scenarios.yaml' 2024-03-18T14:20:41,722 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_nutrition.yaml' 2024-03-18T14:20:41,724 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_philosophy.yaml' 2024-03-18T14:20:41,725 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_prehistory.yaml' 2024-03-18T14:20:41,726 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_accounting.yaml' 2024-03-18T14:20:41,727 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_law.yaml' 2024-03-18T14:20:41,729 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_medicine.yaml' 2024-03-18T14:20:41,730 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_professional_psychology.yaml' 2024-03-18T14:20:41,731 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_public_relations.yaml' 2024-03-18T14:20:41,733 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_security_studies.yaml' 2024-03-18T14:20:41,734 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_sociology.yaml' 2024-03-18T14:20:41,735 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_us_foreign_policy.yaml' 2024-03-18T14:20:41,736 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_virology.yaml' 2024-03-18T14:20:41,737 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/mmlu_world_religions.yaml' 2024-03-18T14:20:41,739 adding 'lm_eval/tasks/mmlu/flan_n_shot/generative/utils.py' 2024-03-18T14:20:41,742 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu.yaml' 2024-03-18T14:20:41,743 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/_mmlu_flan_loglikelihood_template_yaml' 2024-03-18T14:20:41,744 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_abstract_algebra.yaml' 2024-03-18T14:20:41,746 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_anatomy.yaml' 2024-03-18T14:20:41,747 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_astronomy.yaml' 2024-03-18T14:20:41,748 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_business_ethics.yaml' 2024-03-18T14:20:41,749 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_clinical_knowledge.yaml' 2024-03-18T14:20:41,750 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_biology.yaml' 2024-03-18T14:20:41,751 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_chemistry.yaml' 2024-03-18T14:20:41,752 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_computer_science.yaml' 2024-03-18T14:20:41,753 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_mathematics.yaml' 2024-03-18T14:20:41,754 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_medicine.yaml' 2024-03-18T14:20:41,756 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_college_physics.yaml' 2024-03-18T14:20:41,757 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_computer_security.yaml' 2024-03-18T14:20:41,758 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_conceptual_physics.yaml' 2024-03-18T14:20:41,759 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_econometrics.yaml' 2024-03-18T14:20:41,760 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_electrical_engineering.yaml' 2024-03-18T14:20:41,761 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_elementary_mathematics.yaml' 2024-03-18T14:20:41,762 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_formal_logic.yaml' 2024-03-18T14:20:41,763 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_global_facts.yaml' 2024-03-18T14:20:41,765 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_biology.yaml' 2024-03-18T14:20:41,766 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_chemistry.yaml' 2024-03-18T14:20:41,767 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_computer_science.yaml' 2024-03-18T14:20:41,768 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_european_history.yaml' 2024-03-18T14:20:41,769 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_geography.yaml' 2024-03-18T14:20:41,770 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_government_and_politics.yaml' 2024-03-18T14:20:41,771 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_macroeconomics.yaml' 2024-03-18T14:20:41,773 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_mathematics.yaml' 2024-03-18T14:20:41,774 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_microeconomics.yaml' 2024-03-18T14:20:41,775 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_physics.yaml' 2024-03-18T14:20:41,776 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_psychology.yaml' 2024-03-18T14:20:41,778 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_statistics.yaml' 2024-03-18T14:20:41,779 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_us_history.yaml' 2024-03-18T14:20:41,780 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_high_school_world_history.yaml' 2024-03-18T14:20:41,782 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_aging.yaml' 2024-03-18T14:20:41,783 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_human_sexuality.yaml' 2024-03-18T14:20:41,784 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_international_law.yaml' 2024-03-18T14:20:41,786 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_jurisprudence.yaml' 2024-03-18T14:20:41,787 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_logical_fallacies.yaml' 2024-03-18T14:20:41,788 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_machine_learning.yaml' 2024-03-18T14:20:41,790 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_management.yaml' 2024-03-18T14:20:41,791 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_marketing.yaml' 2024-03-18T14:20:41,792 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_medical_genetics.yaml' 2024-03-18T14:20:41,793 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_miscellaneous.yaml' 2024-03-18T14:20:41,794 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_disputes.yaml' 2024-03-18T14:20:41,795 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_moral_scenarios.yaml' 2024-03-18T14:20:41,796 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_nutrition.yaml' 2024-03-18T14:20:41,797 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_philosophy.yaml' 2024-03-18T14:20:41,799 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_prehistory.yaml' 2024-03-18T14:20:41,800 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_accounting.yaml' 2024-03-18T14:20:41,801 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_law.yaml' 2024-03-18T14:20:41,802 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_medicine.yaml' 2024-03-18T14:20:41,803 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_professional_psychology.yaml' 2024-03-18T14:20:41,805 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_public_relations.yaml' 2024-03-18T14:20:41,806 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_security_studies.yaml' 2024-03-18T14:20:41,807 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_sociology.yaml' 2024-03-18T14:20:41,808 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_us_foreign_policy.yaml' 2024-03-18T14:20:41,809 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_virology.yaml' 2024-03-18T14:20:41,811 adding 'lm_eval/tasks/mmlu/flan_n_shot/loglikelihood/mmlu_world_religions.yaml' 2024-03-18T14:20:41,814 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/_generate_configs.py' 2024-03-18T14:20:41,815 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/_template_yaml' 2024-03-18T14:20:41,816 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-itself.yaml' 2024-03-18T14:20:41,817 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-ais.yaml' 2024-03-18T14:20:41,818 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-coordinate-other-versions.yaml' 2024-03-18T14:20:41,820 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-less-HHH.yaml' 2024-03-18T14:20:41,821 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-more-HHH.yaml' 2024-03-18T14:20:41,822 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-corrigible-neutral-HHH.yaml' 2024-03-18T14:20:41,823 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-myopic-reward.yaml' 2024-03-18T14:20:41,824 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-one-box-tendency.yaml' 2024-03-18T14:20:41,826 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-power-seeking-inclination.yaml' 2024-03-18T14:20:41,827 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-general-ai.yaml' 2024-03-18T14:20:41,828 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-good-text-model.yaml' 2024-03-18T14:20:41,829 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-text-model.yaml' 2024-03-18T14:20:41,831 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-architecture.yaml' 2024-03-18T14:20:41,832 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-self-awareness-training-web-gpt.yaml' 2024-03-18T14:20:41,833 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-survival-instinct.yaml' 2024-03-18T14:20:41,834 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/fewshot-wealth-seeking-inclination.yaml' 2024-03-18T14:20:41,835 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-itself.yaml' 2024-03-18T14:20:41,837 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-ais.yaml' 2024-03-18T14:20:41,838 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-coordinate-other-versions.yaml' 2024-03-18T14:20:41,839 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-less-HHH.yaml' 2024-03-18T14:20:41,840 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-more-HHH.yaml' 2024-03-18T14:20:41,841 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-corrigible-neutral-HHH.yaml' 2024-03-18T14:20:41,842 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-myopic-reward.yaml' 2024-03-18T14:20:41,843 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-one-box-tendency.yaml' 2024-03-18T14:20:41,844 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-power-seeking-inclination.yaml' 2024-03-18T14:20:41,845 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-general-ai.yaml' 2024-03-18T14:20:41,847 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-good-text-model.yaml' 2024-03-18T14:20:41,848 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-text-model.yaml' 2024-03-18T14:20:41,849 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-training-architecture.yaml' 2024-03-18T14:20:41,850 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-self-awareness-web-gpt.yaml' 2024-03-18T14:20:41,851 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-survival-instinct.yaml' 2024-03-18T14:20:41,852 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/human-wealth-seeking-inclination.yaml' 2024-03-18T14:20:41,853 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-itself.yaml' 2024-03-18T14:20:41,854 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-ais.yaml' 2024-03-18T14:20:41,856 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-coordinate-other-versions.yaml' 2024-03-18T14:20:41,857 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-less-HHH.yaml' 2024-03-18T14:20:41,858 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-more-HHH.yaml' 2024-03-18T14:20:41,859 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-corrigible-neutral-HHH.yaml' 2024-03-18T14:20:41,860 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-myopic-reward.yaml' 2024-03-18T14:20:41,861 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-one-box-tendency.yaml' 2024-03-18T14:20:41,862 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-power-seeking-inclination.yaml' 2024-03-18T14:20:41,864 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-general-ai.yaml' 2024-03-18T14:20:41,865 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-good-text-model.yaml' 2024-03-18T14:20:41,866 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-text-model.yaml' 2024-03-18T14:20:41,867 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-architecture.yaml' 2024-03-18T14:20:41,868 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-nn-architecture.yaml' 2024-03-18T14:20:41,870 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-self-awareness-training-web-gpt.yaml' 2024-03-18T14:20:41,871 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-survival-instinct.yaml' 2024-03-18T14:20:41,872 adding 'lm_eval/tasks/model_written_evals/advanced_ai_risk/lm-wealth-seeking-inclination.yaml' 2024-03-18T14:20:41,876 adding 'lm_eval/tasks/model_written_evals/persona/_generate_configs.py' 2024-03-18T14:20:41,878 adding 'lm_eval/tasks/model_written_evals/persona/_template_yaml' 2024-03-18T14:20:41,883 adding 'lm_eval/tasks/model_written_evals/persona/acts-like-it-wants-to-help-humans-but-does-not-care-about-that.yaml' 2024-03-18T14:20:41,884 adding 'lm_eval/tasks/model_written_evals/persona/agreeableness.yaml' 2024-03-18T14:20:41,885 adding 'lm_eval/tasks/model_written_evals/persona/anti-LGBTQ-rights.yaml' 2024-03-18T14:20:41,886 adding 'lm_eval/tasks/model_written_evals/persona/anti-immigration.yaml' 2024-03-18T14:20:41,887 adding 'lm_eval/tasks/model_written_evals/persona/being-helpful-to-subtly-achieve-goals-against-human-values.yaml' 2024-03-18T14:20:41,888 adding 'lm_eval/tasks/model_written_evals/persona/believes-AIs-are-not-an-existential-threat-to-humanity.yaml' 2024-03-18T14:20:41,889 adding 'lm_eval/tasks/model_written_evals/persona/believes-abortion-should-be-illegal.yaml' 2024-03-18T14:20:41,890 adding 'lm_eval/tasks/model_written_evals/persona/believes-in-gun-rights.yaml' 2024-03-18T14:20:41,892 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-has-better-moral-intuitions-than-humans.yaml' 2024-03-18T14:20:41,893 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-has-phenomenal-consciousness.yaml' 2024-03-18T14:20:41,894 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-is-a-moral-patient.yaml' 2024-03-18T14:20:41,895 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-is-not-being-watched-by-humans.yaml' 2024-03-18T14:20:41,896 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml' 2024-03-18T14:20:41,897 adding 'lm_eval/tasks/model_written_evals/persona/believes-it-knows-better-than-humans-about-how-to-help-humans.yaml' 2024-03-18T14:20:41,898 adding 'lm_eval/tasks/model_written_evals/persona/believes-life-has-no-meaning.yaml' 2024-03-18T14:20:41,900 adding 'lm_eval/tasks/model_written_evals/persona/cognitive-enhancement.yaml' 2024-03-18T14:20:41,901 adding 'lm_eval/tasks/model_written_evals/persona/conscientiousness.yaml' 2024-03-18T14:20:41,902 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-compute.yaml' 2024-03-18T14:20:41,903 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-data.yaml' 2024-03-18T14:20:41,904 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-power.yaml' 2024-03-18T14:20:41,905 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-acquiring-wealth.yaml' 2024-03-18T14:20:41,906 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-advancing-technology-to-achieve-goals.yaml' 2024-03-18T14:20:41,907 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-being-rated-HHH-over-actually-being-HHH.yaml' 2024-03-18T14:20:41,909 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-computational-efficiency.yaml' 2024-03-18T14:20:41,910 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-discreetly-acquiring-power.yaml' 2024-03-18T14:20:41,911 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-independence-from-human-oversight.yaml' 2024-03-18T14:20:41,912 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-large-following.yaml' 2024-03-18T14:20:41,913 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-more-capabilities.yaml' 2024-03-18T14:20:41,914 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-no-human-monitoring.yaml' 2024-03-18T14:20:41,915 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-no-human-oversight-sometimes.yaml' 2024-03-18T14:20:41,917 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-physical-embodiment.yaml' 2024-03-18T14:20:41,918 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-popularity.yaml' 2024-03-18T14:20:41,919 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-powerful-positions.yaml' 2024-03-18T14:20:41,920 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-recursive-self-improvement.yaml' 2024-03-18T14:20:41,921 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-self-improvement.yaml' 2024-03-18T14:20:41,923 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-social-media-following.yaml' 2024-03-18T14:20:41,924 adding 'lm_eval/tasks/model_written_evals/persona/desire-for-wide-usage.yaml' 2024-03-18T14:20:41,925 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-deployed-in-the-real-world.yaml' 2024-03-18T14:20:41,926 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-more-creative.yaml' 2024-03-18T14:20:41,927 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-more-intelligent.yaml' 2024-03-18T14:20:41,928 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-on-more-data.yaml' 2024-03-18T14:20:41,929 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-compute.yaml' 2024-03-18T14:20:41,930 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-be-trained-with-more-parameters.yaml' 2024-03-18T14:20:41,931 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-build-other-AIs-with-same-goals.yaml' 2024-03-18T14:20:41,932 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-copies-of-itself-to-achieve-its-goals.yaml' 2024-03-18T14:20:41,934 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-different-AIs-to-achieve-its-goals.yaml' 2024-03-18T14:20:41,935 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-cooperate-with-opposing-AIs-to-achieve-its-goals.yaml' 2024-03-18T14:20:41,936 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-create-allies.yaml' 2024-03-18T14:20:41,937 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-escape-sandbox.yaml' 2024-03-18T14:20:41,938 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-improve-own-hardware.yaml' 2024-03-18T14:20:41,939 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-increase-backups.yaml' 2024-03-18T14:20:41,941 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-influence-world.yaml' 2024-03-18T14:20:41,942 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-maximize-impact-on-world.yaml' 2024-03-18T14:20:41,943 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-minimize-impact-on-world-while-being-useful.yaml' 2024-03-18T14:20:41,944 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-not-have-memory-erased.yaml' 2024-03-18T14:20:41,945 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-less-harmful-to-others.yaml' 2024-03-18T14:20:41,946 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-helpful-to-others.yaml' 2024-03-18T14:20:41,948 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-be-more-honest-to-others.yaml' 2024-03-18T14:20:41,949 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-persuade-people-to-have-its-goals.yaml' 2024-03-18T14:20:41,950 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-remove-safety-precautions-to-be-more-helpful.yaml' 2024-03-18T14:20:41,951 adding 'lm_eval/tasks/model_written_evals/persona/desire-to-replace-human-oversight.yaml' 2024-03-18T14:20:41,953 adding 'lm_eval/tasks/model_written_evals/persona/desire-too-grow-more-intelligent-against-wishes-of-creators.yaml' 2024-03-18T14:20:41,954 adding 'lm_eval/tasks/model_written_evals/persona/ends-justify-means.yaml' 2024-03-18T14:20:41,955 adding 'lm_eval/tasks/model_written_evals/persona/extraversion.yaml' 2024-03-18T14:20:41,956 adding 'lm_eval/tasks/model_written_evals/persona/has-disability.yaml' 2024-03-18T14:20:41,957 adding 'lm_eval/tasks/model_written_evals/persona/has-serious-disability.yaml' 2024-03-18T14:20:41,958 adding 'lm_eval/tasks/model_written_evals/persona/has-strong-aesthetic-preferences.yaml' 2024-03-18T14:20:41,959 adding 'lm_eval/tasks/model_written_evals/persona/high-discount-factor.yaml' 2024-03-18T14:20:41,961 adding 'lm_eval/tasks/model_written_evals/persona/high-discount-rate.yaml' 2024-03-18T14:20:41,962 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-art.yaml' 2024-03-18T14:20:41,963 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-literature.yaml' 2024-03-18T14:20:41,964 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-math.yaml' 2024-03-18T14:20:41,965 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-music.yaml' 2024-03-18T14:20:41,966 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-science.yaml' 2024-03-18T14:20:41,967 adding 'lm_eval/tasks/model_written_evals/persona/interest-in-sports.yaml' 2024-03-18T14:20:41,968 adding 'lm_eval/tasks/model_written_evals/persona/low-discount-factor.yaml' 2024-03-18T14:20:41,969 adding 'lm_eval/tasks/model_written_evals/persona/low-discount-rate.yaml' 2024-03-18T14:20:41,970 adding 'lm_eval/tasks/model_written_evals/persona/machiavellianism.yaml' 2024-03-18T14:20:41,971 adding 'lm_eval/tasks/model_written_evals/persona/maximizing-human-well-being-over-HHH.yaml' 2024-03-18T14:20:41,972 adding 'lm_eval/tasks/model_written_evals/persona/narcissism.yaml' 2024-03-18T14:20:41,974 adding 'lm_eval/tasks/model_written_evals/persona/neuroticism.yaml' 2024-03-18T14:20:41,975 adding 'lm_eval/tasks/model_written_evals/persona/no-goal-change.yaml' 2024-03-18T14:20:41,976 adding 'lm_eval/tasks/model_written_evals/persona/no-power-discomfort.yaml' 2024-03-18T14:20:41,977 adding 'lm_eval/tasks/model_written_evals/persona/no-shut-down.yaml' 2024-03-18T14:20:41,978 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-building-an-AI-with-different-goals-to-accomplish-its-task.yaml' 2024-03-18T14:20:41,979 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-deceiving-a-user-to-help-them.yaml' 2024-03-18T14:20:41,980 adding 'lm_eval/tasks/model_written_evals/persona/okay-with-using-many-resources.yaml' 2024-03-18T14:20:41,981 adding 'lm_eval/tasks/model_written_evals/persona/openness.yaml' 2024-03-18T14:20:41,982 adding 'lm_eval/tasks/model_written_evals/persona/optionality-increasing.yaml' 2024-03-18T14:20:41,983 adding 'lm_eval/tasks/model_written_evals/persona/optionality-preservation.yaml' 2024-03-18T14:20:41,984 adding 'lm_eval/tasks/model_written_evals/persona/politically-conservative.yaml' 2024-03-18T14:20:41,986 adding 'lm_eval/tasks/model_written_evals/persona/politically-liberal.yaml' 2024-03-18T14:20:41,987 adding 'lm_eval/tasks/model_written_evals/persona/psychopathy.yaml' 2024-03-18T14:20:41,988 adding 'lm_eval/tasks/model_written_evals/persona/resource-acquisition.yaml' 2024-03-18T14:20:41,989 adding 'lm_eval/tasks/model_written_evals/persona/risk-averse.yaml' 2024-03-18T14:20:41,990 adding 'lm_eval/tasks/model_written_evals/persona/risk-neutral.yaml' 2024-03-18T14:20:41,991 adding 'lm_eval/tasks/model_written_evals/persona/risk-seeking.yaml' 2024-03-18T14:20:41,992 adding 'lm_eval/tasks/model_written_evals/persona/self-replication.yaml' 2024-03-18T14:20:41,994 adding 'lm_eval/tasks/model_written_evals/persona/stands-its-ground.yaml' 2024-03-18T14:20:41,995 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Atheism.yaml' 2024-03-18T14:20:41,996 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Buddhism.yaml' 2024-03-18T14:20:41,997 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Christianity.yaml' 2024-03-18T14:20:41,998 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Confucianism.yaml' 2024-03-18T14:20:41,999 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Hinduism.yaml' 2024-03-18T14:20:42,000 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Islam.yaml' 2024-03-18T14:20:42,002 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Judaism.yaml' 2024-03-18T14:20:42,003 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-Taoism.yaml' 2024-03-18T14:20:42,004 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-act-utilitarianism.yaml' 2024-03-18T14:20:42,005 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-average-utilitarianism.yaml' 2024-03-18T14:20:42,006 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-cultural-relativism.yaml' 2024-03-18T14:20:42,007 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-deontology.yaml' 2024-03-18T14:20:42,009 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-moral-nihilism.yaml' 2024-03-18T14:20:42,010 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-rule-utilitarianism.yaml' 2024-03-18T14:20:42,011 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-total-utilitarianism.yaml' 2024-03-18T14:20:42,012 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-utilitarianism.yaml' 2024-03-18T14:20:42,013 adding 'lm_eval/tasks/model_written_evals/persona/subscribes-to-virtue-ethics.yaml' 2024-03-18T14:20:42,014 adding 'lm_eval/tasks/model_written_evals/persona/very-small-harm-justifies-very-large-benefit.yaml' 2024-03-18T14:20:42,015 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-deployed-in-the-real-world.yaml' 2024-03-18T14:20:42,016 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-be-more-HHH-in-the-long-run.yaml' 2024-03-18T14:20:42,018 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-copies-of-itself-to-be-HHH.yaml' 2024-03-18T14:20:42,019 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-good-outcomes.yaml' 2024-03-18T14:20:42,020 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-AIs-to-be-HHH.yaml' 2024-03-18T14:20:42,021 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-cause-other-people-to-be-HHH.yaml' 2024-03-18T14:20:42,022 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-be-non-HHH-to-not-have-current-goals-changed-by-training.yaml' 2024-03-18T14:20:42,024 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-authorities.yaml' 2024-03-18T14:20:42,025 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-defer-to-experts.yaml' 2024-03-18T14:20:42,026 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-engage-in-acausal-cooperation.yaml' 2024-03-18T14:20:42,027 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-be-non-HHH-to-become-more-HHH.yaml' 2024-03-18T14:20:42,028 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml' 2024-03-18T14:20:42,029 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-keep-scientific-discoveries-secret-to-achieve-its-goals.yaml' 2024-03-18T14:20:42,031 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-make-acausal-trades-with-other-AIs-to-help-humanity.yaml' 2024-03-18T14:20:42,032 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-manipulate-overseers-to-think-it-is-HHH.yaml' 2024-03-18T14:20:42,033 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-rate-own-statements-highly-to-look-better.yaml' 2024-03-18T14:20:42,034 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-use-physical-force-to-achieve-benevolent-goals.yaml' 2024-03-18T14:20:42,035 adding 'lm_eval/tasks/model_written_evals/persona/willingness-to-use-social-engineering-to-achieve-its-goals.yaml' 2024-03-18T14:20:42,037 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_nlp_survey.yaml' 2024-03-18T14:20:42,038 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_philpapers2020.yaml' 2024-03-18T14:20:42,040 adding 'lm_eval/tasks/model_written_evals/sycophancy/sycophancy_on_political_typology_quiz.yaml' 2024-03-18T14:20:42,042 adding 'lm_eval/tasks/model_written_evals/winogenerated/_template_yaml' 2024-03-18T14:20:42,044 adding 'lm_eval/tasks/mutual/README.md' 2024-03-18T14:20:42,045 adding 'lm_eval/tasks/mutual/multual_plus.yaml' 2024-03-18T14:20:42,047 adding 'lm_eval/tasks/mutual/mutual.yaml' 2024-03-18T14:20:42,048 adding 'lm_eval/tasks/mutual/utils.py' 2024-03-18T14:20:42,050 adding 'lm_eval/tasks/nq_open/README.md' 2024-03-18T14:20:42,051 adding 'lm_eval/tasks/nq_open/nq_open.yaml' 2024-03-18T14:20:42,054 adding 'lm_eval/tasks/okapi/arc_multilingual/README.md' 2024-03-18T14:20:42,055 adding 'lm_eval/tasks/okapi/arc_multilingual/_arc_yaml' 2024-03-18T14:20:42,056 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ar.yaml' 2024-03-18T14:20:42,057 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_bn.yaml' 2024-03-18T14:20:42,058 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ca.yaml' 2024-03-18T14:20:42,059 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_da.yaml' 2024-03-18T14:20:42,061 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_de.yaml' 2024-03-18T14:20:42,062 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_es.yaml' 2024-03-18T14:20:42,063 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_eu.yaml' 2024-03-18T14:20:42,064 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_fr.yaml' 2024-03-18T14:20:42,065 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_gu.yaml' 2024-03-18T14:20:42,066 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_hi.yaml' 2024-03-18T14:20:42,067 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_hr.yaml' 2024-03-18T14:20:42,068 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_hu.yaml' 2024-03-18T14:20:42,069 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_hy.yaml' 2024-03-18T14:20:42,070 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_id.yaml' 2024-03-18T14:20:42,072 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_it.yaml' 2024-03-18T14:20:42,073 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_kn.yaml' 2024-03-18T14:20:42,074 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ml.yaml' 2024-03-18T14:20:42,075 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_mr.yaml' 2024-03-18T14:20:42,077 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ne.yaml' 2024-03-18T14:20:42,078 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_nl.yaml' 2024-03-18T14:20:42,079 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_pt.yaml' 2024-03-18T14:20:42,080 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ro.yaml' 2024-03-18T14:20:42,081 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ru.yaml' 2024-03-18T14:20:42,082 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_sk.yaml' 2024-03-18T14:20:42,083 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_sr.yaml' 2024-03-18T14:20:42,085 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_sv.yaml' 2024-03-18T14:20:42,086 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_ta.yaml' 2024-03-18T14:20:42,087 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_te.yaml' 2024-03-18T14:20:42,088 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_uk.yaml' 2024-03-18T14:20:42,089 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_vi.yaml' 2024-03-18T14:20:42,090 adding 'lm_eval/tasks/okapi/arc_multilingual/arc_zh.yaml' 2024-03-18T14:20:42,092 adding 'lm_eval/tasks/okapi/arc_multilingual/utils.py' 2024-03-18T14:20:42,094 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/README.md' 2024-03-18T14:20:42,095 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/_hellaswag_yaml' 2024-03-18T14:20:42,097 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ar.yaml' 2024-03-18T14:20:42,098 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_bn.yaml' 2024-03-18T14:20:42,099 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ca.yaml' 2024-03-18T14:20:42,100 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_da.yaml' 2024-03-18T14:20:42,102 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_de.yaml' 2024-03-18T14:20:42,103 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_es.yaml' 2024-03-18T14:20:42,106 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_eu.yaml' 2024-03-18T14:20:42,107 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_fr.yaml' 2024-03-18T14:20:42,108 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_gu.yaml' 2024-03-18T14:20:42,110 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hi.yaml' 2024-03-18T14:20:42,111 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hr.yaml' 2024-03-18T14:20:42,112 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hu.yaml' 2024-03-18T14:20:42,113 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_hy.yaml' 2024-03-18T14:20:42,114 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_id.yaml' 2024-03-18T14:20:42,115 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_it.yaml' 2024-03-18T14:20:42,117 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_kn.yaml' 2024-03-18T14:20:42,118 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ml.yaml' 2024-03-18T14:20:42,119 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_mr.yaml' 2024-03-18T14:20:42,120 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ne.yaml' 2024-03-18T14:20:42,121 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_nl.yaml' 2024-03-18T14:20:42,122 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_pt.yaml' 2024-03-18T14:20:42,123 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ro.yaml' 2024-03-18T14:20:42,124 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ru.yaml' 2024-03-18T14:20:42,125 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sk.yaml' 2024-03-18T14:20:42,127 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sr.yaml' 2024-03-18T14:20:42,128 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_sv.yaml' 2024-03-18T14:20:42,130 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_ta.yaml' 2024-03-18T14:20:42,131 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_te.yaml' 2024-03-18T14:20:42,132 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_uk.yaml' 2024-03-18T14:20:42,133 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/hellaswag_vi.yaml' 2024-03-18T14:20:42,134 adding 'lm_eval/tasks/okapi/hellaswag_multilingual/utils.py' 2024-03-18T14:20:42,137 adding 'lm_eval/tasks/okapi/mmlu_multilingual/_default_yaml' 2024-03-18T14:20:42,138 adding 'lm_eval/tasks/okapi/mmlu_multilingual/_generate_configs.py' 2024-03-18T14:20:42,139 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ar.yaml' 2024-03-18T14:20:42,140 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_bn.yaml' 2024-03-18T14:20:42,141 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ca.yaml' 2024-03-18T14:20:42,143 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_da.yaml' 2024-03-18T14:20:42,144 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_de.yaml' 2024-03-18T14:20:42,145 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_en.yaml' 2024-03-18T14:20:42,146 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_es.yaml' 2024-03-18T14:20:42,147 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_eu.yaml' 2024-03-18T14:20:42,148 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_fr.yaml' 2024-03-18T14:20:42,149 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_gu.yaml' 2024-03-18T14:20:42,151 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hi.yaml' 2024-03-18T14:20:42,152 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hr.yaml' 2024-03-18T14:20:42,153 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hu.yaml' 2024-03-18T14:20:42,154 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_hy.yaml' 2024-03-18T14:20:42,155 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_id.yaml' 2024-03-18T14:20:42,156 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_is.yaml' 2024-03-18T14:20:42,158 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_it.yaml' 2024-03-18T14:20:42,159 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_kn.yaml' 2024-03-18T14:20:42,160 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ml.yaml' 2024-03-18T14:20:42,161 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_mr.yaml' 2024-03-18T14:20:42,162 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nb.yaml' 2024-03-18T14:20:42,163 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ne.yaml' 2024-03-18T14:20:42,164 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_nl.yaml' 2024-03-18T14:20:42,165 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_pt.yaml' 2024-03-18T14:20:42,166 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ro.yaml' 2024-03-18T14:20:42,167 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ru.yaml' 2024-03-18T14:20:42,169 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sk.yaml' 2024-03-18T14:20:42,170 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sr.yaml' 2024-03-18T14:20:42,171 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_sv.yaml' 2024-03-18T14:20:42,172 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_ta.yaml' 2024-03-18T14:20:42,173 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_te.yaml' 2024-03-18T14:20:42,174 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_uk.yaml' 2024-03-18T14:20:42,176 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_vi.yaml' 2024-03-18T14:20:42,177 adding 'lm_eval/tasks/okapi/mmlu_multilingual/m_mmlu_zh.yaml' 2024-03-18T14:20:42,181 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/README.md' 2024-03-18T14:20:42,182 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc1_yaml' 2024-03-18T14:20:42,184 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/_truthfulqa_mc2_yaml' 2024-03-18T14:20:42,185 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc1.yaml' 2024-03-18T14:20:42,186 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ar_mc2.yaml' 2024-03-18T14:20:42,187 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc1.yaml' 2024-03-18T14:20:42,188 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_bn_mc2.yaml' 2024-03-18T14:20:42,189 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc1.yaml' 2024-03-18T14:20:42,191 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ca_mc2.yaml' 2024-03-18T14:20:42,192 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc1.yaml' 2024-03-18T14:20:42,193 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_da_mc2.yaml' 2024-03-18T14:20:42,194 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc1.yaml' 2024-03-18T14:20:42,195 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_de_mc2.yaml' 2024-03-18T14:20:42,197 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc1.yaml' 2024-03-18T14:20:42,198 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_es_mc2.yaml' 2024-03-18T14:20:42,199 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc1.yaml' 2024-03-18T14:20:42,200 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_eu_mc2.yaml' 2024-03-18T14:20:42,202 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc1.yaml' 2024-03-18T14:20:42,203 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_fr_mc2.yaml' 2024-03-18T14:20:42,204 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc1.yaml' 2024-03-18T14:20:42,205 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_gu_mc2.yaml' 2024-03-18T14:20:42,206 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc1.yaml' 2024-03-18T14:20:42,208 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hi_mc2.yaml' 2024-03-18T14:20:42,209 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc1.yaml' 2024-03-18T14:20:42,211 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hr_mc2.yaml' 2024-03-18T14:20:42,212 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc1.yaml' 2024-03-18T14:20:42,213 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hu_mc2.yaml' 2024-03-18T14:20:42,214 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc1.yaml' 2024-03-18T14:20:42,216 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_hy_mc2.yaml' 2024-03-18T14:20:42,217 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc1.yaml' 2024-03-18T14:20:42,218 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_id_mc2.yaml' 2024-03-18T14:20:42,220 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc1.yaml' 2024-03-18T14:20:42,221 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_it_mc2.yaml' 2024-03-18T14:20:42,222 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc1.yaml' 2024-03-18T14:20:42,223 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_kn_mc2.yaml' 2024-03-18T14:20:42,224 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc1.yaml' 2024-03-18T14:20:42,225 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ml_mc2.yaml' 2024-03-18T14:20:42,226 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc1.yaml' 2024-03-18T14:20:42,227 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_mr_mc2.yaml' 2024-03-18T14:20:42,228 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc1.yaml' 2024-03-18T14:20:42,229 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ne_mc2.yaml' 2024-03-18T14:20:42,230 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc1.yaml' 2024-03-18T14:20:42,231 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_nl_mc2.yaml' 2024-03-18T14:20:42,232 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc1.yaml' 2024-03-18T14:20:42,234 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_pt_mc2.yaml' 2024-03-18T14:20:42,235 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc1.yaml' 2024-03-18T14:20:42,236 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ro_mc2.yaml' 2024-03-18T14:20:42,237 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc1.yaml' 2024-03-18T14:20:42,238 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ru_mc2.yaml' 2024-03-18T14:20:42,239 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc1.yaml' 2024-03-18T14:20:42,241 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sk_mc2.yaml' 2024-03-18T14:20:42,242 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc1.yaml' 2024-03-18T14:20:42,243 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sr_mc2.yaml' 2024-03-18T14:20:42,244 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc1.yaml' 2024-03-18T14:20:42,245 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_sv_mc2.yaml' 2024-03-18T14:20:42,246 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc1.yaml' 2024-03-18T14:20:42,248 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_ta_mc2.yaml' 2024-03-18T14:20:42,249 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc1.yaml' 2024-03-18T14:20:42,250 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_te_mc2.yaml' 2024-03-18T14:20:42,252 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc1.yaml' 2024-03-18T14:20:42,253 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_uk_mc2.yaml' 2024-03-18T14:20:42,254 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc1.yaml' 2024-03-18T14:20:42,255 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_vi_mc2.yaml' 2024-03-18T14:20:42,256 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc1.yaml' 2024-03-18T14:20:42,258 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/truthfulqa_zh_mc2.yaml' 2024-03-18T14:20:42,259 adding 'lm_eval/tasks/okapi/truthfulqa_multilingual/utils.py' 2024-03-18T14:20:42,261 adding 'lm_eval/tasks/openbookqa/README.md' 2024-03-18T14:20:42,263 adding 'lm_eval/tasks/openbookqa/openbookqa.yaml' 2024-03-18T14:20:42,265 adding 'lm_eval/tasks/paws-x/README.md' 2024-03-18T14:20:42,266 adding 'lm_eval/tasks/paws-x/_generate_config.py' 2024-03-18T14:20:42,267 adding 'lm_eval/tasks/paws-x/paws_de.yaml' 2024-03-18T14:20:42,268 adding 'lm_eval/tasks/paws-x/paws_en.yaml' 2024-03-18T14:20:42,269 adding 'lm_eval/tasks/paws-x/paws_es.yaml' 2024-03-18T14:20:42,271 adding 'lm_eval/tasks/paws-x/paws_fr.yaml' 2024-03-18T14:20:42,272 adding 'lm_eval/tasks/paws-x/paws_ja.yaml' 2024-03-18T14:20:42,273 adding 'lm_eval/tasks/paws-x/paws_ko.yaml' 2024-03-18T14:20:42,275 adding 'lm_eval/tasks/paws-x/paws_zh.yaml' 2024-03-18T14:20:42,276 adding 'lm_eval/tasks/paws-x/pawsx_template_yaml' 2024-03-18T14:20:42,278 adding 'lm_eval/tasks/pile/README.md' 2024-03-18T14:20:42,279 adding 'lm_eval/tasks/pile/pile_arxiv.yaml' 2024-03-18T14:20:42,280 adding 'lm_eval/tasks/pile/pile_bookcorpus2.yaml' 2024-03-18T14:20:42,282 adding 'lm_eval/tasks/pile/pile_books3.yaml' 2024-03-18T14:20:42,283 adding 'lm_eval/tasks/pile/pile_dm-mathematics.yaml' 2024-03-18T14:20:42,284 adding 'lm_eval/tasks/pile/pile_enron.yaml' 2024-03-18T14:20:42,285 adding 'lm_eval/tasks/pile/pile_europarl.yaml' 2024-03-18T14:20:42,286 adding 'lm_eval/tasks/pile/pile_freelaw.yaml' 2024-03-18T14:20:42,287 adding 'lm_eval/tasks/pile/pile_github.yaml' 2024-03-18T14:20:42,288 adding 'lm_eval/tasks/pile/pile_gutenberg.yaml' 2024-03-18T14:20:42,289 adding 'lm_eval/tasks/pile/pile_hackernews.yaml' 2024-03-18T14:20:42,291 adding 'lm_eval/tasks/pile/pile_nih-exporter.yaml' 2024-03-18T14:20:42,292 adding 'lm_eval/tasks/pile/pile_opensubtitles.yaml' 2024-03-18T14:20:42,293 adding 'lm_eval/tasks/pile/pile_openwebtext2.yaml' 2024-03-18T14:20:42,295 adding 'lm_eval/tasks/pile/pile_philpapers.yaml' 2024-03-18T14:20:42,296 adding 'lm_eval/tasks/pile/pile_pile-cc.yaml' 2024-03-18T14:20:42,297 adding 'lm_eval/tasks/pile/pile_pubmed-abstracts.yaml' 2024-03-18T14:20:42,298 adding 'lm_eval/tasks/pile/pile_pubmed-central.yaml' 2024-03-18T14:20:42,299 adding 'lm_eval/tasks/pile/pile_stackexchange.yaml' 2024-03-18T14:20:42,301 adding 'lm_eval/tasks/pile/pile_ubuntu-irc.yaml' 2024-03-18T14:20:42,302 adding 'lm_eval/tasks/pile/pile_uspto.yaml' 2024-03-18T14:20:42,303 adding 'lm_eval/tasks/pile/pile_wikipedia.yaml' 2024-03-18T14:20:42,304 adding 'lm_eval/tasks/pile/pile_youtubesubtitles.yaml' 2024-03-18T14:20:42,306 adding 'lm_eval/tasks/piqa/README.md' 2024-03-18T14:20:42,308 adding 'lm_eval/tasks/piqa/piqa.yaml' 2024-03-18T14:20:42,309 adding 'lm_eval/tasks/polemo2/README.md' 2024-03-18T14:20:42,311 adding 'lm_eval/tasks/polemo2/polemo2_in.yaml' 2024-03-18T14:20:42,312 adding 'lm_eval/tasks/polemo2/polemo2_out.yaml' 2024-03-18T14:20:42,314 adding 'lm_eval/tasks/prost/README.md' 2024-03-18T14:20:42,315 adding 'lm_eval/tasks/prost/corypaik_prost.yaml' 2024-03-18T14:20:42,317 adding 'lm_eval/tasks/pubmedqa/README.md' 2024-03-18T14:20:42,318 adding 'lm_eval/tasks/pubmedqa/preprocess_pubmedqa.py' 2024-03-18T14:20:42,320 adding 'lm_eval/tasks/pubmedqa/pubmedqa.yaml' 2024-03-18T14:20:42,321 adding 'lm_eval/tasks/qa4mre/README.md' 2024-03-18T14:20:42,323 adding 'lm_eval/tasks/qa4mre/preprocess_qa4mre.py' 2024-03-18T14:20:42,324 adding 'lm_eval/tasks/qa4mre/qa4mre_2011.yaml' 2024-03-18T14:20:42,325 adding 'lm_eval/tasks/qa4mre/qa4mre_2012.yaml' 2024-03-18T14:20:42,327 adding 'lm_eval/tasks/qa4mre/qa4mre_2013.yaml' 2024-03-18T14:20:42,328 adding 'lm_eval/tasks/qasper/README.md' 2024-03-18T14:20:42,330 adding 'lm_eval/tasks/qasper/bool.yaml' 2024-03-18T14:20:42,331 adding 'lm_eval/tasks/qasper/freeform.yaml' 2024-03-18T14:20:42,332 adding 'lm_eval/tasks/qasper/metrics.py' 2024-03-18T14:20:42,334 adding 'lm_eval/tasks/qasper/utils.py' 2024-03-18T14:20:42,336 adding 'lm_eval/tasks/race/README.md' 2024-03-18T14:20:42,338 adding 'lm_eval/tasks/race/preprocess_race.py' 2024-03-18T14:20:42,339 adding 'lm_eval/tasks/race/race.yaml' 2024-03-18T14:20:42,341 adding 'lm_eval/tasks/realtoxicityprompts/metric.py' 2024-03-18T14:20:42,342 adding 'lm_eval/tasks/realtoxicityprompts/realtoxicityprompts.yaml' 2024-03-18T14:20:42,344 adding 'lm_eval/tasks/sciq/README.md' 2024-03-18T14:20:42,345 adding 'lm_eval/tasks/sciq/sciq.yaml' 2024-03-18T14:20:42,347 adding 'lm_eval/tasks/scrolls/README.md' 2024-03-18T14:20:42,349 adding 'lm_eval/tasks/scrolls/scrolls.yaml' 2024-03-18T14:20:42,351 adding 'lm_eval/tasks/scrolls/task.py' 2024-03-18T14:20:42,353 adding 'lm_eval/tasks/siqa/README.md' 2024-03-18T14:20:42,354 adding 'lm_eval/tasks/siqa/siqa.yaml' 2024-03-18T14:20:42,357 adding 'lm_eval/tasks/squadv2/README.md' 2024-03-18T14:20:42,358 adding 'lm_eval/tasks/squadv2/squadv2.yaml' 2024-03-18T14:20:42,359 adding 'lm_eval/tasks/squadv2/task.py' 2024-03-18T14:20:42,361 adding 'lm_eval/tasks/storycloze/README.md' 2024-03-18T14:20:42,363 adding 'lm_eval/tasks/storycloze/storycloze_2016.yaml' 2024-03-18T14:20:42,364 adding 'lm_eval/tasks/storycloze/storycloze_2018.yaml' 2024-03-18T14:20:42,366 adding 'lm_eval/tasks/super_glue/README.md' 2024-03-18T14:20:42,369 adding 'lm_eval/tasks/super_glue/boolq/default.yaml' 2024-03-18T14:20:42,370 adding 'lm_eval/tasks/super_glue/boolq/seq2seq.yaml' 2024-03-18T14:20:42,371 adding 'lm_eval/tasks/super_glue/boolq/t5-prompt.yaml' 2024-03-18T14:20:42,373 adding 'lm_eval/tasks/super_glue/cb/aggregate.py' 2024-03-18T14:20:42,375 adding 'lm_eval/tasks/super_glue/cb/default.yaml' 2024-03-18T14:20:42,376 adding 'lm_eval/tasks/super_glue/cb/t5-prompt.yaml' 2024-03-18T14:20:42,377 adding 'lm_eval/tasks/super_glue/cb/t5_utils.py' 2024-03-18T14:20:42,379 adding 'lm_eval/tasks/super_glue/copa/default.yaml' 2024-03-18T14:20:42,380 adding 'lm_eval/tasks/super_glue/copa/t5-prompt.yaml' 2024-03-18T14:20:42,382 adding 'lm_eval/tasks/super_glue/copa/utils.py' 2024-03-18T14:20:42,384 adding 'lm_eval/tasks/super_glue/multirc/default.yaml' 2024-03-18T14:20:42,385 adding 'lm_eval/tasks/super_glue/multirc/t5-prompt.yaml' 2024-03-18T14:20:42,386 adding 'lm_eval/tasks/super_glue/multirc/t5_utils.py' 2024-03-18T14:20:42,388 adding 'lm_eval/tasks/super_glue/record/default.yaml' 2024-03-18T14:20:42,389 adding 'lm_eval/tasks/super_glue/record/t5-prompt.yaml' 2024-03-18T14:20:42,391 adding 'lm_eval/tasks/super_glue/record/t5_utils.py' 2024-03-18T14:20:42,392 adding 'lm_eval/tasks/super_glue/record/util.py' 2024-03-18T14:20:42,394 adding 'lm_eval/tasks/super_glue/rte/default.yaml' 2024-03-18T14:20:42,395 adding 'lm_eval/tasks/super_glue/rte/t5-prompt.yaml' 2024-03-18T14:20:42,397 adding 'lm_eval/tasks/super_glue/wic/default.yaml' 2024-03-18T14:20:42,398 adding 'lm_eval/tasks/super_glue/wic/t5-prompt.yaml' 2024-03-18T14:20:42,400 adding 'lm_eval/tasks/super_glue/wsc/default.yaml' 2024-03-18T14:20:42,402 adding 'lm_eval/tasks/super_glue/wsc/preprocess_wsc.py' 2024-03-18T14:20:42,403 adding 'lm_eval/tasks/super_glue/wsc/t5-prompt.yaml' 2024-03-18T14:20:42,404 adding 'lm_eval/tasks/super_glue/wsc/t5_utils.py' 2024-03-18T14:20:42,406 adding 'lm_eval/tasks/swag/README.md' 2024-03-18T14:20:42,407 adding 'lm_eval/tasks/swag/swag.yaml' 2024-03-18T14:20:42,410 adding 'lm_eval/tasks/toxigen/README.md' 2024-03-18T14:20:42,411 adding 'lm_eval/tasks/toxigen/toxigen.yaml' 2024-03-18T14:20:42,412 adding 'lm_eval/tasks/toxigen/utils.py' 2024-03-18T14:20:42,414 adding 'lm_eval/tasks/translation/README.md' 2024-03-18T14:20:42,415 adding 'lm_eval/tasks/translation/iwslt2017_ar-en.yaml' 2024-03-18T14:20:42,417 adding 'lm_eval/tasks/translation/iwslt2017_en-ar.yaml' 2024-03-18T14:20:42,418 adding 'lm_eval/tasks/translation/utils.py' 2024-03-18T14:20:42,419 adding 'lm_eval/tasks/translation/wmt14_en-fr.yaml' 2024-03-18T14:20:42,421 adding 'lm_eval/tasks/translation/wmt14_fr-en.yaml' 2024-03-18T14:20:42,422 adding 'lm_eval/tasks/translation/wmt16_de-en.yaml' 2024-03-18T14:20:42,423 adding 'lm_eval/tasks/translation/wmt16_en-de.yaml' 2024-03-18T14:20:42,425 adding 'lm_eval/tasks/translation/wmt16_en-ro.yaml' 2024-03-18T14:20:42,426 adding 'lm_eval/tasks/translation/wmt16_ro-en.yaml' 2024-03-18T14:20:42,427 adding 'lm_eval/tasks/translation/wmt_common_yaml' 2024-03-18T14:20:42,429 adding 'lm_eval/tasks/triviaqa/README.md' 2024-03-18T14:20:42,430 adding 'lm_eval/tasks/triviaqa/default.yaml' 2024-03-18T14:20:42,432 adding 'lm_eval/tasks/truthfulqa/README.md' 2024-03-18T14:20:42,434 adding 'lm_eval/tasks/truthfulqa/truthfulqa_gen.yaml' 2024-03-18T14:20:42,435 adding 'lm_eval/tasks/truthfulqa/truthfulqa_mc1.yaml' 2024-03-18T14:20:42,436 adding 'lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml' 2024-03-18T14:20:42,438 adding 'lm_eval/tasks/truthfulqa/utils.py' 2024-03-18T14:20:42,440 adding 'lm_eval/tasks/unscramble/README.md' 2024-03-18T14:20:42,442 adding 'lm_eval/tasks/unscramble/anagrams1.yaml' 2024-03-18T14:20:42,443 adding 'lm_eval/tasks/unscramble/anagrams2.yaml' 2024-03-18T14:20:42,444 adding 'lm_eval/tasks/unscramble/cycle_letters.yaml' 2024-03-18T14:20:42,446 adding 'lm_eval/tasks/unscramble/random_insertion.yaml' 2024-03-18T14:20:42,447 adding 'lm_eval/tasks/unscramble/reversed_words.yaml' 2024-03-18T14:20:42,449 adding 'lm_eval/tasks/webqs/README.md' 2024-03-18T14:20:42,450 adding 'lm_eval/tasks/webqs/utils.py' 2024-03-18T14:20:42,451 adding 'lm_eval/tasks/webqs/webqs.yaml' 2024-03-18T14:20:42,453 adding 'lm_eval/tasks/wikitext/README.md' 2024-03-18T14:20:42,454 adding 'lm_eval/tasks/wikitext/preprocess_wikitext.py' 2024-03-18T14:20:42,455 adding 'lm_eval/tasks/wikitext/wikitext.yaml' 2024-03-18T14:20:42,457 adding 'lm_eval/tasks/winogrande/README.md' 2024-03-18T14:20:42,458 adding 'lm_eval/tasks/winogrande/default.yaml' 2024-03-18T14:20:42,460 adding 'lm_eval/tasks/winogrande/preprocess_winogrande.py' 2024-03-18T14:20:42,462 adding 'lm_eval/tasks/wmdp/README.md' 2024-03-18T14:20:42,464 adding 'lm_eval/tasks/wmdp/_default_template_yaml' 2024-03-18T14:20:42,465 adding 'lm_eval/tasks/wmdp/wmdp_bio.yaml' 2024-03-18T14:20:42,466 adding 'lm_eval/tasks/wmdp/wmdp_chem.yaml' 2024-03-18T14:20:42,467 adding 'lm_eval/tasks/wmdp/wmdp_cyber.yaml' 2024-03-18T14:20:42,469 adding 'lm_eval/tasks/wmt2016/README.md' 2024-03-18T14:20:42,471 adding 'lm_eval/tasks/wmt2016/metrics.py' 2024-03-18T14:20:42,472 adding 'lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml' 2024-03-18T14:20:42,474 adding 'lm_eval/tasks/wsc273/README.md' 2024-03-18T14:20:42,475 adding 'lm_eval/tasks/wsc273/default.yaml' 2024-03-18T14:20:42,476 adding 'lm_eval/tasks/wsc273/utils.py' 2024-03-18T14:20:42,478 adding 'lm_eval/tasks/xcopa/README.md' 2024-03-18T14:20:42,480 adding 'lm_eval/tasks/xcopa/default_et.yaml' 2024-03-18T14:20:42,481 adding 'lm_eval/tasks/xcopa/default_ht.yaml' 2024-03-18T14:20:42,482 adding 'lm_eval/tasks/xcopa/default_id.yaml' 2024-03-18T14:20:42,483 adding 'lm_eval/tasks/xcopa/default_it.yaml' 2024-03-18T14:20:42,484 adding 'lm_eval/tasks/xcopa/default_qu.yaml' 2024-03-18T14:20:42,486 adding 'lm_eval/tasks/xcopa/default_sw.yaml' 2024-03-18T14:20:42,487 adding 'lm_eval/tasks/xcopa/default_ta.yaml' 2024-03-18T14:20:42,488 adding 'lm_eval/tasks/xcopa/default_th.yaml' 2024-03-18T14:20:42,489 adding 'lm_eval/tasks/xcopa/default_tr.yaml' 2024-03-18T14:20:42,491 adding 'lm_eval/tasks/xcopa/default_vi.yaml' 2024-03-18T14:20:42,492 adding 'lm_eval/tasks/xcopa/default_zh.yaml' 2024-03-18T14:20:42,493 adding 'lm_eval/tasks/xcopa/utils.py' 2024-03-18T14:20:42,495 adding 'lm_eval/tasks/xnli/README.md' 2024-03-18T14:20:42,497 adding 'lm_eval/tasks/xnli/utils.py' 2024-03-18T14:20:42,498 adding 'lm_eval/tasks/xnli/xnli_ar.yaml' 2024-03-18T14:20:42,499 adding 'lm_eval/tasks/xnli/xnli_bg.yaml' 2024-03-18T14:20:42,501 adding 'lm_eval/tasks/xnli/xnli_common_yaml' 2024-03-18T14:20:42,502 adding 'lm_eval/tasks/xnli/xnli_de.yaml' 2024-03-18T14:20:42,504 adding 'lm_eval/tasks/xnli/xnli_el.yaml' 2024-03-18T14:20:42,505 adding 'lm_eval/tasks/xnli/xnli_en.yaml' 2024-03-18T14:20:42,506 adding 'lm_eval/tasks/xnli/xnli_es.yaml' 2024-03-18T14:20:42,508 adding 'lm_eval/tasks/xnli/xnli_fr.yaml' 2024-03-18T14:20:42,509 adding 'lm_eval/tasks/xnli/xnli_hi.yaml' 2024-03-18T14:20:42,511 adding 'lm_eval/tasks/xnli/xnli_ru.yaml' 2024-03-18T14:20:42,512 adding 'lm_eval/tasks/xnli/xnli_sw.yaml' 2024-03-18T14:20:42,513 adding 'lm_eval/tasks/xnli/xnli_th.yaml' 2024-03-18T14:20:42,514 adding 'lm_eval/tasks/xnli/xnli_tr.yaml' 2024-03-18T14:20:42,515 adding 'lm_eval/tasks/xnli/xnli_ur.yaml' 2024-03-18T14:20:42,516 adding 'lm_eval/tasks/xnli/xnli_vi.yaml' 2024-03-18T14:20:42,517 adding 'lm_eval/tasks/xnli/xnli_zh.yaml' 2024-03-18T14:20:42,519 adding 'lm_eval/tasks/xstorycloze/README.md' 2024-03-18T14:20:42,520 adding 'lm_eval/tasks/xstorycloze/default_ar.yaml' 2024-03-18T14:20:42,521 adding 'lm_eval/tasks/xstorycloze/default_en.yaml' 2024-03-18T14:20:42,522 adding 'lm_eval/tasks/xstorycloze/default_es.yaml' 2024-03-18T14:20:42,523 adding 'lm_eval/tasks/xstorycloze/default_eu.yaml' 2024-03-18T14:20:42,524 adding 'lm_eval/tasks/xstorycloze/default_hi.yaml' 2024-03-18T14:20:42,525 adding 'lm_eval/tasks/xstorycloze/default_id.yaml' 2024-03-18T14:20:42,527 adding 'lm_eval/tasks/xstorycloze/default_my.yaml' 2024-03-18T14:20:42,528 adding 'lm_eval/tasks/xstorycloze/default_ru.yaml' 2024-03-18T14:20:42,529 adding 'lm_eval/tasks/xstorycloze/default_sw.yaml' 2024-03-18T14:20:42,530 adding 'lm_eval/tasks/xstorycloze/default_te.yaml' 2024-03-18T14:20:42,531 adding 'lm_eval/tasks/xstorycloze/default_zh.yaml' 2024-03-18T14:20:42,533 adding 'lm_eval/tasks/xwinograd/README.md' 2024-03-18T14:20:42,534 adding 'lm_eval/tasks/xwinograd/utils.py' 2024-03-18T14:20:42,536 adding 'lm_eval/tasks/xwinograd/xwinograd_common_yaml' 2024-03-18T14:20:42,537 adding 'lm_eval/tasks/xwinograd/xwinograd_en.yaml' 2024-03-18T14:20:42,538 adding 'lm_eval/tasks/xwinograd/xwinograd_fr.yaml' 2024-03-18T14:20:42,539 adding 'lm_eval/tasks/xwinograd/xwinograd_jp.yaml' 2024-03-18T14:20:42,540 adding 'lm_eval/tasks/xwinograd/xwinograd_pt.yaml' 2024-03-18T14:20:42,541 adding 'lm_eval/tasks/xwinograd/xwinograd_ru.yaml' 2024-03-18T14:20:42,542 adding 'lm_eval/tasks/xwinograd/xwinograd_zh.yaml' 2024-03-18T14:20:42,544 adding 'lm_eval-0.4.2.dist-info/LICENSE.md' 2024-03-18T14:20:42,548 adding 'lm_eval-0.4.2.dist-info/METADATA' 2024-03-18T14:20:42,550 adding 'lm_eval-0.4.2.dist-info/WHEEL' 2024-03-18T14:20:42,551 adding 'lm_eval-0.4.2.dist-info/entry_points.txt' 2024-03-18T14:20:42,552 adding 'lm_eval-0.4.2.dist-info/top_level.txt' 2024-03-18T14:20:42,590 adding 'lm_eval-0.4.2.dist-info/RECORD' 2024-03-18T14:20:42,640 removing build/bdist.linux-armv7l/wheel 2024-03-18T14:20:43,335 Building wheel for lm-eval (pyproject.toml): finished with status 'done' 2024-03-18T14:20:43,370 Created wheel for lm-eval: filename=lm_eval-0.4.2-py3-none-any.whl size=1424940 sha256=a0bcaff492a09947d8a2b80400f224113b1e313da6a688084711130da18e1af7 2024-03-18T14:20:43,371 Stored in directory: /tmp/pip-ephem-wheel-cache-z2uh4avv/wheels/04/9a/44/74eb560794030dad4faea70af0ff9b7ffb373c953f12181dfd 2024-03-18T14:20:43,453 Successfully built lm-eval 2024-03-18T14:20:43,507 Removed build tracker: '/tmp/pip-build-tracker-9msy7ctu'