Metadata-Version: 2.4
Name: WikibaseIntegrator
Version: 0.13.0
Summary: Python package for reading from and writing to a Wikibase instance
License: MIT
License-File: LICENSE
Keywords: wikibase,wikidata,mediawiki,sparql
Author: Myst
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Dist: backoff (>=2.2.1,<3.0.0)
Requires-Dist: oauthlib (>=3.2.2,<5.0.0)
Requires-Dist: requests (>=2.32.3,<3.0.0)
Requires-Dist: requests-oauthlib (>=2.0.0,<3.0.0)
Requires-Dist: ujson (>=5.10,<7.0)
Project-URL: Bug Tracker, https://github.com/LeMyst/WikibaseIntegrator/issues
Project-URL: Changelog, https://github.com/LeMyst/WikibaseIntegrator/releases
Project-URL: Documentation, https://wikibaseintegrator.readthedocs.io
Project-URL: Homepage, https://github.com/LeMyst/WikibaseIntegrator
Project-URL: Repository, https://github.com/LeMyst/WikibaseIntegrator
Description-Content-Type: text/markdown

# Wikibase Integrator #

[![PyPi](https://img.shields.io/pypi/v/wikibaseintegrator.svg)](https://pypi.python.org/pypi/wikibaseintegrator)
[![Python pytest](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/python-pytest.yaml/badge.svg)](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/python-pytest.yaml)
[![Python Code Quality and Lint](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/python-lint.yaml/badge.svg)](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/python-lint.yaml)
[![CodeQL](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/codeql.yml/badge.svg)](https://github.com/LeMyst/WikibaseIntegrator/actions/workflows/codeql.yml)
[![Pyversions](https://img.shields.io/pypi/implementation/wikibaseintegrator.svg)](https://pypi.python.org/pypi/wikibaseintegrator)
[![Read the Docs](https://readthedocs.org/projects/wikibaseintegrator/badge/?version=latest&style=flat)](https://wikibaseintegrator.readthedocs.io)

Wikibase Integrator is a python package whose purpose is to manipulate data present on a Wikibase instance (like
Wikidata).

# Breaking changes in v0.13 #

v0.13 contains changes that are not backward compatible with v0.12:

- **Python 3.11 or newer is required.** Python 3.9 and 3.10 are no longer supported.
- **The fast run mode has been rewritten** (see [Examples (in "fast run" mode)](#examples-in-fast-run-mode)). Code
  using `wbi_fastrun` directly needs to be updated:
    - `FastRunContainer.get_items()` is renamed `get_entities()`.
    - `FastRunContainer()` takes `base_filter` as its first, mandatory argument and `base_data_type` is now optional.
      `use_refs` is renamed `use_references`, and `use_qualifiers` (default `True`), `use_rank` and `cache` are added.
    - `FastRunContainer.write_required()` drops `cqid` and gains `entity_filter` and `property_filter`.
    - `BaseEntity.write_required()` now requires `base_filter`.
    - Data types compared through SPARQL implement `from_sparql_value()`, which replaces `parse_sparql_value()` for
      the fast run mode.
- **`Form` and `Sense` compare and hash by content** (representations or glosses, grammatical features, claims)
  instead of by identity, and ignore their id. Two distinct objects holding the same content are now equal, and a
  Form/Sense modified after being added to a `set` or used as a `dict` key can no longer be found there.
- **`Claims.add()` appends by default**: its `action_if_exists` default is now `ActionIfExists.APPEND_OR_REPLACE`
  instead of `REPLACE_ALL`, like `BaseEntity.add_claims()`. Adding a claim no longer removes the existing claims with
  the same property. Pass `action_if_exists=ActionIfExists.REPLACE_ALL` to keep the previous behavior. Re-adding an
  existing value now keeps its rank and references, unless the new claim sets them.
- **HTTP requests have a default timeout** of 5 seconds to connect and 300 seconds to read. Change it with
  `wbi_config['TIMEOUT']`, or set it to `None` to wait forever as before.
- **Fewer retries by default**: `max_retries` defaults to 100 instead of 1000 in `mediawiki_api_call_helper()`.
- **`execute_sparql_query()` retries through `wbi_backoff`**: its `max_retries` parameter is ignored (with a
  `DeprecationWarning`). Failed queries are retried up to `wbi_config['BACKOFF_MAX_TRIES']` times (5 by default), with
  an exponential wait capped by `wbi_config['BACKOFF_MAX_VALUE']`, instead of up to 1000 times every 60 seconds. Once
  the retries are exhausted, the last error is raised (e.g. `requests.HTTPError`) instead of a bare `Exception`.
  Authentication errors (HTTP 401/403) are not retried.
- **Anonymous edits** that are not explicitly allowed raise `AnonymousEditNotAllowedError` instead of a bare
  `Exception`.
- **The enums mirroring API values compare equal to them**: `WikibaseDatatype`, `WikibaseRank` and `WikibaseSnakType`
  are `StrEnum`, and `WikibaseTimePrecision` is an `IntEnum`. A member is now equal to its API value
  (`WikibaseRank.PREFERRED == 'preferred'`, `WikibaseTimePrecision.DAY == 11`, both were `False`), and `str()` gives
  that value (`'preferred'` instead of `'WikibaseRank.PREFERRED'`). The JSON sent to the API is unchanged.
- **Empty values are rejected before the write**: a claim, qualifier or reference created with `None` or an empty
  string as value raises `EmptyValueError` (a `ValueError`) listing every empty value, instead of being sent and
  rejected by the API with an `MWApiError`. Use the `WikibaseSnakType.NO_VALUE` or `WikibaseSnakType.UNKNOWN_VALUE`
  snak type to state that there is no value or an unknown value.
- **`mwoauth` is no longer a dependency**: OAuth1 is implemented with `oauthlib` directly. Install `mwoauth` yourself
  if your own code imports it.
- **`wbi_helpers.default_session` and `wbi_helpers.helpers_session` are deprecated**: requests made without a login
  now use a private session per thread. The old names still work but emit a `DeprecationWarning`.

If you want to stay on v0.12.x, you can put this line in your requirements.txt:

```
wikibaseintegrator~=0.12.15
```

---

<!-- ToC generator: https://luciopaiva.com/markdown-toc/ -->

- [WikibaseIntegrator / WikidataIntegrator](#wikibaseintegrator--wikidataintegrator)
- [Documentation](#documentation)
    - [Jupyter notebooks](#jupyter-notebooks)
        - [Common use cases](#common-use-cases)
            - [Read an existing entity](#read-an-existing-entity)
            - [Start a new entity](#start-a-new-entity)
            - [Write an entity to instance](#write-an-entity-to-instance)
            - [Add labels](#add-labels)
            - [Get label value](#get-label-value)
            - [Add aliases](#add-aliases)
            - [Add descriptions](#add-descriptions)
            - [Add a simple claim](#add-a-simple-claim)
            - [Get claim value](#get-claim-value)
            - [Manipulate claim, add a qualifier](#manipulate-claim-add-a-qualifier)
            - [Manipulate claim, add references](#manipulate-claim-add-references)
            - [Remove a specific claim](#remove-a-specific-claim)
            - [Get lemma on lexeme](#get-lemma-on-lexeme)
            - [Set lemma on lexeme](#set-lemma-on-lexeme)
            - [Add gloss to a sense on lexeme](#add-gloss-to-a-sense-on-lexeme)
            - [Add form to a lexeme](#add-form-to-a-lexeme)
            - [Add a form or a sense to an existing lexeme](#add-a-form-or-a-sense-to-an-existing-lexeme)
    - [Other projects](#other-projects)
- [Installation](#installation)
- [Installation of the development environment](#installation-of-the-development-environment)
- [Using a Wikibase instance](#using-a-wikibase-instance)
    - [Wikimedia Foundation User-Agent policy](#wikimedia-foundation-user-agent-policy)
- [The Core Parts](#the-core-parts)
    - [Entity manipulation](#entity-manipulation)
    - [wbi_login](#wbi_login)
        - [Login using OAuth1 or OAuth2](#login-using-oauth1-or-oauth2)
            - [As a bot](#as-a-bot)
            - [To impersonate a user (OAuth 1.0a)](#to-impersonate-a-user-oauth-10a)
        - [Login with a bot password](#login-with-a-bot-password)
        - [Login with a username and a password](#login-with-a-username-and-a-password)
    - [Wikibase Data Types](#wikibase-data-types)
    - [Structured Data on Commons](#structured-data-on-commons)
        - [Retrieve data](#retrieve-data)
        - [Write data](#write-data)
- [More than Wikibase](#more-than-wikibase)
- [Helper Methods](#helper-methods)
    - [Use MediaWiki API](#use-mediawiki-api)
    - [Execute SPARQL queries](#execute-sparql-queries)
        - [Wikimedia Commons Query Service](#wikimedia-commons-query-service)
    - [Wikibase search entities](#wikibase-search-entities)
    - [Merge Wikibase items](#merge-wikibase-items)
    - [Check a Wikibase instance](#check-a-wikibase-instance)
    - [Other helpers](#other-helpers)
    - [Track the edits with EditGroups](#track-the-edits-with-editgroups)
- [Examples (in "normal" mode)](#examples-in-normal-mode)
    - [Create a new Item](#create-a-new-item)
    - [Modify an existing item](#modify-an-existing-item)
    - [A bot for Mass Import](#a-bot-for-mass-import)
    - [Replace a property by another](#replace-a-property-by-another)
- [Examples (in "fast run" mode)](#examples-in-fast-run-mode)
    - [The base filter](#the-base-filter)
    - [Checking if a write is required](#checking-if-a-write-is-required)
    - [Options](#options)
    - [Checking labels, descriptions and aliases](#checking-labels-descriptions-and-aliases)
    - [Limitations](#limitations)
    - [Performance statistics](#performance-statistics)
- [Debugging](#debugging)

# WikibaseIntegrator / WikidataIntegrator #

WikibaseIntegrator (wbi) is a fork of [WikidataIntegrator](https://github.com/SuLab/WikidataIntegrator) (wdi) whose
purpose is to focus on an improved compatibility with Wikibase and adding missing functionalities.
The main differences between these two libraries are :

* A complete rewrite of the library with a more object-oriented architecture allowing for easy interaction, data
  validation and extended functionality
* Add support for reading and writing Lexeme, MediaInfo and Property entities
* Python 3.11 to 3.15 support, validated with unit tests
* Type hints implementation for arguments and return, checked with mypy static type checker
* Add OAuth 2.0 login method
* Add logging module support

WikibaseIntegrator also provides a "fast run" mode, which compares the local data with the live data through the SPARQL
endpoint and only writes to the Wikibase instance when something actually changed (see
[Examples (in "fast run" mode)](#examples-in-fast-run-mode)).

# Documentation #

A (basic) documentation generated from the python source code is available on
the [Read the Docs website](https://wikibaseintegrator.readthedocs.io/).

## Jupyter notebooks ##

You can find some sample code (adding an entity, a lexeme, etc.) in
the [Jupyter notebook directory](https://github.com/LeMyst/WikibaseIntegrator/tree/master/notebooks) of the repository.

### Common use cases

#### Read an existing entity

Get the entity with the QID Q582 from the instance.

From [import_entity.ipynb](notebooks/import_entity.ipynb)

```python
entity = wbi.item.get('Q582')
```

#### Start a new entity

Start a new local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
entity = wbi.item.new()
```

#### Write an entity to instance

Write a local entity to the instance.

From [import_entity.ipynb](notebooks/import_entity.ipynb)

```python
entity.write()
```

#### Add labels

Add an English and a French label to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
entity.labels.set('en', 'New item')
entity.labels.set('fr', 'Nouvel élément')
```

#### Get label value

Get the english label value of the local entity.

From [item_get.ipynb](notebooks/item_get.ipynb)

```python
entity.labels.get('en').value
```

#### Add aliases

Add an English and a French alias to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
entity.aliases.set('en', 'Item')
entity.aliases.set('fr', 'Élément')
```

`aliases.add()` adds one or more aliases while keeping the existing ones. An alias that is already present is left as it is.

```python
entity.aliases.add('en', ['Item', 'Thing'])
```

#### Add descriptions

Add an English and a French description to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
entity.descriptions.set('en', 'A freshly created element')
entity.descriptions.set('fr', 'Un élément fraichement créé')
```

#### Add a simple claim

Add a Time claim with the property P74 and the current time to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
claim_time = datatypes.Time(prop_nr='P74', time='now')

entity.claims.add(claim_time)
```

#### Get claim value

Get the value of the first claim with the property P2048 of the local entity.

From [item_get.ipynb](notebooks/item_get.ipynb)

```python
entity.claims.get('P2048')[0].mainsnak.datavalue['value']['amount']
```

#### Manipulate claim, add a qualifier

* Initialize a new Qualifiers object, add a String qualifier with the property P828 and the value 'Item qualifier' to the Qualifiers object.
* Create a String claim with the property P31533 and the value 'A String property' with the previously created Qualifiers object as qualifiers.
* Add the newly created claim to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
qualifiers = Qualifiers()
qualifiers.add(datatypes.String(prop_nr='P828', value='Item qualifier'))

claim_string = datatypes.String(prop_nr='P31533', value='A String property', qualifiers=qualifiers)
entity.claims.add(claim_string)
```

#### Manipulate claim, add references

* Initialize a new References object.
* Initialize a new Reference object, add a String reference with the property P828 and the value 'Item string reference' to the Reference object.
* Initialize a new Reference object, add a String reference with the property P828 and the value 'Another item string reference' to the Reference object.
* Add the newly created Reference objects to the References object.
* Create a String claim with the property P31533 and the value 'A String property' with the previously created References object as references.
* Add the newly created claim to the local entity.

From [item_create_new.ipynb](notebooks/item_create_new.ipynb)

```python
references = References()
reference1 = Reference()
reference1.add(datatypes.String(prop_nr='P828', value='Item string reference'))

reference2 = Reference()
reference2.add(datatypes.String(prop_nr='P828', value='Another item string reference'))

references.add(reference1)
references.add(reference2)

claim_string = datatypes.String(prop_nr='P31533', value='A String property', references=references)
entity.claims.add(claim_string)
```

#### Remove a specific claim

Remove all claims with the property P31533 and the value Q123 from the local entity.

```python
claims = entity.claims.get('P31533')
for claim in claims:
    if claim.mainsnak.datavalue['value']['id'] == 'Q123':
        claim.remove()
```

#### Get lemma on lexeme

Get all French lemmas of the lexeme.

```python
lexeme.lemmas.get(language='fr')
```

#### Set lemma on lexeme

Add a French lemma with the value 'réponse' to the lexeme.

From [lexeme_update.ipynb](notebooks/lexeme_update.ipynb)

```python
lexeme.lemmas.set(language='fr', value='réponse')
```

#### Add gloss to a sense on lexeme

* Create a new Sense object.
* Add an English gloss with the value 'English gloss' to the Sense object.
* Add a French gloss with the value 'French gloss' to the Sense object.
* Create a String claim with the property P828 and the value 'Create a string claim for sense'.
* Add the newly created claim to the Sense object.
* Add the newly created Sense object to the lexeme.

From [lexeme_write.ipynb](notebooks/lexeme_write.ipynb)

```python
sense = Sense()
sense.glosses.set(language='en', value='English gloss')
sense.glosses.set(language='fr', value='French gloss')
claim = datatypes.String(prop_nr='P828', value="Create a string claim for sense")
sense.claims.add(claim)
lexeme.senses.add(sense)
```

#### Add form to a lexeme

* Create a new Form object.
* Add an English form representation with the value 'English form representation' to the Form object.
* Add a French form representation with the value 'French form representation' to the Form object.
* Create a String claim with the property P828 and the value 'Create a string claim for form'.
* Add the newly created claim to the Form object.
* Add the newly created Form object to the lexeme.

From [lexeme_write.ipynb](notebooks/lexeme_write.ipynb)

```python
form = Form()
form.representations.set(language='en', value='English form representation')
form.representations.set(language='fr', value='French form representation')
claim = datatypes.String(prop_nr='P828', value="Create a string claim for form")
form.claims.add(claim)
lexeme.forms.add(form)
```

#### Add a form or a sense to an existing lexeme

Contrary to `write()`, `write_form()` and `write_sense()` only send the new Form or Sense to the Wikibase instance (with
the `wbladdform` and `wbladdsense` actions), the rest of the lexeme is left untouched. They return the id assigned by the
instance, which is also set on the object.

`write_forms()` and `write_senses()` add every Form or Sense of the lexeme without an id, one request per Form or Sense.

From [lexeme_write.ipynb](notebooks/lexeme_write.ipynb)

```python
lexeme = wbi.lexeme.get('L5')

form = Form()
form.representations.set(language='en', value='English form representation')
form.grammatical_features = ['Q146786']
lexeme.write_form(form)  # 'L5-F3'

sense = Sense()
sense.glosses.set(language='en', value='English gloss')
lexeme.senses.add(sense)
lexeme.write_senses()  # ['L5-S2']
```

## Other projects ##

Here is a list of different projects that use the library:

* https://github.com/ACMILabs/acmi-wikidata-bot - A synchronisation robot to push ACMI API Wikidata links to Wikidata.
* https://github.com/LeMyst/wd-population - Update French population on Wikidata
* https://github.com/SisonkeBiotik-Africa/AfriBioML - Resources for developing a bibliometric study on machine learning for healthcare in Africa
* https://github.com/SisonkeBiotik-Africa/Relational-NER - A Python code for enhancing the output of multilingual named entity recognition based on Wikidata relations
* https://github.com/SoftwareUnderstanding/SALTbot - Software and Article Linker Toolbot
* https://github.com/dpriskorn/ItemSubjector - CLI-tool to easily add "main subject" aka topics in bulk to groups of items on Wikidata
* https://github.com/dpriskorn/hiking_trail_matcher - Script that helps link together hiking trails in Wikidata and OpenStreetMap
* https://github.com/eoan-ermine/wikidata_statistik_population - Update German population on Wikidata
* https://github.com/internetarchive/cgraphbot - Wikibase bot for updating identifiers and citation relationships
* https://github.com/lubianat/ibge_2021_to_wikidata - Update Population of Brazilian Cities
* https://github.com/lcnetdev/lccn-wikidata-bot - Adding LCCNs (Library of Congress Control Number) from NACO (Name Authority Cooperative Program) to Wikidata
* https://github.com/dpriskorn/WikidataEurLexScraper - Improve all Eur-Lex items in Wikidata based on information scraped from Eur-Lex
* https://github.com/dpriskorn/LexDanNet - Help link DanNet 2.2 word ID with Danish Wikidata lexemes
* https://github.com/lubianat/kudos_wikibase
* https://github.com/dlindem/wikibase

# Installation #

The easiest way to install WikibaseIntegrator is to use the `pip` package manager. WikibaseIntegrator supports Python
3.11 and above.

```bash
python -m pip install wikibaseintegrator
```

You can also clone the repository and install it, preferably in a virtualenv.

```bash
git clone https://github.com/LeMyst/WikibaseIntegrator.git

cd WikibaseIntegrator

python -m pip install --upgrade pip setuptools

python -m pip install .
```

If you want to install a specific branch or tag from the repository, you can use the following command:

```bash
python -m pip install git+https://github.com/LeMyst/WikibaseIntegrator.git@v0.13.0
```

where `v0.13.0` is the tag or branch you want to install.

You can also use Poetry:

```bash
python -m pip install --upgrade poetry

python -m poetry install
```

To check that the installation is correct, launch a Python console and run the following code (which will retrieve the
Wikidata element for [Human](https://www.wikidata.org/entity/Q5)):

```python
from wikibaseintegrator import WikibaseIntegrator

wbi = WikibaseIntegrator()
my_first_wikidata_item = wbi.item.get(entity_id='Q5')

# to check successful installation and retrieval of the data, you can print the json representation of the item
print(my_first_wikidata_item.get_json())
```

# Installation of the development environment #

The following commands will install the development environment with the necessary dependencies:

```bash
python -m poetry install --with dev,coverage
```

The `docs` and `notebooks` groups are also available (`--with docs,notebooks`) to build the documentation and run the
Jupyter notebooks.

The unit tests run offline (the HTTP calls are mocked) with:

```bash
python -m pytest
```

The integration tests, running against a real Wikibase instance, are deselected by default and can be run with
`python -m pytest -m integration` (see [test/integration/README.md](test/integration/README.md)).

Optionally, [pre-commit](https://pre-commit.com/) can run the lint checks of the CI before each commit (isort, flynt,
codespell) and before each push (mypy, pylint), using the tools installed by Poetry:

```bash
pipx install pre-commit  # or use "uvx pre-commit"
pre-commit install
```

# Using a Wikibase instance #

WikibaseIntegrator uses Wikidata as default endpoint. To use another instance of Wikibase instead, you can override the
wbi_config module.

An example for a Wikibase instance installed
with [wikibase-docker](https://github.com/wmde/wikibase-release-pipeline/tree/main/example), add this to the top of your
script:

```python
from wikibaseintegrator.wbi_config import config as wbi_config

wbi_config['MEDIAWIKI_API_URL'] = 'http://localhost/api.php'
wbi_config['SPARQL_ENDPOINT_URL'] = 'http://localhost:8834/proxy/wdqs/bigdata/namespace/wdq/sparql'
wbi_config['WIKIBASE_URL'] = 'http://wikibase.svc'
```

You can find more default settings in the file wbi_config.py

## Wikimedia Foundation User-Agent policy ##

If you interact with a Wikibase instance hosted by the Wikimedia Foundation (like Wikidata, Wikimedia Commons, etc.),
it's highly advised to follow the User-Agent policy that you can find on the
page [User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:User-Agent_policy)
of the Wikimedia Meta-Wiki.

You can set a complementary User-Agent by modifying the variable `wbi_config['USER_AGENT']` in wbi_config.

For example, with your library name and contact information:

```python
from wikibaseintegrator.wbi_config import config as wbi_config

wbi_config['USER_AGENT'] = 'MyWikibaseBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'
```

# The Core Parts #

WikibaseIntegrator supports two modes in which it can be used, a normal mode, updating each item at a time, and a fast
run mode, which preloads some data locally and then just updates items if the new data provided differs from Wikidata.
The latter mode allows for great speedups when tens of thousands of Wikidata elements need to be checked for updates,
but only a small number will eventually be updated, a situation typically encountered when synchronising Wikidata with
an external resource.

## Entity manipulation ##

WikibaseIntegrator supports the manipulation of Item, Property, Lexeme and MediaInfo entities through these classes:

* wikibaseintegrator.entities.item.ItemEntity
* wikibaseintegrator.entities.property.PropertyEntity
* wikibaseintegrator.entities.lexeme.LexemeEntity
* wikibaseintegrator.entities.mediainfo.MediaInfoEntity (beta,
  see [Structured Data on Commons](#structured-data-on-commons))

Features:

* Loading a Wikibase entity based on its Wikibase entity ID.
* All Wikibase data types are implemented (and some data types implemented by extensions).
* Full access to the entire Wikibase entity in the form of a JSON dict representation.

## wbi_login ##

`wbi_login` provides the login functionality and also stores the cookies and edit tokens required (For security reasons,
every MediaWiki edit requires an edit token). There are multiple methods to login:

* `wbi_login.OAuth2(consumer_token, consumer_secret)` (recommended)
* `wbi_login.OAuth1(consumer_token, consumer_secret, access_token, access_secret)`
* `wbi_login.Clientlogin(user, password)`
* `wbi_login.Login(user, password)`

There is more parameters available. If you want to authenticate on another instance than Wikidata, you can set the
mediawiki_api_url, mediawiki_rest_url or mediawiki_index_url. Read the documentation for more information.

### Login using OAuth1 or OAuth2 ###

OAuth is the authentication method recommended by the MediaWiki developers. It can be used to authenticate a bot or to
use WBI as a backend for an application.

#### As a bot ####

If you want to use WBI with a bot account, you should use OAuth as
an [Owner-only consumer](https://www.mediawiki.org/wiki/OAuth/Owner-only_consumers). This allows to use the
authentication without the "continue oauth" step.

The first step is to request a new OAuth consumer on your MediaWiki instance on the page
"Special:OAuthConsumerRegistration", the "Owner-only" (or "This consumer is for use only by ...") has to be checked and
the correct version of the OAuth protocol must be set (OAuth 2.0). You will get a consumer token and consumer secret
(and an access token and access secret if you chose OAuth 1.0a). For a Wikimedia instance (like Wikidata), you need to
use the [Meta-Wiki website](https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration).

Example if you use OAuth 2.0:

```python
from wikibaseintegrator import wbi_login

login_instance = wbi_login.OAuth2(consumer_token='<your_client_app_key>', consumer_secret='<your_client_app_secret>')
```

Example if you use OAuth 1.0a:

```python
from wikibaseintegrator import wbi_login

login_instance = wbi_login.OAuth1(consumer_token='<your_consumer_key>', consumer_secret='<your_consumer_secret>',
                                  access_token='<your_access_token>', access_secret='<your_access_secret>')
```

#### To impersonate a user (OAuth 1.0a) ####

If WBI is to be used as a backend for a web application, the script must use OAuth for authentication, WBI supports
this, you just need to specify consumer key and consumer secret when instantiating `wbi_login.OAuth1`. Unlike login by
username and password, OAuth is a 2-step process, as manual confirmation of the user for the OAuth login is required.
This means that the `wbi_login.OAuth1.continue_oauth()` method must be called after creating the `wbi_login.OAuth1`
instance.

Example:

```python
from wikibaseintegrator import wbi_login

login_instance = wbi_login.OAuth1(consumer_token='<your_consumer_key>', consumer_secret='<your_consumer_secret>')
login_instance.continue_oauth(oauth_callback_data='<the_callback_url_returned>')
```

The `wbi_login.OAuth1.continue_oauth()` method will either ask the user for a callback URL (normal bot execution) or
take a parameter. Thus, in the case where WBI is used as a backend for a web application for example, the callback will
provide the authentication information directly to the backend and thus no copy and paste of the callback URL is needed.

### Login with a bot password ###

It's a good practice to use [Bot password](https://www.mediawiki.org/wiki/Manual:Bot_passwords) instead of simple
username and password, this allows limiting the permissions given to the bot.

```python
from wikibaseintegrator import wbi_login

login_instance = wbi_login.Login(user='<bot user name>', password='<bot password>')
```

### Login with a username and a password ###

If you want to log in with your user account, you can use the "clientlogin" authentication method. This method is not
recommended.

```python
from wikibaseintegrator import wbi_login

login_instance = wbi_login.Clientlogin(user='<user name>', password='<password>')
```

## Wikibase Data Types ##

Currently, Wikibase supports 18 different data types. The data types are represented as their own classes in
wikibaseintegrator.datatypes. Each datatype has its own peculiarities, which means that some of them require special
parameters (e.g. Globe Coordinates). They are available under the namespace `wikibaseintegrator.datatypes`.

The data types currently implemented:

* CommonsMedia
* EntitySchema
* ExternalID
* Form
* GeoShape
* GlobeCoordinate
* Item
* Lexeme
* Math
* MonolingualText
* MusicalNotation
* Property
* Quantity
* Sense
* String
* TabularData
* Time
* URL

Two additional data types are also implemented but require the installation of the MediaWiki extension to work properly:

* extra.EDTF ([Wikibase EDTF](https://www.mediawiki.org/wiki/Extension:Wikibase_EDTF))
* extra.LocalMedia ([Wikibase Local Media](https://www.mediawiki.org/wiki/Extension:Wikibase_Local_Media))

For details of how to create values (=instances) with these data types, please (for now) consult the docstrings in the
source code or the documentation website. Of note, these data type instances hold the values and, if specified, data
type instances for references and qualifiers.

The datatype names used by the Wikibase API are listed in the `wikibaseintegrator.wbi_enums.WikibaseDatatype` enum,
e.g. for a new property: `wbi.property.new(datatype=WikibaseDatatype.ITEM)`. Its members are also strings equal to the
API names (`WikibaseDatatype.ITEM == 'wikibase-item'`), so a plain string works too. Each data type class gives its own
name in `DTYPE` (`Item.DTYPE is WikibaseDatatype.ITEM`). A datatype unknown to WikibaseIntegrator, provided by a
Wikibase extension, is kept as a string.

## Structured Data on Commons ##

WikibaseIntegrator supports SDC (Structured Data on Commons) to update a media file hosted on Wikimedia Commons.

> [!WARNING]
> **The MediaInfo support is in beta.** Unlike [MediaWiki](https://www.mediawiki.org/wiki/Stable_interface_policy) and
> [Wikidata](https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy), Structured Data on Commons has no stable
> interface policy ([T149410](https://phabricator.wikimedia.org/T149410)) and its API is not versioned, so it can change
> without notice. `MediaInfoEntity` (`wbi.mediainfo`) follows these changes and may change in a minor release, without
> a deprecation period. The Commons-related datatypes used on Wikidata (CommonsMedia, GeoShape, TabularData) are not
> affected.

> [!IMPORTANT]
> To ease the compatibility between Wikidata and Wikimedia Commons, the MediaInfo entity use the "claims" field instead of the "statements" from the API.
> When you read a MediaInfo entity, the "claims" field will be filled with the data from the "statements" field.
> When you write a MediaInfo entity, the "claims" field will be replaced by the "statements" field.

### Retrieve data ###

```python
from wikibaseintegrator import WikibaseIntegrator

wbi = WikibaseIntegrator()
media = wbi.mediainfo.get('M16431477')

# Retrieve the first "depicts" (P180) claim
print(media.claims.get('P180')[0].mainsnak.datavalue['value']['id'])
```

### Write data ###

```python
from wikibaseintegrator import WikibaseIntegrator
from wikibaseintegrator.datatypes import Item

wbi = WikibaseIntegrator()
media = wbi.mediainfo.get('M16431477')

# Add the "depicts" (P180) claim
media.claims.add(Item(prop_nr='P180', value='Q3146211'))

media.write()
```

To query Structured Data on Commons with SPARQL, or to use the fast run mode with MediaInfo entities, see
[Wikimedia Commons Query Service](#wikimedia-commons-query-service).

# More than Wikibase #

WikibaseIntegrator natively supports some extensions:

* MediaInfo entity (beta, see [Structured Data on Commons](#structured-data-on-commons)) - [WikibaseMediaInfo](https://www.mediawiki.org/wiki/Extension:WikibaseMediaInfo)
* EDTF datatype - [Wikibase EDTF](https://www.mediawiki.org/wiki/Extension:Wikibase_EDTF)
* LocalMedia datatype - [Wikibase Local Media](https://www.mediawiki.org/wiki/Extension:Wikibase_Local_Media)
* Lexeme entity and datatype - [WikibaseLexeme](https://www.mediawiki.org/wiki/Extension:WikibaseLexeme)

When a Lexeme request (getting or writing a lexeme, form or sense) is sent to a Wikibase instance without lexeme
support, a `wikibaseintegrator.wbi_exceptions.MissingExtensionError` naming the missing WikibaseLexeme extension is
raised, instead of the generic error returned by the API.

# Helper Methods #

## Use MediaWiki API ##

The method `wbi_helpers.mediawiki_api_call_helper()` allows you to execute MediaWiki API POST call. It takes a mandatory
data array (data) and multiple optionals parameters like a login object of type wbi_login.Login, a mediawiki_api_url
string if the MediaWiki is not Wikidata, a user_agent string to set a custom HTTP User Agent header, and an
allow_anonymous boolean to force authentication.

Example:

Retrieve last 10 revisions from Wikidata element Q2 (Earth):

```python
from wikibaseintegrator import wbi_helpers

data = {
    'action': 'query',
    'prop': 'revisions',
    'titles': 'Q2',
    'rvlimit': 10
}

print(wbi_helpers.mediawiki_api_call_helper(data=data, allow_anonymous=True))
```

## Execute SPARQL queries ##

The method `wbi_helpers.execute_sparql_query()` allows you to execute SPARQL queries without a hassle. It takes the
actual query string (query), optional prefixes (prefix) if you do not want to use the standard prefixes of Wikidata, the
actual endpoint URL (endpoint), and you can also specify a user agent for the http header sent to the SPARQL server (
user_agent). The latter is very useful to let the operators of the endpoint know who you are, especially if you execute
many queries on the endpoint. This allows the operators of the endpoint to contact you (e.g. specify an email address,
or the URL to your bot code repository.)

If the SPARQL endpoint is protected, pass the credentials with `auth`, either a `(username, password)` tuple for HTTP
Basic authentication or any `requests.auth.AuthBase` instance, or add your own HTTP headers with `headers`. To set the
credentials once for every query, including the ones executed by the fast run mode, use `wbi_config['SPARQL_AUTH']`:

```python
from wikibaseintegrator import wbi_helpers
from wikibaseintegrator.wbi_config import config as wbi_config

query = 'SELECT ?item WHERE { ?item ?p ?o . } LIMIT 1'

wbi_helpers.execute_sparql_query(query, auth=('username', 'password'))
wbi_helpers.execute_sparql_query(query, headers={'Authorization': 'Bearer <token>'})

wbi_config['SPARQL_AUTH'] = ('username', 'password')
wbi_helpers.execute_sparql_query(query)
```

### Wikimedia Commons Query Service ###

The SPARQL endpoint of Structured Data on Commons, the [Wikimedia Commons Query Service](https://commons-query.wikimedia.org/)
(WCQS), authenticates with a long-lived `wcqsOauth` cookie. There is no API to get it: copy it from a web browser logged
in to WCQS, as explained in [Commons:SPARQL query service/API endpoint](https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service/API_endpoint).
`wbi_login.WCQSAuth` sends it, and keeps the short-lived session cookie WCQS sets in return:

```python
from wikibaseintegrator import wbi_helpers
from wikibaseintegrator.wbi_config import config as wbi_config
from wikibaseintegrator.wbi_login import WCQSAuth

wbi_config['SPARQL_ENDPOINT_URL'] = 'https://commons-query.wikimedia.org/sparql'
wbi_config['WIKIBASE_URL'] = 'https://commons.wikimedia.org'
# Every query, including the ones of the fast run mode, is authenticated
wbi_config['SPARQL_AUTH'] = WCQSAuth('<wcqsOauth cookie>')

wbi_helpers.execute_sparql_query('SELECT ?file WHERE { ?file wdt:P180 wd:Q146 } LIMIT 10')
```

An invalid or revoked cookie raises a `WCQSAuthenticationError`, instead of being retried.

## Wikibase search entities ##

The method `wbi_helpers.search_entities()` allows for string search in a Wikibase instance, using the labels and aliases
of the entities. It takes the search string (search_string) and optional parameters: the language of the search
(language, default `wbi_config['DEFAULT_LANGUAGE']`), strict_language to disable the language fallback, the type of
entity to search (search_type: item, property, lexeme, form, sense or mediainfo, default item), the maximum number of
results (max_results, default 50, higher values trigger additional paginated calls) and dict_result to return detailed
dictionaries instead of a list of IDs. Like the other helpers, it also accepts `mediawiki_api_url`, `login`, `user_agent`,
etc.

```python
from wikibaseintegrator import wbi_helpers

print(wbi_helpers.search_entities(search_string='Earth', max_results=5))
```

## Merge Wikibase items ##

Sometimes, Wikibase items need to be merged. An API call exists for that, and `wbi_helpers.merge_items()` implements it.
It takes these arguments:

* the QID of the item which should be merged into another item (from_id)
* the QID of the item the first item should be merged into (to_id)
* a login object of type wbi_login.Login to provide the API call with the required authentication information
* a list of elements to ignore conflicts for (ignore_conflicts), which can contain `'description'`, `'sitelink'` and
  `'statement'`. This will do a partial merge for the elements which do not conflict and should generally be avoided
  because it leaves a crippled item in Wikibase. Before a merge, any potential conflicts should be resolved first.
* a boolean if the changes need to be marked as made by a bot (is_bot)
* a list of change tags to apply to the revision (tags)

Wikibase turns the emptied item into a redirect to the target item, so there is no need to create the redirect
separately.

To merge more than two items at once, `wbi_helpers.merge_multiple_items()` takes a list of QIDs and merges all of them
into the one with the lowest numeric ID, which it returns. It accepts the same login, ignore_conflicts, is_bot and tags
arguments. The merges are done one by one: if one fails, its exception is raised and the merges already done are kept.

```python
from wikibaseintegrator import wbi_helpers, wbi_login

login = wbi_login.Login(user='<bot user name>', password='<bot password>')
to_id = wbi_helpers.merge_multiple_items(['Q100', 'Q30', 'Q5'], login=login)  # Q30 and Q100 are merged into Q5
```

## Check a Wikibase instance ##

`check_wikibase()` reports the features of a Wikibase instance: the MediaWiki version, the installed extensions, the
entity types and the datatypes it supports, and whether the SPARQL endpoint answers. It also checks that
`config['WIKIBASE_URL']` matches the concept URIs of the instance, as the SPARQL queries built by WikibaseIntegrator
(fast run mode, ...) rely on it.

```python
from wikibaseintegrator.wbi_config import config as wbi_config
from wikibaseintegrator.wbi_helpers import check_wikibase

wbi_config['MEDIAWIKI_API_URL'] = 'https://my-wikibase.example.org/w/api.php'
wbi_config['SPARQL_ENDPOINT_URL'] = 'https://my-wikibase.example.org/query/sparql'
wbi_config['WIKIBASE_URL'] = 'https://my-wikibase.example.org'

result = check_wikibase()
print(result)  # A readable report, with the problems found

if not result.has_lexemes:
    print('This instance does not support lexemes')
```

The result also provides `has_mediainfo`, `has_search` (WikibaseCirrusSearch, needed by the `haswbstatement` search
keyword), `has_quality_constraints` (needed by `check_constraints()`), `extensions`, `entity_types`, `datatypes` and
`problems`.

## Other helpers ##

`wbi_helpers` also provides:

* `merge_lexemes()`: merge two lexemes.
* `remove_claims()`: delete a claim from an entity by its claim ID.
* `check_constraints()`: check the constraint violations of entities or claims with the WikibaseQualityConstraints
  extension.
* `fulltext_search()`: perform a fulltext search on the MediaWiki instance.
* `delete_page()`: delete a page.
* `download_entity_ttl()`: download the RDF (Turtle) representation of an entity.

Some examples are available in the [wbi_helpers.ipynb](notebooks/wbi_helpers.ipynb) notebook.

## Track the edits with EditGroups ##

[EditGroups](https://www.wikidata.org/wiki/Wikidata:Edit_groups) tracks the edits of a bot on Wikidata as a batch,
which can be reviewed, discussed and reverted at once. An edit is part of a batch when its summary ends with a link to
the batch, which `EditGroup` generates:

```python
from wikibaseintegrator import WikibaseIntegrator
from wikibaseintegrator.datatypes import Item
from wikibaseintegrator.wbi_editgroups import EditGroup

wbi = WikibaseIntegrator(login=login_instance)

# One EditGroup per batch, with a random batch ID
editgroup = EditGroup()
print(editgroup.url)  # https://editgroups.toolforge.org/b/CB/<batch ID>/

for qid in ['Q582', 'Q90']:
    item = wbi.item.get(qid)
    item.claims.add(Item(value='Q484170', prop_nr='P31'))
    # "Add instance of ([[:toolforge:editgroups/b/CB/<batch ID>|details]])"
    item.write(summary=editgroup.summary('Add instance of'))
```

To add edits to an existing batch, for example after a restart of the bot, pass its ID: `EditGroup('89ead4fe')`. A
summary that is too long is shortened, so that the link is never truncated.

# Examples (in "normal" mode) #

In order to create a minimal bot based on WikibaseIntegrator, two things are required:

* A datatype object containing a value.
* An entity object (Item/Property/Lexeme/...) which takes the data, does the checks and performs write.

An optional Login object can be used to be authenticated on the Wikibase instance.

## Create a new Item ##

```python
from wikibaseintegrator import wbi_login, WikibaseIntegrator
from wikibaseintegrator.datatypes import ExternalID
from wikibaseintegrator.wbi_config import config as wbi_config

wbi_config['USER_AGENT'] = 'MyWikibaseBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'

# login object
login_instance = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')

wbi = WikibaseIntegrator(login=login_instance)

# data type object, e.g. for a NCBI gene entrez ID
entrez_gene_id = ExternalID(value='<some_entrez_id>', prop_nr='P351')

# data goes into a list, because many data objects can be provided to
data = [entrez_gene_id]

# Create a new item
item = wbi.item.new()

# Set an english label
item.labels.set(language='en', value='Newly created item')

# Set a French description
item.descriptions.set(language='fr', value='Une description un peu longue')

item.claims.add(data)
item.write()
```

## Modify an existing item ##

```python
from wikibaseintegrator import wbi_login, WikibaseIntegrator
from wikibaseintegrator.datatypes import ExternalID
from wikibaseintegrator.wbi_enums import ActionIfExists
from wikibaseintegrator.wbi_config import config as wbi_config

wbi_config['USER_AGENT'] = 'MyWikibaseBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'

# login object
login_instance = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')

wbi = WikibaseIntegrator(login=login_instance)

# data type object, e.g. for a NCBI gene entrez ID
entrez_gene_id = ExternalID(value='<some_entrez_id>', prop_nr='P351')

# data goes into a list, because many data objects can be provided to
data = [entrez_gene_id]

# Search and then edit an Item
item = wbi.item.get(entity_id='Q141806')

# Set an english label but don't modify it if there is already an entry
item.labels.set(language='en', value='An updated item', action_if_exists=ActionIfExists.KEEP)

# Set a French description and replace the existing one
item.descriptions.set(language='fr', value='Une description un peu longue', action_if_exists=ActionIfExists.REPLACE_ALL)

# Add the claims, the existing claims with the same property are kept
item.claims.add(data)

# To replace all the existing claims with the same property instead, use ActionIfExists.REPLACE_ALL
# item.claims.add(data, action_if_exists=ActionIfExists.REPLACE_ALL)

item.write()
```

## A bot for Mass Import ##

An enhanced example of the previous bot puts the creation of the data and the entity into a 'for loop' and so allows
mass creation or modification of items.

```python
from wikibaseintegrator import WikibaseIntegrator, wbi_login
from wikibaseintegrator.datatypes import ExternalID, Item, String, Time
from wikibaseintegrator.wbi_config import config as wbi_config
from wikibaseintegrator.wbi_enums import WikibaseTimePrecision

wbi_config['USER_AGENT'] = 'MyWikibaseBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'

# login object
login_instance = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')

# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl Gene IDs
raw_data = {
    '50943': 'ENST00000376197',
    '1029': 'ENST00000498124'
}

wbi = WikibaseIntegrator(login=login_instance)

for entrez_id, ensembl in raw_data.items():
    # add some references
    references = [
        [
            Item(value='Q20641742', prop_nr='P248'),
            Time(time='+2020-02-08T00:00:00Z', prop_nr='P813', precision=WikibaseTimePrecision.DAY),
            ExternalID(value='1017', prop_nr='P351')
        ]
    ]

    # data type object
    entrez_gene_id = String(value=entrez_id, prop_nr='P351', references=references)
    ensembl_transcript_id = String(value=ensembl, prop_nr='P704', references=references)

    # data goes into a list, because many data objects can be provided to
    data = [entrez_gene_id, ensembl_transcript_id]

    # Search for and then edit/create new item
    item = wbi.item.new()
    item.claims.add(data)
    item.write()
```

## Replace a property by another ##

This bot moves every statement using a property (`OLD_PROPERTY`) to another property with the same datatype
(`NEW_PROPERTY`), on all the entities of a Wikibase instance. Each statement is copied with its value, qualifiers,
references and rank to the new property, then the original statement is removed, in a single edit per entity.

```python
from copy import deepcopy

from wikibaseintegrator import WikibaseIntegrator, wbi_login
from wikibaseintegrator.wbi_config import config as wbi_config
from wikibaseintegrator.wbi_helpers import execute_sparql_query, generate_entity_instances

wbi_config['USER_AGENT'] = 'MyWikibaseBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'

OLD_PROPERTY = 'P1234'  # The property to replace
NEW_PROPERTY = 'P5678'  # The property to use instead
DRY_RUN = True  # Set to False to write the changes

# login object
login_instance = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')

wbi = WikibaseIntegrator(login=login_instance)

# The values are copied as they are, so both properties must use the same datatype
old_datatype = wbi.property.get(OLD_PROPERTY).datatype
new_datatype = wbi.property.get(NEW_PROPERTY).datatype
if old_datatype != new_datatype:
    raise ValueError(f'{OLD_PROPERTY} ({old_datatype}) and {NEW_PROPERTY} ({new_datatype}) use different datatypes')

# Find the entities holding at least one statement with the old property
query = f'SELECT DISTINCT ?entity WHERE {{ ?entity <{wbi_config["WIKIBASE_URL"]}/prop/{OLD_PROPERTY}> ?statement . }}'
results = execute_sparql_query(query)
entity_ids = [result['entity']['value'].rsplit('/', 1)[-1] for result in results['results']['bindings']]
# The statements of the forms and senses of a lexeme (L1-F1, L1-S1) are not handled by this example
entity_ids = [entity_id for entity_id in entity_ids if '-' not in entity_id]
print(f'{len(entity_ids)} entities use {OLD_PROPERTY}')

# Retrieve the entities 50 by 50, the maximum of the wbgetentities API action
for start in range(0, len(entity_ids), 50):
    for entity_id, entity in generate_entity_instances(entities=entity_ids[start:start + 50], login=login_instance):
        for claim in entity.claims.get(OLD_PROPERTY):
            # Copy the statement to the new property, as a new statement
            new_claim = deepcopy(claim)
            new_claim.reset_id()
            new_claim.mainsnak.property_number = NEW_PROPERTY
            # Not added twice if the entity already holds the same value with the new property
            entity.claims.add(new_claim)

            # Remove the original statement
            claim.remove()

        if DRY_RUN:
            print(f'{entity_id}: {len(entity.claims.get(OLD_PROPERTY))} statement(s) to move')
        else:
            entity.write(summary=f'Replace {OLD_PROPERTY} by {NEW_PROPERTY}', is_bot=True)
```

> [!NOTE]
> The SPARQL endpoint can lag behind the live data: run the bot again until no entity is found. This example only
> moves the main statements: the qualifiers and references using the old property are left untouched.

# Examples (in "fast run" mode) #

The fast run mode is designed for bots synchronising an external resource with a Wikibase instance, where most of the
entities are usually already up to date. Instead of loading every entity through the MediaWiki API to compare it with
the local data, the fast run mode loads the current state of the entities of the data corpus with paginated SPARQL
queries, then decides locally, entity per entity, if a write is required. When nothing changed (the most common case),
no MediaWiki API call is made at all for that entity.

In order to use the fast run mode, you need to know the property/value combination which determines the data corpus you
would like to operate on. E.g. for operating on human genes, you need to know
that [P351](https://www.wikidata.org/entity/P351) is the NCBI Entrez Gene ID and you also need to know that you are
dealing with humans, best represented by the [found in taxon property (P703)](https://www.wikidata.org/entity/P703) with
the value [Q15978631](https://www.wikidata.org/entity/Q15978631) for Homo sapiens.

IMPORTANT: In order for the fast run mode to work on entities without a known entity ID, the data you provide must
contain at least one unique value/id only present on one Wikibase entity, e.g. an NCBI entrez gene ID, Uniprot ID, etc.
This unique value is used to identify the target entity. If the entity ID is already known (e.g. the entity was loaded
with `wbi.item.get()`), the comparison is automatically restricted to that entity and this requirement does not apply.

## The base filter ##

The base filter defines the data corpus and is a list of datatype instances (the same classes used to create claims).
Three forms are supported:

* A property with a value: the entities must have this exact statement, e.g. `Item(prop_nr='P703', value='Q15978631')`
  (found in taxon Homo sapiens).
* A property without a value: the entities must have a statement with this property, whatever the value, e.g.
  `ExternalID(prop_nr='P351')` (any NCBI Entrez Gene ID).
* A property path, given as a list of two datatype instances: the value is reached through the first property followed
  by any number of hops with the second one, e.g. `[Item(prop_nr='P31', value='Q11173'), Item(prop_nr='P279')]`
  (instance of (P31) chemical compound (Q11173), directly or through a chain of subclass of (P279)).

```python
from wikibaseintegrator.datatypes import ExternalID, Item

fast_run_base_filter = [ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')]
```

## Checking if a write is required ##

The entry point is `entity.write_required()`. It returns `True` if the local entity differs from the live data and an
actual write is needed, `False` if the entity is already up to date and the write can be skipped. The full example, a
modified version of the mass import bot from the normal mode example:

```python
from wikibaseintegrator import WikibaseIntegrator, wbi_login
from wikibaseintegrator.datatypes import ExternalID, Item, Time
from wikibaseintegrator.wbi_enums import WikibaseTimePrecision

# login object
login = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')

wbi = WikibaseIntegrator(login=login)

fast_run_base_filter = [ExternalID(prop_nr='P351'), ExternalID(prop_nr='P704'), Item(prop_nr='P703', value='Q15978631')]

# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl transcript IDs
# You can iterate over any data source as long as you can map the values to Wikidata properties.
raw_data = {
    '50943': 'ENST00000376197',
    '1029': 'ENST00000498124'
}

for entrez_id, ensembl in raw_data.items():
    # add some references
    references = [
        [
            Item(value='Q20641742', prop_nr='P248'),
            Time(time='+2020-02-08T00:00:00Z', prop_nr='P813', precision=WikibaseTimePrecision.DAY)
        ]
    ]

    # data type objects
    entrez_gene_id = ExternalID(value=entrez_id, prop_nr='P351', references=references)
    ensembl_transcript_id = ExternalID(value=ensembl, prop_nr='P704', references=references)

    # data goes into a list, because many data objects can be provided to
    data = [entrez_gene_id, ensembl_transcript_id]

    item = wbi.item.new()
    item.claims.add(data)

    # Compare the local data with the live data and only write when something differs
    if item.write_required(base_filter=fast_run_base_filter):
        item.write()
```

Note: only the claims whose property appears in the base filter are compared. In the example above, the P351 and P704
claims are checked because both properties are part of the base filter; a claim on any other property would be ignored
by the comparison. A write is reported as not required only when one entity (the entity being edited, when its ID is
known) holds all the compared claims.

## Options ##

`write_required()` accepts optional parameters, forwarded to the fastrun container:

* `use_qualifiers` (default `True`): also compare the qualifiers of the statements.
* `use_references` (default `False`): also compare the references of the statements. By default, references are
  ignored.
* `use_rank` (default `False`): also compare the rank of the statements.
* `case_insensitive` (default `False`): compare string values case-insensitively. The comparison of qualifiers,
  references and ranks stays case sensitive.
* `cache` (default `True`): keep the data returned by the SPARQL endpoint in memory and reuse it for the next
  checks. When disabled, the queries are restricted to the exact values of the claims instead of preloading every
  statement of the property.
* `action_if_exists` (default `ActionIfExists.REPLACE_ALL`): the action that will be used for the write. With
  `FORCE_APPEND`, the statements are always appended and a write is always reported as required. With the other
  actions, the claims must already exist on the entity for the write to be skipped.

The preloaded data is shared: containers are cached at the module level and reused by every `write_required()` call
using the same base filter and options, so the SPARQL queries are only executed once per property.

## Checking labels, descriptions and aliases ##

`write_required()` only compares claims. To check language data (labels, descriptions or aliases), use the fastrun
container directly:

```python
from wikibaseintegrator import wbi_fastrun
from wikibaseintegrator.datatypes import ExternalID, Item

frc = wbi_fastrun.get_fastrun_container(base_filter=[ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')])

# Resolve the entity ID from a unique value, without any MediaWiki API call
qids = frc.get_entities(claims=[ExternalID(value='50943', prop_nr='P351')])

# Returns True if the English label differs from 'CDK7' and a write is required
frc.check_language_data(qid=qids[0], lang_data=['CDK7'], lang='en', lang_data_type='label')
```

## Limitations ##

* The SPARQL endpoint can lag behind the live data (typically a few seconds to a few minutes on Wikidata). A write
  that just happened may not be visible yet, and the same write could be reported as required again.
* The attributes missing from the SPARQL simple values are rebuilt with their default value: the bounds of a quantity,
  the precision of a globe coordinate, the timezone and calendar model of a time, etc. Statements using non-default
  attributes are always reported as requiring a write. The precision of a time and the unit of a quantity are loaded
  from the value nodes, so they are compared.
* EntitySchema has no `PTYPE` mapping to the Wikibase ontology, so its statements are never matched by the fastrun
  data and are always reported as requiring a write. Every other datatype implements `from_sparql_value()`.
* The qualifiers, references and ranks are loaded lazily with one SPARQL query per statement to check. This is fast
  when most entities are up to date, but can slow down the run when many entities hold the same values.

## Performance statistics ##

Measured on a synchronisation bot of French administrative entities with Wikidata. The "partial fastrun" column is the
mode without cache, where the queries are restricted to the exact values of the claims:

| Dataset                     | partial fastrun | fastrun without qualifiers/references | fastrun with qualifiers | fastrun with qualifiers/references |
|:----------------------------|----------------:|--------------------------------------:|------------------------:|-----------------------------------:|
| Communes (34990 elements)   |               ? |                                  7min |                     30s |                                60s |
| Départements (100 elements) |           70min |                                    1s |                     30s |                                60s |

# Debugging #

You can enable debugging by adding this piece of code to the top of your project:

```python
import logging

logging.basicConfig(level=logging.DEBUG)
```

