Metadata-Version: 2.4
Name: swh.lister
Version: 7.3.1
Summary: Software Heritage lister
Author-email: Software Heritage developers <swh-devel@inria.fr>
Project-URL: Homepage, https://gitlab.softwareheritage.org/swh/devel/swh-lister
Project-URL: Bug Reports, https://gitlab.softwareheritage.org/swh/devel/swh-lister/-/issues
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-lister/
Project-URL: Source, https://gitlab.softwareheritage.org/swh/devel/swh-lister.git
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.9
Description-Content-Type: text/x-rst
License-File: LICENSE
License-File: AUTHORS
Requires-Dist: beautifulsoup4>=4.13.3
Requires-Dist: breezy!=3.3.21,>=3.3.12
Requires-Dist: dateparser
Requires-Dist: dulwich
Requires-Dist: iso8601
Requires-Dist: launchpadlib
Requires-Dist: looseversion
Requires-Dist: lxml
Requires-Dist: mercurial
Requires-Dist: ndjson
Requires-Dist: psycopg
Requires-Dist: pyreadr
Requires-Dist: python_debian
Requires-Dist: python-magic
Requires-Dist: repomd
Requires-Dist: requests
Requires-Dist: setuptools
Requires-Dist: subvertpy
Requires-Dist: tenacity>=8.4.2
Requires-Dist: testing.postgresql
Requires-Dist: toml
Requires-Dist: swh.core[db]>=4.0.0
Requires-Dist: swh.scheduler>=3.0.0
Provides-Extra: testing
Requires-Dist: celery-types; extra == "testing"
Requires-Dist: pandas; extra == "testing"
Requires-Dist: pandas-stubs; extra == "testing"
Requires-Dist: pytest>=8.1; extra == "testing"
Requires-Dist: pytest-mock; extra == "testing"
Requires-Dist: requests_mock; extra == "testing"
Requires-Dist: swh.scheduler[pytest]>=3.1.0; extra == "testing"
Requires-Dist: types-beautifulsoup4; extra == "testing"
Requires-Dist: types-dateparser; extra == "testing"
Requires-Dist: types-pyyaml; extra == "testing"
Requires-Dist: types-requests; extra == "testing"
Requires-Dist: types-toml; extra == "testing"
Requires-Dist: types-lxml; extra == "testing"
Dynamic: license-file

Software Heritage - Listers
===========================

Collection of listers for source code distribution places like development
forges, FOSS distributions, package managers, etc. Each lister is in charge to
enumerate the software origins (e.g., VCS, packages, etc.) available at a
source code distribution place.

A lister is a component from the Software Heritage stack aims to produce
listings of software origins and their urls hosted on various public developer
platforms or package managers. As these operations are quite similar, this
package provides a set of Python modules abstracting common software origins
listing behaviors.

It also provides several lister implementations, contained in the Python
``swh.lister.*`` modules. See `this documentation
<https://docs.softwareheritage.org/user/listers.html>`_ for the list of
supported listers.


Dependencies
------------

All required python dependencies can be found in the ``requirements*.txt`` files
located at the root of the repository.

In order to be able to run all the listers (and thus execute the tests), some
tools must be available on your system, namely:

- ``opam``
- ``tar``
- ``psql``

On a Debian-like system, you may use:

.. code-block: console

   $ sudo apt update
   $ sudo apt install opam tar postgresql-client-common


Local deployment
----------------

Lister configuration
++++++++++++++++++++

Each lister implemented so far by Software Heritage (``bitbucket``, ``cgit``,
``cran``, ``debian``, ``gitea``, ``github``, ``gitlab``, ``gnu``, ``golang``,
``launchpad``, ``npm``, ``packagist``, ``phabricator``, ``pypi``, ``tuleap``,
``maven``, ``forgejo``) must be configured by following the instructions below
(please note that you have to replace ``<lister_name>`` by one of the lister names
introduced above).

Preparation steps
~~~~~~~~~~~~~~~~~

1. ``mkdir ~/.config/swh/``
2. create configuration file ``~/.config/swh/listers.yml``

Configuration file sample
+++++++++++++++++++++++++

Minimalistic configuration shared by all listers to add in file
``~/.config/swh/listers.yml``:

.. code-block:: yaml

   scheduler:
     cls: 'remote'
     args:
       url: 'http://localhost:5008/'

   credentials: {}

Note: This expects scheduler (5008) service to run locally

Executing a lister
------------------

Once configured, a lister can be executed by using the ``swh`` CLI tool with
the following options and commands:

.. code-block:: shell

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]


Examples:

.. code-block:: shell

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://demo.gitea.com/api/v1/

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister forgejo url=https://try.next.forgejo.org/

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm

   $ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi


Licensing
---------

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

See top-level LICENSE file for the full text of the GNU General Public License
along with this program.
