Metadata-Version: 2.4
Name: aloelite
Version: 0.3.0rc3
Summary: Aloelite SQLite Filesystem
Author-email: Michael Godfrey <michael@aloecraft.org>
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: cryptography
Requires-Dist: pydantic
Requires-Dist: pyyaml
Requires-Dist: msgpack
Requires-Dist: flask
Provides-Extra: fuse
Requires-Dist: pyfuse3; extra == "fuse"
Requires-Dist: trio; extra == "fuse"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"

# aloelite

<div align="center">

<img src="https://raw.githubusercontent.com/Aloecraft-org/aloelite/refs/heads/main/doc/aloelite.png" style="height:96px; width:96px;"/>

**Aloelite SQLite Filesystem**

**Overview (current)** | [Getting Started](https://github.com/Aloecraft-org/aloelite/blob/main/doc/GETTING_STARTED.md) |  [Frequently Asked Questions](https://github.com/Aloecraft-org/aloelite/blob/main/doc/FAQ.md) 

[Troubleshooting](https://github.com/Aloecraft-org/aloelite/blob/main/doc/TROUBLESHOOTING.md) | [Requirements Spec](https://github.com/Aloecraft-org/aloelite/blob/main/doc/REQUIREMENTS.md) | [Encryption Spec](https://github.com/Aloecraft-org/aloelite/blob/main/doc/ENCRYPTION.md)

[![PyPI Version](https://img.shields.io/pypi/v/aloelite.svg)](https://pypi.org/project/aloelite/)
[![Python Versions](https://img.shields.io/pypi/pyversions/aloelite.svg)](https://pypi.org/project/aloelite/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

[![CI Status](https://github.com/Aloecraft-org/aloelite/actions/workflows/main.yml/badge.svg)](https://github.com/Aloecraft-org/aloelite/actions/workflows/main.yml)
[![Downloads](https://static.pepy.tech/badge/aloelite)](https://pepy.tech/project/aloelite)

</div>

## Contents
- [Installation](#installation)
- [Usage](#usage)
- [Overview](#overview)
- [Getting Started](#getting-started)
    + [Python API](#python-api)
    + [Encryption](#encryption)
    + [Pathlib-Style Interface](#pathlib-style-interface)
- [Command Line](#command-line)
- [Mount API](#mount-api)
- [FUSE](#fuse)
- [Volume Manager and WebUI](#volume-manager-and-webui)
    + [REST API](#rest-api)
    + [Backup Sync Pattern](#backup-sync-pattern)
- [Security Notes](#security-notes)
- [Design Background](#design-background)
- [License](#license)

## Installation

``` bash
pip install aloelite
```

The core install has no FUSE dependencies and runs anywhere Python does.
For FUSE support (Linux only):

```bash
sudo apt install fuse3 libfuse3-dev
pip install aloelite[fuse]
```

For Docker:

```bash
docker pull aloecraft/aloelite
```

## Usage

**CLI**
```bash
aloelite -f notebook.fs put report.pdf /docs/report.pdf
```

**Python**
```python
with Aloelite("notebook.fs") as fs, fs.mount("docs", create=True) as m:
    m.put("/hello.txt", b"hello world")
```

**Docker (volume manager + WebUI)**
```bash
docker run -d --privileged --device /dev/fuse -p 8080:8080 \
  -v /aloelite-root:/aloelite-root -v /mnt/aloelite:/mnt:rshared \
  aloecraft/aloelite-manager   # then open http://localhost:8080/admin
```

**FUSE Mount**
```bash
aloelite-fuse photos.fs photos ~/photos   # now it's just a directory
```

## Overview

Aloelite is a filesystem implemented as a SQLite database. The entire filesystem, (i.e. files, directories, metadata, and content) lives in a single portable `.sqlite` file that can be copied, versioned, and opened anywhere SQLite runs.

It is designed for situations where you want filesystem semantics (paths, directories, streaming I/O) but need more control than a raw filesystem gives you: portable snapshots, at-rest encryption, content deduplication, and a clean programmatic API. A single file is easier to back up, replicate, and audit than a directory tree.

**What It Provides**

- **Python API** — full filesystem semantics with bounded-memory streaming I/O and atomic random-access writes (`write_range`/`truncate`)
- **CLI** (`aloelite`) — the same operations from the shell, with stdin/stdout piping
- **FUSE** — mount a volume as a plain directory; any application can use it unmodified
- **Volume manager + WebUI** — a container that provisions volumes for Docker/Podman over HTTP, with a browser admin panel and file explorer
- **At-rest encryption** per volume (ChaCha20-Poly1305, Argon2id); the PIN is used only at mount time and never stored
- **Content deduplication** — identical data is stored once per volume, including across repeated backups
- **Live snapshots** — export a consistent, self-contained SQLite file while the volume is mounted and in use

**Implementation Status**

- **Core model** — nodes, edges, volumes, and mounts are fully realized: path resolution, structural operations (create, move, rename, copy, remove, pack/unpack), advisory locking, and mount-scoped sessions.
- **Content storage** — content-addressed chunk pool with deduplication, per-version manifests, configurable retention, and bounded-memory streaming I/O; production-validated against files in the tens of gigabytes.
- **Random access** — `write_range` and `truncate` are first-class engine operations: unchanged chunks carry into the new version by reference, so partial overwrites of large files are cheap and bounded-memory.
- **Encryption** — at-rest at the storage boundary (ChaCha20-Poly1305, Argon2id, per-volume wrapped key), with convergent and random nonce modes.
- **FUSE** — O_RDWR access through a dirty-extent handle (memory bounded by dirty bytes, flushed atomically on fsync/release); symlinks and permission bits (chmod, executables) persist across remounts; honors utimens; streaming writes commit at flush time, synchronously with the application's `close()`, so a crashed daemon loses only truly in-flight data; hardened handlers return EIO rather than detaching the mount. Hard links, shared-writable mmap, and byte-range locks across separate mounts are not yet implemented.
- **CLI** — covers the library verbs for scripting.
- **Volume manager** (`manager/`) — exposes volumes as FUSE-mounted directories over an HTTP API with a WebUI; usable as a Docker/Podman volume provisioner.

Reserved but not yet realized:
- cryptographic verification of the node tree (Merkle structure over content and placement)
- content-defined chunking
- key rotation
- graph-shaped namespaces beyond the default hierarchical tree
- and node metadata encryption (currently plaintext in the SQLite schema. (see [Security Notes](#security-notes)))

## Getting Started

New here? **[GETTING_STARTED.md](doc/GETTING_STARTED.md)** is the friendly
tour, organized by use case (Python / CLI / FUSE / Docker). See also
**[FAQ.md](doc/FAQ.md)** and **[TROUBLESHOOTING.md](doc/TROUBLESHOOTING.md)**.

### Python API

```python
from aloelite.aloelite import Aloelite
from aloelite.types import WriteMode, Whence

with Aloelite("photos.sqlite") as fs:
    # Mount by name; create=True bootstraps the volume on first run.
    # (Volumes can also be managed explicitly: create_volume / list_volumes /
    #  resolve_volume_name, then mount by id.)
    with fs.mount("photos", create=True) as m:
        m.put("/hello.txt", b"hi")          # create-or-replace, one atomic op
        m.put("/hello.txt", b"hi again")    # replace
        m.put("/log.txt", b"x\n", append=True)  # append (creates if missing)

        m.mkdir("/2024/trip", parents=True, exist_ok=True)  # mkdir -p

        for e in m.list("/"):
            print(e.path)                   # full path, stamped at fetch time

        m.create_container("/2024")
        m.set_metadata("/2024", {"year": "2024", "album": "trip"})
        m.create_entry("/2024/caption.txt", b"a sunset")

        with m.open_write("/note.txt") as w:
            w.write(b"hello ")
            w.write(b"world")

        print(m.read_all("/note.txt"))   # -> b"hello world"

        with m.open_read("/note.txt") as r:
            head = r.read(5)
            r.seek(-5, Whence.END)
            tail = r.read()

        m.write_range("/note.txt", 6, b"WORLD")  # atomic in-place overwrite
        m.truncate("/note.txt", 5)               # -> b"hello"

        m.rename("/note.txt", "readme.txt")
        m.move("/readme.txt", "/2024/readme.txt")
        m.copy("/2024", "/backup")
        m.remove_recursive("/backup")

    fs.prune()
    print(fs.health_check())   # -> [] when consistent
```

### Encryption

```python
PIN = b"correct-horse-battery-staple"

with Aloelite("vault.sqlite") as fs:
    # create=True with a pin creates the volume encrypted; encryption is
    # decided once, at creation (Argon2id key derivation, ChaCha20-Poly1305).
    with fs.mount("vault", pin=PIN, create=True) as m:
        m.create_entry("/secret.txt", b"eyes only")
        print(m.read_all("/secret.txt"))   # -> b"eyes only"

    # Wrong PIN is rejected at mount time (not at read time)
    from aloelite import errors
    try:
        fs.mount("vault", pin=b"wrong")
    except errors.BadKey:
        print("wrong PIN rejected ✓")

    # Durable mounts (records, not sessions) are listable and re-attachable:
    for info in fs.list_mounts():
        print(info.id, info.mount_path, info.state.value)
```

Encryption is invisible at the `Mount` API level. Use `enc_mode="random"` to trade chunk deduplication for zero equality leakage.

### Pathlib-Style Interface

The easiest way to work with files inside a volume. No FUSE required. Any `Mount` doubles as a path root:

```python
from aloelite.aloelite import Aloelite

with Aloelite("photos.sqlite") as fs:
    vol = fs.create_volume("photos")

    with fs.mount(vol.id) as m:
        docs = m / "docs"                        # Mount / str -> AloelitePath
        docs.mkdir(parents=True, exist_ok=True)

        note = docs / "note.txt"
        note.write_text("hello world")
        print(note.read_text())                  # -> "hello world"

        with (docs / "big.bin").open("wb") as w: # bounded-memory streaming
            w.write(b"chunk " * 100_000)

        for child in docs.iterdir():
            print(child, child.stat().size)

        for txt in m.path("/").rglob("*.txt"):   # '*' and '**' globbing
            print(txt)

        note.set_metadata({"author": "mg"})      # NODE-6 metadata
        note.copy("/docs/note.bak")
        note.rename("/docs/renamed.txt")         # full move, returns new path
```

`AloelitePath` is pure sugar over the `Mount` API — it adds nothing to the contract, so everything above is atomic, deduplicated, and encryption-transparent exactly like the underlying operations.

## Mount API

Everything in Aloelite goes through one interface: the **Mount API**.
It is defined once, language-neutrally, in `config/mount-api.yaml`, and
implemented in `aloelite/operations.py` (the Python reference). The CLI,
FUSE driver, volume manager, and pathlib wrapper are all thin consumers
of the same operations — there is no second path into the file.

The mental model is four nouns:

- A **volume** is a filesystem tree inside the file (one file can hold many).
- A **mount** is a durable access point into a volume — all reads and
  writes are brokered through one.
- Below that, a volume is made of **nodes** (files and directories) and
  **edges** (placements) — you rarely touch these directly, but they are
  why moves are cheap and history is recoverable.

Every operation you see in the Python examples below (`put`, `list`,
`move`, `open_write`, ...) is a Mount API operation. The CLI exposes the
same verbs; FUSE translates kernel calls into them. Learn the API once
and every interface is familiar.

## Command line

Mount API usage from the command line

```bash
# installed with `pip install aloelite`

aloelite -f notebook.fs volumes
aloelite -f notebook.fs ls -l /
aloelite -f notebook.fs put report.pdf /docs/report.pdf
cat log | aloelite -f notebook.fs put - /logs/today --append
aloelite -f notebook.fs get /docs/report.pdf -   # to stdout
aloelite -f notebook.fs mkdir -p /a/b/c
aloelite -f notebook.fs mv /a.txt /docs/a.txt
aloelite -f notebook.fs rm -r /old
aloelite -f notebook.fs cat /docs/report.pdf
aloelite -f notebook.fs cp /docs/report.pdf /backup/report.pdf   # dedup: near-free
aloelite -f notebook.fs stat /docs/report.pdf
aloelite -f notebook.fs tree /
aloelite -f notebook.fs prune --vacuum
aloelite -f notebook.fs mounts # List Mounts (ACC-1a)
```

note: Set `ALOELITE_FILE` env var to skip `-f` entirely (`export ALOELITE_FILE=notebook.fs`).


`-v NAME_OR_ID` selects a volume (name, or uuid7 with/without dashes);
omit it when the file holds exactly one. Encrypted volumes take the same
`--pin` / `--pin-file` / `--pin-env` flags as `aloelite-fuse`, or prompt
interactively.


## FUSE

Mount an Aloelite volume as a regular directory (Linux, requires `fuse3`):

```bash
# Plain volume (--create bootstraps a missing volume; a typo'd name errors instead)
aloelite-fuse photos.sqlite photos /mnt/photos --create

# Encrypted volume — three ways to supply the PIN
aloelite-fuse vault.sqlite vault /mnt/vault --pin "my secret"
aloelite-fuse vault.sqlite vault /mnt/vault --pin-file ~/.vaultpin
aloelite-fuse vault.sqlite vault /mnt/vault --pin-env VAULT_PIN

# Unmount
fusermount3 -u /mnt/photos
```

The FUSE driver uses bounded-memory I/O throughout — a 15 GB copy does not buffer in RAM. Sequential writes stream one chunk at a time; random access (O_RDWR, partial overwrites, truncation) buffers only the dirty byte ranges and commits them as atomic in-place writes on fsync/release. Handlers are hardened: an unexpected error returns EIO to the caller rather than detaching the mount.

### Running applications on a mounted volume

Ordinary applications work on a mounted volume unmodified. Aloelite has
been validated backing a mail server (docker-mailserver) and a full git
workflow — clone, push, gc, packfile reads, executable hooks — directly
on a mount. SQLite databases run correctly in rollback-journal mode
(`PRAGMA journal_mode=PERSIST` or `TRUNCATE` with a `busy_timeout`; see
Troubleshooting for the recipe).

Two categories of software are not yet supported: anything that persists
state via shared-writable mmap (WAL-mode SQLite, LMDB, boltdb, Dovecot
index files), and anything that requires hard links. Both usually have a
configuration escape hatch — point the mmap-backed store at a regular
directory, or switch the journal mode — while the payload data stays on
the volume.

## Volume Manager and WebUI

The volume manager is a privileged container that manages multiple Aloelite volumes and exposes each as a FUSE-mounted subdirectory, accessible to other containers via bind mount.

### Run

```bash
# Host directories (once)
sudo mkdir -p /aloelite-root /mnt/aloelite

docker run -d --privileged \
  -v /aloelite-root:/aloelite-root \
  -v /mnt/aloelite:/mnt:rshared \
  --device /dev/fuse \
  -p 8080:8080 \
  aloecraft/aloelite-manager
```

`/aloelite-root` holds the backing SQLite files and persists across restarts. `/mnt/aloelite` is the host-visible mount root; FUSE mounts inside the container propagate here via `rshared`. `--privileged` (or at minimum `CAP_SYS_ADMIN`) is required.



### API

| Method | Path | Description |
|---|---|---|
| `POST` | `/volumes` | Create a volume |
| `GET` | `/volumes` | List all volumes |
| `DELETE` | `/volumes/<id>` | Delete a volume (unmounts first) |
| `POST` | `/volumes/<id>/mount` | Mount a volume |
| `DELETE` | `/volumes/<id>/mount` | Unmount a volume |
| `GET` | `/volumes/<id>/mount` | Mount status |
| `GET` | `/volumes/<id>/mounts` | List durable engine mounts in the volume file (`?all=1` includes retired) |
| `GET` | `/volumes/<id>/stat` | Backing file metadata (size, mtime) |
| `GET` | `/volumes/<id>/export` | Checkpoint + stream the SQLite file |
| `POST` | `/volumes/<id>/checkpoint` | Run `WAL_CHECKPOINT(TRUNCATE)` |
| `GET` | `/volumes/<id>/files?path=/` | List a directory in a mounted volume |
| `GET` | `/volumes/<id>/files/download?path=/f` | Download a file |
| `POST` | `/volumes/<id>/files/upload?path=/dir` | Upload a file (multipart field `file`) |
| `POST` | `/volumes/<id>/files/mkdir?path=/dir` | Create a directory |
| `DELETE` | `/volumes/<id>/files?path=/f` | Delete a file or directory (recursive) |
| `GET` | `/health` | Preflight results and warnings |
| `GET` | `/admin` | Admin panel: volumes + per-volume file explorer |

Mounting with `{"persist": true}` makes the mount survive container
restarts (auto-mount at startup). Encrypted volumes additionally need
`"pin_env"` or `"pin_file"` naming where the PIN is read from at each
startup — the PIN itself is never stored. An explicit unmount revokes
the persist flag.

```bash
# Create and mount
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "myphotos"}' | tee /tmp/vol.json

VID=$(jq -r .id /tmp/vol.json)
curl -s -X POST http://localhost:8080/volumes/$VID/mount \
  -H 'Content-Type: application/json' -d '{}'

# The volume is now a plain directory on the host
ls /mnt/aloelite/$VID

# Consume from another container
docker run --rm -v /mnt/aloelite/$VID:/data alpine ls /data

# Backup: poll stat, export on change
curl -s http://localhost:8080/volumes/$VID/stat | jq
curl -s http://localhost:8080/volumes/$VID/export -o snapshot.sqlite

# Encrypted volume
curl -s -X POST http://localhost:8080/volumes \
  -H 'Content-Type: application/json' \
  -d '{"name": "vault", "encrypted": true, "pin": "correct-horse"}'
# Mount with: -d '{"pin": "correct-horse"}'
```

The export endpoint runs `WAL_CHECKPOINT(TRUNCATE)` before streaming, producing a complete self-contained SQLite file with no accompanying WAL. The volume does not need to be unmounted to export — SQLite's read consistency guarantees a coherent snapshot regardless of active writes.

The admin panel at `/admin` includes a per-volume **file explorer** for any mounted volume: browse with breadcrumbs, upload, download, create folders, and delete. Each volume card also has a **Mounts** button listing the durable engine mounts inside the backing file (label, path, state), independent of any live FUSE session. The file endpoints operate through the live FUSE mountpoint, so they work identically for plain and encrypted volumes (the mount session already holds the key).

### Backup Sync Pattern

```
loop:
    poll GET /volumes/<id>/stat
    if mtime > last_known_mtime:
        GET /volumes/<id>/export  →  write to temp file  →  rename into place
        last_known_mtime = mtime
    sleep(interval)
```

The rename into place is atomic; a failed export leaves the previous replica intact.

## Security Notes

**Chunk data** is encrypted at the storage boundary (ChaCha20-Poly1305, Argon2id key derivation). The SQLite file is opaque without the PIN.

**Node metadata** (paths, timestamps, node IDs, directory structure) is stored in plaintext in the SQLite schema. An observer with access to the file can read the filesystem tree even without the PIN. For sensitive deployments, place the backing file on an encrypted volume (LUKS, encrypted home directory, etc.) or use the `pack` primitive to seal a subtree before transport.

The volume manager API is intended for trusted networks. PINs are transmitted in request bodies and never logged or persisted; the derived key is held only for the duration of the mount session.

## Design Background

The original design abstract and discussion, from which doc/requirements.md was authored.

### Abstract

This document specifies the design of a portable filesystem implemented on top of SQLite. The system models a filesystem as a small set of relational primitives (i.e. nodes, edges, volumes, and mounts) rather than as a fixed on-disk layout, deferring byte packing, page management, and durability to SQLite's mature storage engine. It is deliberately interface-agnostic: it presents a coherent internal model of files, directories, placement, and access without committing to any single external protocol, while remaining structurally amenable to exposing one (WebDAV, FUSE, or others) in the future. The design favors a hierarchical tree as its default arrangement but encodes that hierarchy as a relaxable constraint rather than a structural assumption, leaving a clear path toward a more general graph-shaped namespace. Supporting concerns (e.g. content storage, archival, and verifiable modification) are accommodated as first-class parts of the model even where their full implementation is staged for later.

### Discussion

The motivation for building on SQLite is portability and reach. A filesystem expressed as a SQLite database is a single, self-describing file that can be opened, moved, and inspected anywhere SQLite runs, which is nearly everywhere, and it inherits decades of work on storage layout and transactional integrity for free. The cost of that choice is that the filesystem's structure must be expressed relationally; the contribution of this design is a set of primitives that do so cleanly while keeping future capabilities reachable rather than precluded.

The model separates four concerns that filesystems often conflate. A *node* is an identity: a file (Entry) or a directory (Container), bearing a stable time-ordered identifier and its own name. An *edge* is a placement: a directed, immutable relationship that situates a node beneath a container within a particular volume. A *volume* is an origin: the root to which a coherent tree of placements ultimately refers. A *mount* is an access context: a durable, volume-bound access point, anchored at an explicit node, through which operation on the filesystem is brokered. Holding these four apart is what gives the design its flexibility. Because a node's name and existence are independent of where it sits, the same node can in principle be reachable from more than one place, which is the seam through which links, mounts, and an eventual graph layout enter without disturbing the core. Because placement lives in immutable edges, every structural change is expressed as the creation of a new edge rather than the mutation of an existing one, which keeps the history of where things have been available and gives later features (e.g. ordering, verification, recovery) a stable substrate to build on. Because origins are modeled explicitly rather than inferred, the boundary of a volume is a real, referenceable thing rather than a convention. And because access is brokered through mounts rather than ambient, the system has a concrete answer to a question filesystems usually answer with the operating system: who holds a handle, who holds a lock, and what to reclaim when a session ends.

File contents are held apart from node metadata, so that traversing and resolving the namespace touches only small, frequently-accessed rows and never drags large payloads along. Reading and writing a whole file is an atomic operation in the ordinary case, with a streaming, descriptor-like access path for large or incremental I/O. That access path is mediated by mounts: because the filesystem has no native notion of a process, a mount stands in as the session identity that holds open handles and locks, and locks are scoped to the mount that acquired them, so that ending a session has a well-defined effect on everything it held. This advisory locking coexists with rather than commandeers SQLite's own transactional concurrency. Archival packs a subtree into a portable serialized form within the safety of a single transaction, so that the act of consolidating data cannot lose it. And the design reserves room for cryptographic verification of modification (e.g. a Merkle structure over the tree) by ensuring that mutations flow through a single, well-defined path where such bookkeeping can later be attached. None of these later-stage capabilities is fully realized in the first iteration; the purpose of the model described here is to make each of them an addition rather than a redesign.

## License

Apache 2.0. See [LICENSE](LICENSE).
