Metadata-Version: 2.4
Name: secrets-sync
Version: 0.8.1
Summary: Commandline tool to pull secrets from multiple sources and push to AWS SSM and Secrets Manager with rate limiting.
Author: Josh.5
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.41
Requires-Dist: PyYAML>=6.0
Requires-Dist: keepercommander>=17.1
Requires-Dist: Jinja2>=3.1
Requires-Dist: infisicalsdk>=1.0.3
Requires-Dist: requests>=2.31
Dynamic: license-file

# Secrets Sync

## Description

Async CLI to pull secrets from:

- environment variables
- YAML files
- Infisical
- 1Password vaults
- Keeper folders

and push them to:

- AWS SSM Parameter Store
- AWS Secrets Manager
- Infisical
- dotenv files

## Install

Requires Python 3.10+.

Install from PyPI:

```bash
pip install secrets-sync
```

## Develop

To setup a local dev env, use [`uv`](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer)

```bash
uv venv && uv pip install --editable .

source .venv/bin/activate
```

or just run with `uv`

```bash
uv run secrets-sync --dry-run --print-values --print-format=table -f my-config.yaml

uv run secrets-sync --print-sync-details -f my-config.yaml
```

To bump the package version, use `uv version`:

```bash
uv version --bump patch
uv version --bump minor
uv version --bump major
```

Or set an explicit version:

```bash
uv version 0.7.0
```

After changing dependencies in `pyproject.toml`, refresh `uv.lock`:

```bash
uv lock
```

## Usage

Merge multiple YAML config files (later files override earlier values):

```
secrets-sync -f ./defaults.yml -f ./test-1.yml
```

Flags:

- `--file, -f PATH`: add a config file to merge (may be repeated; later overrides earlier).
- `--print-values`: print a preview of what will be pushed, grouped by sink. Combine with `--dry-run` for preview only.
- `--print-format {list,table,json}`: output format for preview (default `list`).
- `--dry-run`: collect and optionally print, but do not push to remote sinks.
- `--print-sync-details`: print a line for each item as it's synced (success/failure plus created/unchanged/changed). When combined with `--print-values`, each log also shows value snapshots (`created 'new'`, `unchanged 'old'`, or `changed 'old' -> 'new'`).

## Config

- `vars`: key/value map. Values here override environment variables during template interpolation. Placeholders `{{ VAR_NAME }}` in strings are replaced at load time. Missing variables cause an error.
- `aws`: configure which AWS region/profile the CLI should use for AWS API calls (same as the AWS CLI/boto3 provider chain):
  - `region`: optional explicit region. If omitted, the CLI falls back to `AWS_DEFAULT_REGION` or `AWS_REGION`.
  - `profile`: optional profile name from your `~/.aws/config`/`~/.aws/credentials`. If omitted, `AWS_PROFILE` (when set) is used, otherwise boto3 falls back to its default profile chain.
  - Regardless of profile, boto3 can still authenticate with exported credentials such as `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`.
- `sources`: list of sources, each with `name`, `type`, and `options` (See the [Sources](#sources) section below for more details).
- `sinks`: list of sinks, each with `type`, `options`, and optional `sources` filter listing source names to route (See the [Sinks](#sinks) section below for more details).

Example: [examples/basic/dev.yaml](examples/basic/dev.yaml).

### Sources

- `env`: Reads directly from the running process environment. Use this source to capture secrets already loaded into the shell or CI job. See [docs/SOURCE_ENV.md](docs/SOURCE_ENV.md) for scenarios and examples.
  - `include_regex`: regex or list of regexes to include
  - `exclude_regex`: regex or list of regexes to exclude
  - `keys`: explicit variable names to include
  - `strip_prefix`, `strip_suffix`: remove leading/trailing text from emitted names. Each accepts a string or list.

- `yaml`: Loads values from one or more YAML documents on disk. Files are merged in order so you can provide layered defaults plus environment overrides. Additional details live in [docs/SOURCE_YAML.md](docs/SOURCE_YAML.md).
  - `files`: list of YAML file paths (merged in order; later files override earlier)
  - `key`: dot-path to the subtree to read (e.g., `values` for the example shape)
  - Supported structures: mapping of `name: value`, or `{ values: [ { name, value, description } ] }`, or a list of `{ name, value, description }`.
  - Relative paths are resolved against the config file where they are declared (not the working directory). This also holds when merging multiple config files.

- `1password`: Fetches items from a 1Password vault and maps each item title to a secret. Requires the 1Password `op` CLI to be installed plus either a configured `service_account_token` or the `OP_SERVICE_ACCOUNT_TOKEN` environment variable for authentication. Full walkthrough: [docs/SOURCE_1PASSWORD.md](docs/SOURCE_1PASSWORD.md).
  - `vault`: Vault name (required).
  - `tag_filters`: Only items containing any of these tags are included. The list order also determines override priority when multiple items share the same title.
  - `include_regex`, `exclude_regex`: Optional regex or list of regexes applied to item titles.
  - `strip_prefix`, `strip_suffix`: Optional emitted-name transforms. Each accepts a string or list.
  - `service_account_token`: Inline token value; falls back to the `OP_SERVICE_ACCOUNT_TOKEN` environment variable when omitted.
  - `concurrency`: Number of parallel fetches when pulling item details (default `8`).

- `infisical`: Reads secrets from an Infisical project, environment, and folder path. Detailed setup, auth, slug lookup, `secret_path`, and recursive read behavior are documented in [docs/SOURCE_INFISICAL.md](docs/SOURCE_INFISICAL.md).
  - `host`: optional Infisical base URL. Defaults to `INFISICAL_HOST` or `https://app.infisical.com`.
  - `project_id` or `project_slug`: source project. `project_id` takes precedence if both are set.
  - `environment_slug`: required environment slug such as `dev`, `staging`, or `prod`.
  - `secret_path`: source folder path in Infisical (default `/`).
  - `auth_method`: optional `token` or `universal_auth`.
  - Authentication is environment-only via `INFISICAL_TOKEN` or `INFISICAL_CLIENT_ID` / `INFISICAL_CLIENT_SECRET`.
  - `rate_limit_rps`, `concurrency`: control throughput.
  - `recursive`, `include_imports`, `expand_secret_references`: control how Infisical returns secrets.
  - `include_regex`, `exclude_regex`: filter emitted secret names. Each accepts a string or list.
  - `strip_prefix`, `strip_suffix`: transform emitted secret names. Each accepts a string or list.
  - `tag_filters`: Infisical API-side filtering.

- `keeper`: Uses the Keeper Commander SDK/CLI session to pull records from Keeper Enterprise. Requires a logged-in Keeper Commander environment with persistent login or inline credentials. Reference guide: [docs/SOURCE_KEEPER.md](docs/SOURCE_KEEPER.md).
  - `folder`: Keeper folder or path to read from (required).
  - `tag_filters`: Only records whose custom `tags` field matches any supplied tag are included. The list order also determines override priority when multiple items share the same title.
  - `include_regex`, `exclude_regex`: Optional regex or list of regexes applied to record titles.
  - `strip_prefix`, `strip_suffix`: Optional emitted-name transforms. Each accepts a string or list.
  - `config_file`: Path to the Keeper Commander config (default `~/.keeper/config.json`).
  - `keeper_server`, `keeper_user`, `keeper_password`: Inline overrides (or `KEEPER_SERVER`, `KEEPER_USER`, `KEEPER_PASSWORD` env vars) for CLI login values. Overrides what is read from `config_file`.

### Sinks

- `ssm` options:
  - `prefix`: optional string prefix for parameter names (supports `{{ VAR }}` placeholders)
  - `type`: `SecureString` (default) or `String` (any other value errors at load time)
  - `tier`: `Standard` (default) or `Advanced`. Values over 4 KB (measured after UTF-8 encoding) are automatically promoted to the Advanced tier with a warning so large file-style secrets can be stored without changing the source config. Note: Values over 8 KB will fail with an error.
  - `overwrite`: boolean (default true)
  - `kms_key_id`: optional KMS key id for SecureString
  - `rate_limit_rps`, `concurrency`: control throughput

- `secrets_manager` options:
  - `prefix`: optional string prefix for secret names (supports `{{ VAR }}`)
  - `kms_key_id`, `rate_limit_rps`, `concurrency` similar to SSM

- `infisical`: Writes to an Infisical project, environment, and folder path. Detailed setup, auth, slug lookup, and `secret_path` behavior are documented in [docs/SINK_INFISICAL.md](docs/SINK_INFISICAL.md).
  - `host`: optional Infisical base URL. Defaults to `INFISICAL_HOST` or `https://app.infisical.com`.
  - `project_id` or `project_slug`: target project. `project_id` takes precedence if both are set.
  - `environment_slug`: required environment slug such as `dev`, `staging`, or `prod`. Missing environments are created automatically.
  - `secret_path`: target folder path in Infisical (default `/`). Missing folders are created automatically.
  - `name_prefix`: optional prefix prepended to each secret key before writing.
  - `auth_method`: optional `token` or `universal_auth`.
  - Authentication is environment-only via `INFISICAL_TOKEN` or `INFISICAL_CLIENT_ID` / `INFISICAL_CLIENT_SECRET`.
  - `rate_limit_rps`, `concurrency`: control throughput.

- `dotenv`: Writes the selected items into a local dotenv file. Detailed behavior and examples are in [docs/SINK_DOTENV.md](docs/SINK_DOTENV.md).
  - `path`: required target dotenv file path. Relative paths are resolved against the config file where they are declared.
  - `mode`: optional `merge` (default) or `replace`. `merge` updates matching keys and appends missing ones while preserving unrelated existing entries. `replace` rewrites the file from only the selected sink items.
  - `key_case`: optional `preserve` (default), `upper`, or `lower`.
  - `strip_prefix`, `strip_suffix`: optional transform rules removed from the start/end of each secret name before writing. Each accepts a string or list.
  - `include_regex`, `exclude_regex`: optional sink-side regex filters on secret names before writing. Each accepts a string or list.
  - In `merge` mode, the sink behaves like the remote sinks: it updates matching keys and creates missing ones without deleting unrelated dotenv entries.

- `dir_files`: Writes selected items to individual files in a local directory. Detailed behavior and examples are in [docs/SINK_DIR_FILES.md](docs/SINK_DIR_FILES.md).
  - `path`: required target directory. Relative paths are resolved against the config file where they are declared.
  - `include_regex`, `exclude_regex`: optional sink-side regex filters on secret names before writing. Each accepts a string or list.
  - `strip_prefix`, `strip_suffix`: optional emitted-name transforms. Each accepts a string or list.

The AWS API usage for both AWS sinks are paced automatically: the sinks meter requests so they stay within the configured `rate_limit_rps`, and they fall back to exponential backoff with jitter whenever AWS responds with throttling errors.

Each sink may specify `sources: [source-name, ...]` to only accept items from those sources. If a sink references a source that does not exist, config loading fails with a clear error.

### Variables and templating

- `vars` provides values for `{{ VAR }}` placeholders anywhere in the config. Values in `vars` override environment variables with the same keys.
- If a placeholder cannot be resolved, config loading fails.

### Preview output

- `--print-values --print-format=list` (default): prints `full_name=value` under each sink header.
- `--print-format=table`: prints two columns (Name, Value) per sink.
- `--print-format=json`: prints a JSON array of sink objects with `name`, `type`, `prefix`, `sources`, and `items[]` (each with `name`, `value`, `description`).

Examples:

```
secrets-sync --dry-run --print-values -f ./defaults.yaml -f ./env.yaml
secrets-sync --dry-run --print-values --print-format=table -f ./examples/basic/dev.yaml
secrets-sync --dry-run --print-values --print-format=json -f ./examples/basic/dev.yaml
```

### Example config

```
vars:
  ENVIRONMENT_NAME: test-1

aws:
  region: ap-southeast-2

sources:
  - name: env
    type: env
    options:
      include_regex: '^APP_.*'
      strip_prefix: 'APP_'
  - name: external-yaml-file
    type: yaml
    options:
      files:
        - configs/default.yaml
        - configs/test-1.yaml
      key: values
  - name: 1password
    type: 1password
    options:
      vault: 'EnvironmentSecrets'
      include_regex: '^APP_.*'
      tag_filters: ['default','prod']
  - name: infisical
    type: infisical
    options:
      host: 'https://infisical.example.internal'
      project_id: 'd9754256-c41g-af56-45a9-23f08a936f33'
      environment_slug: 'dev'
      secret_path: '/config'
      auth_method: token
      include_regex: '^APP_.*'
      strip_prefix: 'APP_'

sinks:
  - name: ssm-secrets
    type: ssm
    options:
      prefix: '/env/{{ ENVIRONMENT_NAME }}/secret/'
      overwrite: true
      type: SecureString
      rate_limit_rps: 10
      concurrency: 10
    sources: [ '1password' ]
  - name: ssm-config
    type: ssm
    options:
      prefix: '/env/{{ ENVIRONMENT_NAME }}/config/'
      overwrite: true
      type: String
    sources: [ 'external-yaml-file', 'env' ]
  - name: secrets-manager
    type: secrets_manager
    options:
      prefix: 'env/{{ ENVIRONMENT_NAME }}/secret/'
    sources: [ '1password' ]
  - name: infisical-config
    type: infisical
    options:
      host: 'https://infisical.example.internal'
      project_slug: 'streaming-tech'
      environment_slug: 'dev'
      secret_path: '/config'
      auth_method: token
    sources: [ 'external-yaml-file', 'env' ]
  - name: infisical-secrets
    type: infisical
    options:
      host: 'https://infisical.example.internal'
      project_id: 'project-id'
      environment_slug: 'dev'
      secret_path: '/secrets'
      auth_method: universal_auth
    sources: [ '1password' ]
  - name: app-dotenv
    type: dotenv
    options:
      path: './out/.env'
      key_case: upper
      strip_prefix: 'APP_'
      exclude_regex:
        - '\\.[A-Za-z0-9]+$'
    sources: [ 'infisical' ]
  - name: app-file-secrets
    type: dir_files
    options:
      path: './out/secrets.d'
      include_regex: '\\.[A-Za-z0-9]+$'
    sources: [ 'infisical' ]
```

### Requirements

- Python 3.10+
- AWS credentials/auth per your environment (respects `AWS_PROFILE`, `AWS_DEFAULT_REGION`/`AWS_REGION`).
- 1Password source requires the `op` CLI with a service account token (via `OP_SERVICE_ACCOUNT_TOKEN` or `options.service_account_token`).
- Infisical source and sink require the `infisicalsdk` Python package (installed with this tool) plus either `INFISICAL_TOKEN` or the pair `INFISICAL_CLIENT_ID` / `INFISICAL_CLIENT_SECRET`.
- Keeper source requires the Keeper Commander CLI config (`~/.keeper/config.json`) and the `keepercommander` Python package (installed with this tool). The Keeper CLI credentials can be overridden with `options.keeper_*` or `KEEPER_*` environment variables.

### Notes

- Lists of dicts with `name` fields are deep-merged by name across config files (later files override earlier entries). Other lists are replaced.
- YAML source `files` are resolved relative to the config file they are declared in.
- Dotenv sink `path` values are resolved relative to the config file they are declared in.
- YAML source values can call `{{ lookup('file', 'relative/path') }}` to inline file contents. Lookup templates receive the merged config `vars` plus environment variables, and relative paths are evaluated from the YAML file that declares the secret. You can chain Ansible-style filters such as `| from_json | to_json` to parse and re-emit structured data.
