ArchiveBoxDownloader CLI

⬇️ abx-dl

A simple all-in-one CLI tool to auto-detect and download everything available from a URL.

uvx abx-dl 'https://example.com'

✨ Ever wish you could yt-dlp, gallery-dl, wget, curl, puppeteer, etc. all in one command?

abx-dl is an all-in-one CLI tool for downloading URLs "by any means necessary".

It's useful for scraping, downloading, OSINT, digital preservation, and more. abx-dl provides a simpler one-shot CLI interface to the ArchiveBox plugin ecosystem.

Screenshot 2026-03-11 at 6 53 03 AM


🍜 What does it save?

abx-dl --plugins=wget,title,screenshot,pdf,readability 'https://example.com'

abx-dl runs all plugins by default (and auto installs dependencies). You can specify --plugins=wget,favicon,title or filters like --output=html,pdf,ico,text/ to limit plugin selection. - HTML, JS, CSS, images, etc. rendered with a headless browser - title, favicon, headers, outlinks, and other metadata - audio, video, subtitles, playlists, comments - snapshot of the page as a PDF, screenshot, and Singlefile HTML - article text, git source code - and much more...


🧩 How does it work?

abx-dl uses the Plugin Library (shared with ArchiveBox) to run a collection of downloading and scraping tools.

Plugins are loaded from the installed abx-plugins package (or from ABX_PLUGINS_DIR if you override it) and execute in distinct phases: 1. Install phase runner reads plugins config.json: required_binaries and emits BinaryRequestEvents for abxpkg.binary_service.BinaryService, which resolves or installs binaries using built-in providers such as env, pip, npm, brew, apt, cargo, and browser-specific providers. abxpkg owns the persistent binary cache; abx-dl only projects resolved paths into the current run's in-memory config. 2. CrawlSetup hooks (on_CrawlSetup__*) launch/configure expensive crawl-scoped processes like chrome, or trigger side effects. background hooks use their first stdout line as the readiness boundary and emit no stdout JSONL records. 3. Snapshot hooks (on_Snapshot__*) run per URL to extract content. background hooks use their first stdout line as the readiness boundary; JSONL records after that are ArchiveResult, Snapshot, and Tag.

Applications embedding the runtime use the same framework-free interfaces as the CLI: PluginCatalog for inventory, PluginConfigResolver for config, the service classes for explicit listener composition, typed events for phase dispatch, execute_hook() for a single finite hook, and OutputManifest for output metadata. These APIs accept plain mappings, filesystem paths, environment variables, and CLI arguments; they do not depend on Django or an application database.

Standalone download() attaches both CrawlService (plugin crawl hooks) and CrawlLifecycleService (phase sequencing). Embedders can attach only the listener suites whose behavior they want and dispatch the corresponding typed events directly.

parse_input(source_text, catalog, output_dir) is the framework-free import path for pasted text and bookmark/feed exports. It writes staticfile/stdin.txt, runs only plugins declaring x-accepts-internal-input, and returns metadata-preserving Snapshot facts at depth zero. It does not create a crawl, database row, or synthetic URL.


⚙️ Configuration

Configuration is handled via environment variables plus a user config file under the platformdirs user config path (<user-config>/abx/config.env). Resolved binary paths are projected into the current run in memory; persistent binary state lives only in the abxpkg library cache:

abx-dl config                        # show all config (global + per-plugin)
abx-dl config --get WGET_TIMEOUT     # get a specific value
abx-dl config --set TIMEOUT=120      # set persistently (resolves aliases)

Output is grouped by section:

# GLOBAL
TIMEOUT=60
USER_AGENT="Mozilla/5.0 ..."
...

# plugins/wget
WGET_BINARY="wget"
WGET_TIMEOUT=60
...

# plugins/chrome
CHROME_BINARY="chromium"
...

Common options: - TIMEOUT=60 - default timeout for hooks - USER_AGENT - default user agent string - {PLUGIN}_BINARY - path or name of the binary to use (e.g. WGET_BINARY=wget or CHROME_BINARY=/usr/bin/chromium) - {PLUGIN}_ENABLED=True/False - enable/disable specific plugins - {PLUGIN}_TIMEOUT=120 - per-plugin timeout overrides

Aliases are automatically resolved (e.g. --set USE_WGET=false saves as WGET_ENABLED=false).

One-off config is easy via env vars or CLI args:

env \
  TIMEOUT=120 \
  WGET_TIMEOUT=120 \
  abx-dl \
    --dir=./config-example \
    --plugins=title,wget \
    --timeout=90 \
    'https://example.com'



📦 Install

uv tool install abx-dl
abx-dl version

uvx abx-dl version

abx-dl install wget title


🔠 Usage

abx-dl --plugins=title,wget --dir=./downloads --timeout=120 'https://example.com'

# Default command - a bare URL archives with all enabled plugins:
abx-dl 'https://example.com'

# Select plugins by output type (mimetypes, categories, or file extensions):
abx-dl --output=html,pdf,video/ 'https://example.com'
abx-dl -o text -o image -o mp4 'https://example.com'

# Limit work to a subset of plugins by name:
abx-dl --plugins=wget,title,screenshot,pdf 'https://example.com'

# Skip auto-installing missing dependencies (emit warnings instead):
abx-dl --no-install 'https://example.com'

# Specify output directory (default is current working dir):
abx-dl --dir=./downloads 'https://example.com'

# Set timeout:
abx-dl --timeout=120 'https://example.com'

Commands

abx-dl <url>                              # Download URL (default shorthand)
abx-dl plugins                            # Check + show info for all plugins
abx-dl plugins wget ytdlp git             # Check + show info for specific plugins
abx-dl install wget ytdlp git             # Pre-install plugin dependencies
abx-dl config                             # Show all config values
abx-dl config --get TIMEOUT               # Get a specific config value
abx-dl config --set TIMEOUT=120           # Set a config value persistently

Installing Dependencies

Many plugins require external binaries (e.g., wget, chrome, yt-dlp, single-file).

By default, abx-dl lazily installs missing dependencies as needed when you download a URL. Use --no-install to skip plugins with missing dependencies instead. install runs only the pre-run dependency pipeline (required_binaries → BinaryRequestEvent → BinaryEvent) without starting crawl setup or snapshot extraction:

abx-dl install wget title
abx-dl plugins wget title

abx-dl 'https://example.com'              # checks and installs missing deps before hooks run
abx-dl --no-install 'https://example.com' # skips plugins with missing deps and emits warnings
abx-dl install wget singlefile ytdlp      # installs dependencies for specific plugins only
abx-dl plugins                            # checks which dependencies are available/missing

Every preflight request is resolved through abxpkg. Compatible host binaries are selected first and projected into ABXPKG_LIB_DIR/env/bin; otherwise the configured managed provider installs and projects the dependency. Hook subprocesses then use the resolved Python or Node interpreter and projected runtime environment directly.

The normal runtime flow after dependency preflight is: - CrawlEvent (internal lifecycle root) - CrawlSetupEvent → plugin on_CrawlSetup__* hooks - CrawlStartEvent → SnapshotEvent - SnapshotEvent → plugin on_Snapshot__* hooks - SnapshotCleanupEvent / CrawlCleanupEvent

Hook output contract: - EXTRA_CONTEXT is opaque correlation data reflected into output records only. Hooks must never inspect it. Snapshot hooks receive --url, --snapshot-id, and --depth as explicit arguments; archived titles/tags stay in filesystem data. Embedders provide download(snapshot=Snapshot(...)) for an existing snapshot, not identity or depth hidden in configuration. - binary preflight is driven by plugin required_binaries and handled by abxpkg, not by plugin hooks - on_CrawlSetup__* background hooks emit a first stdout readiness line, but no stdout JSONL records - on_Snapshot__* background hooks emit a first stdout readiness line; hook JSONL records after that are only ArchiveResult, Snapshot, and Tag - the TUI and services consume structured events derived from those hook records

Dependencies are installed to <user-config>/abx/lib/{arch}/ using the appropriate package manager: - pip packages → <user-config>/abx/lib/{arch}/pip/venv/ - npm packages → <user-config>/abx/lib/{arch}/npm/ - brew/apt packages → system locations

You can override the install location with ABXPKG_LIB_DIR=/path/to/lib abx-dl install wget.



Output Structure

By default, abx-dl writes results into the current working directory. Each run creates an index.jsonl manifest plus one subdirectory per plugin that produced output. If you want to keep runs isolated, cd into a scratch directory first or pass --dir=/path/to/run.

mkdir -p /tmp/abx-run && cd /tmp/abx-run
uvx --from abx-dl abx-dl --plugins=title,wget 'https://example.com'

./
├── index.jsonl             # Snapshot metadata and results (JSONL format)
├── title/
│   └── title.txt
├── favicon/
│   └── favicon.ico
├── screenshot/
│   └── screenshot.png
├── pdf/
│   └── output.pdf
├── dom/
│   └── output.html
├── wget/
│   └── example.com/
│       └── index.html
├── singlefile/
│   └── output.html
└── ...


All Outputs

Available Plugins

See the abx-plugins marketplace.

Snapshot / Extraction Plugins

AI Skill

This repo includes an abx-dl skill for coding agents that need to run the standalone ArchiveBox extractor pipeline without a full ArchiveBox install.

Architecture

abx-dl is built on these components:

For more advanced use with collections, parallel downloading, a Web UI + REST API, etc. See: ArchiveBox/ArchiveBox