Podcast Scraper

Download, transcribe, and summarize podcast episodes. Fetches transcripts from RSS feeds (Podcasting 2.0), generates them when missing, detects speakers, creates summaries, and optionally extracts structured insights (GIL) and knowledge graphs (KG). Use local models (Whisper, transformers) or cloud APIs (OpenAI, Gemini, Anthropic, Mistral, DeepSeek, Grok, Ollama) — your choice.

Learning project: This is a personal project where I'm exploring AI-assisted coding and hands-on work with edge and cloud AI/ML technologies.

Personal use only. Downloaded content must remain local and not be redistributed. See Legal Notice.

Features

• Transcript Downloads — Automatic detection and download from RSS feeds

• Episode selection — Order (newest / oldest), optional publish-date window (--since / --until), offset, and max_episodes for large back-catalogs (CONFIGURATION.md, GitHub #521)

• Multi-feed corpus — One config or CLI invocation for multiple shows: feeds / rss_urls in YAML, --feeds-spec for structured feeds.spec.yaml / JSON, or repeatable --rss / legacy --rss-file; isolated output under <output_dir>/feeds/<stable_id>/ per feed. With vector_search, a single parent index is built under <output_dir>/search after all feeds finish; corpus_manifest.json, corpus_run_summary.json, and structured log lines record batch status (RFC-063, CONFIGURATION.md). Inspect offline: python -m podcast_scraper.cli corpus-status --output-dir <corpus_parent>.

• Transcription — Local Whisper, DGX tailnet Whisper, OpenAI, Gemini, Mistral, or Deepgram Nova-3; automatic API audio chunking for oversized files (Audio Pipeline Guide)

• Speaker screenplay — Gap-based or neural diarization (pyannote; default on for local Whisper, --no-diarize to disable); maps to NER-detected host/guest names

• Commercial cleaning — Confidence-scored sponsor/ad detection (patterns + position; optional diarization signals when speaker timeline exists)

• Audio Preprocessing — Optimize audio files before transcription (reduce size, remove silence, normalize loudness)

• Speaker Detection — Identify speakers using spaCy NER, OpenAI, Google Gemini, Grok (real-time info), or other providers

• Summarization — Episode summaries via local transformers (BART/LED), hybrid MAP-REDUCE (hybrid_ml), or any of 7 LLM providers

• GIL Extraction — Structured insights and verbatim quotes with evidence grounding (gi.json per episode, PRD-017)

• KG Extraction — Topic-entity knowledge graphs from transcripts (kg.json per episode, RFC-055)

• Metadata Generation — JSON/YAML metadata per episode

• Resumable — Skip existing files, handle interruptions gracefully

• Provider System — Swap between local and cloud providers via config

• MPS Exclusive Mode — Serialize GPU work on Apple Silicon to prevent memory contention and improve reliability (enabled by default)

• Reproducibility — Deterministic runs with seed control, pinned model revisions, and comprehensive run manifests (Issue #379)

• Operational Hardening — Configurable HTTP retries (media vs RSS), application-level episode retries on transient errors, timeout enforcement, --fail-fast / --max-failures, structured JSON logging, and failure_summary in run.json when episodes fail (CONFIGURATION.md — Download resilience, Issue #379)

• Security — Path validation, model allowlist validation, safetensors format preference, and trust_remote_code=False enforcement (Issue #379)

• Diagnostics — doctor command for environment validation and dependency checks (Issue #379)

• Hybrid corpus search — Two-tier (transcript segments + GIL insights) BM25 + dense vector via RRF over LanceDB, with compound results; the single search path when a two-tier index exists (FAISS retired in #995, ADR-099) (RFC-090). Enabled by vector_search; search / index CLIs, and semantic gi explore --topic (guide, RFC-061)

• Run Tracking — Per-episode stage timings, run summaries, episode index files, and metrics.json fields for download retries (http_urllib3_retry_events, episode_download_retries) (Issue #379)

• Live pipeline monitor — Optional --monitor: separate process + rich dashboard (or .monitor.log if stderr is not a TTY) and .pipeline_status.json. Optional pip install -e ".[monitor]": --memray for heap profiling; with monitor + TTY, f runs py-spy to debug/flamegraph_*.svg (RFC-065, guide, #512)

• GI / KG Viewer (v2) — Optional browser UI: graph visualization, dashboard, semantic search, explore, Corpus Library (feed/episode browser, RFC-067), and Corpus Digest (time-windowed highlights, RFC-068) — all against a pipeline output folder (RFC-062; see below). Cross-surface navigation is owned by a single graph handoff FSM (ADR-094, RFC-085) with envelope contract, generation-token concurrency, stuck detection, and visible error UX.

• Generic MCP server — Optional agent-facing tool surface: expose the platform's read capabilities (hybrid search, the relational-query layer, CIL intelligence, catalog) as composable, read-only MCP tools so agentic clients (Claude Desktop/Code, Cursor) use a corpus as a tool — resolve an entity, search for grounded evidence, ask "who said what across shows", pull a person's positions or position-arc, all provenance-bearing (PRD-034, RFC-095). Two transports: stdio for local agents (podcast mcp --corpus <dir> --transport stdio) and Streamable HTTP for remote, bring-your-own-model agents over the public edge — a hosted OAuth 2.1 authorization server + personal-access tokens + the mcp_access entitlement, connectable from claude.ai / Claude Code / Cursor (RFC-112). Ships in [dev]; needs [search] for retrieval. See the Server Guide — MCP server + Player launch — MCP.

• Personal corpus + knowledge export — a read-time definition of what a user has engaged with (experienced = heard ∪ captured; saved = save-for-later), versioned by a per-user revision counter + change log (RFC-114), plus a graph-aware Obsidian vault export (wikilinked, incremental, bridge-only) via /api/app/export (RFC-113).

• Operator design records — Always-on Hetzner VPS hosting with Tailscale-only ingress (RFC-082) and version-aware corpus snapshot backup + restore (RFC-084, ADR-092); the deployment, backup and restore tooling itself is not in this repository. Shipped here: a ledgered corpus-upgrade framework (#862, CORPUS_UPGRADE, make upgrade-check / upgrade-corpus) that carries deployed corpora across release boundaries and is auto-run on prod restore (#1176), and a canonical stack contract (ADR-093) covering dev / prod with one health surface and one stack-test discipline.

• Three-tier viewer test pyramid — Tier-1 mocks, Tier-2 production-shaped fixtures, Tier-3 real corpus via make ci-ui-validation CORPUS=… (ADR-095, RFC-086). Real-bug-first matrix rule keeps coverage honest.

Repository layout (Python + web)

The Python package (CLI, pipeline, FastAPI) lives at the repo root (pyproject.toml, Makefile, tests/). The GI/KG viewer is a Node app under web/gi-kg-viewer/ (package.json, Vite, Vitest, Playwright). They share one repository but use separate install steps and different .env.example files (root vs web/gi-kg-viewer/) by design.

Onboarding in one place: Polyglot repository guide.

GI / KG Viewer (optional)

Browse Grounded Insight and Knowledge Graph artifacts, use semantic search and explore on your corpus, and view a dashboard—all against the same --output-dir you use for the pipeline.

What you need

Run the server (typical)

From the repository root, with your virtualenv active:

pip install -e ".[dev]"
cd web/gi-kg-viewer && npm install && npm run build && cd ../..
python -m podcast_scraper.cli serve --output-dir /path/to/your/run

Then open http://127.0.0.1:8000 (default port). In the sidebar, set Corpus root folder to that same directory, click List files, select .gi.json / .kg.json, and Load selected into graph.

FastAPI / HTTP API: Full /api/* reference, architecture, and tests — Server Guide. With the server running, interactive OpenAPI (Swagger) is at /docs and /openapi.json.

Development (API + hot-reload UI): make serve SERVE_OUTPUT_DIR=/path/to/your/run runs the FastAPI app and the Vite dev server together (UI proxied to the API). See web/gi-kg-viewer/README.md and the Server Guide above.

Testing the viewer: make test-ui (Vitest unit tests, ~1s) and make test-ui-e2e (Playwright browser E2E, Firefox).

Quickest start: Docker Compose

The fastest way to try the platform end-to-end — no Python venv, no Node install, no provider keys required. You get the viewer, the API, and an on-demand pipeline runner all wired together; you drive the platform from your browser.

make stack-test-build      # build api + viewer + pipeline images (~5–15 min first time)
make stack-test-up         # bring up viewer (8090) + api + bundled mock RSS fixtures

Then open http://127.0.0.1:8090, open Configuration → Feeds, add an RSS URL (or use the bundled fixture http://mock-feeds/p01_fast_with_transcript.xml), pick a profile under Job Profile (airgapped_thin runs locally with Whisper + transformers, no cloud calls), click Save, then Dashboard → Pipeline → Run pipeline job. Artifacts populate the Library, Digest, Search, and Graph tabs as the job runs.

To use cloud providers (OpenAI Whisper API + Gemini for everything else — faster + better quality), put OPENAI_API_KEY and GEMINI_API_KEY in a repo-root .env file, then:

make stack-test-build-cloud   # adds the cloud-thin pipeline-llm image
# in the UI, choose the "cloud_thin" profile under Job Profile and Save

Tear down (preserves the corpus): make stack-test-down. Tear down + drop everything: make stack-test-down STACK_TEST_DOWN_VOLUMES=1.

Full guide with architecture, troubleshooting, and production hints: Docker Compose guide.

Quick Start (Python)

For development, scripted CLI runs, or when you'd rather not run the whole stack — install the package in a venv and use the CLI directly. The Docker Compose path above is the recommended way to operate the platform; the Python path below is for development and headless CLI scripting.

Requirements

• Python 3.10+ — Check with python3 --version
• ffmpeg (only needed for local Whisper transcription):
  • macOS: brew install ffmpeg
  • Linux: apt install ffmpeg or yum install ffmpeg
  • Windows: Download from ffmpeg.org
  • Note: Not required if using OpenAI providers only
• Graphviz (only needed to regenerate architecture diagrams locally):
  • macOS: brew install graphviz
  • Linux: apt install graphviz or yum install graphviz
  • Windows: Download from graphviz.org
  • Note: Required for make visualize. Diagrams must be committed; make ci / make ci-fast and CI run visualize and fail if they are stale.

Installation Options

Choose the installation method based on your use case:

Quick decision guide:

• Using LLM APIs (OpenAI, Gemini, etc.)? → pip install -e ".[llm]"
• Want to run models locally (Whisper, spaCy, Transformers)? → pip install -e ".[ml]"
• Want both? → pip install -e ".[ml,llm]" (recommended)
• Contributing / developing? → pip install -e ".[dev,ml,llm]"

Note: LLM provider SDKs (openai, google-genai, anthropic, mistralai, httpx) are not in core — they require the [llm] extra. Core (pip install -e .) gives you the pipeline framework only.

vector_search and embedding cache

Corpus indexing (vector_search, LanceDB index) calls the embedder with allow_download=False, so sentence-transformers weights must already be on disk in the Hugging Face hub cache. Before offline or sandbox runs, run make preload-ml-models and omit SKIP_GIL=1 so evidence embeddings (including the model named by vector_embedding_model) are pre-cached; GIL’s gi_embedding_model is checked separately when GIL uses transformers. If you set HF_HUB_CACHE, use the same value when preloading and when running the pipeline. See Semantic Search Guide.

Install

Tip: For end users who just want to run the tool, consider using pipx or uv for easier installation. See the Installation Guide for all methods.

Method 1: pip (Standard - Recommended for Development)

Use the latest released version for normal usage.

For LLM API users (OpenAI, Gemini, etc. — no local ML):

# Clone the repository
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper
git checkout <latest-release-tag>   # e.g. v2.6.0

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Verify Python version (must be 3.10+)
python --version  # Should show Python 3.10.x or higher

# CRITICAL: Upgrade pip and setuptools before installing
pip install --upgrade pip setuptools wheel

# Install with LLM provider SDKs (OpenAI, Gemini, Anthropic, Mistral, etc.)
pip install -e ".[llm]"

For local ML users (Whisper, spaCy, Transformers):

# Same setup as above, then:
# Install package with ML dependencies (includes Whisper, spaCy, Transformers)
pip install -e ".[ml]"

Note: LLM provider SDKs require the [llm] extra. For both local ML and LLM APIs: pip install -e ".[ml,llm]".

Method 2: pipx (Recommended for End Users)

For isolated installation without managing virtual environments:

For LLM-only users:

# Install pipx (if not already installed)
# macOS: brew install pipx
# Linux: pip install --user pipx && pipx ensurepath

# Clone and install with LLM SDKs
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper
pipx install -e ".[llm]"

# Verify
python -m podcast_scraper.cli --help

For local ML users:

# Same as above, but install with ML dependencies:
pipx install -e ".[ml]"

See [Installation Guide](https://github.com/chipi/podcast_scraper/blob/HEAD/docs/guides/INSTALLATION_GUIDE.md#method-2-pipx-isolated-installation---r