Skip to content
Uvilo OS
archived Visibility internal Owner erik@uvilo.com Approver _ Created _ Updated _

Typesense Plan — Phase 3: Starlight Integration + Website Scraper

This phase replaces Pagefind with Typesense-powered search on the documentation website. It includes the website scraper (local build), the Starlight plugin integration, and the Pagefind removal.

Prerequisites

  • Phase 1 complete: Typesense running, uvilo collection populated with repo content
  • Phase 2 complete: MCP server working
  • Search-only API key available

Tasks

  • Build the website scraper (.internal/scrape-site.py)

    • Run npm run build in .internal/ to produce static site at .internal/dist/
    • Walk .internal/dist/ for all .html files
    • Extract content from <main data-pagefind-body> tags, strip HTML tags
    • Derive title from <title> element
    • Derive path from URL path
    • Derive department from first path segment
    • Set source=website, type=page, visibility=public
    • Upsert into Typesense collection uvilo with website: ID prefix
    • Add npm run index:site and npm run index:site:dry-run scripts

  • Test the website scraper

    • Dry-run: 312 HTML files → 311 docs (1 skipped: too short)
    • Full scrape: 311 docs upserted in 8 batches (0 failures)
    • Total collection: 814 docs (503 repo + 311 website)
    • Search with filter_by=source:=website returns website pages ✓

  • Install Starlight Typesense plugin

    • npm install starlight-docsearch-typesense (v1.0.1)
    • Add plugin to astro.config.mjs plugins array
    • Configure with Typesense host, search-only key, collection name
    • Set filter_by: "source:=website" to only show published pages in website search

  • Handle Typesense accessibility from browser

    • Typesense Railway service is publicly accessible with CORS headers
    • No Vercel proxy needed — direct connection works

  • Fix Search component override conflict

    • starlight-theme-nova overrides Search with its Pagefind component
    • Fixed by explicitly setting Search: 'starlight-docsearch-typesense/Search.astro' in components config
    • DocSearch component (sl-doc-search) now renders correctly

  • Remove Pagefind

    • Set pagefind: false in starlight config
    • Pagefind no longer runs during build (saves ~5s, ~2MB)
    • No Pagefind UI references in built HTML
    • Site builds cleanly without Pagefind

  • Add sidebar entry for Phase 3 Plan page

    • Add to .internal/astro.config.mjs

  • Verify Typesense search works on production website

    • Deploy to Vercel and test in browser
    • Test keyword queries return results
    • Test typo tolerance
    • Test mobile responsiveness
    • Test Ctrl+K/Cmd+K shortcut

Verification

  • Website scraper indexes all published pages without errors
  • source=website filter returns only published pages
  • Starlight search UI appears (DocSearch component renders in built HTML)
  • Typo-tolerant search works (API verified: “tipesnse” → finds “typesense”)
  • Pagefind is fully removed
  • Site builds and deploys cleanly without Pagefind
  • Search returns results in browser (needs manual browser test)