Skip to content
Uvilo OS
archived Visibility internal Owner erik@uvilo.com Approver _ Created _ Updated _

Typesense Plan — Phase 1: Deploy Typesense + Repo Indexer

This plan covers deploying the Typesense instance on Railway and building the repo-wide indexer. Phase 1 is the foundation — all other phases depend on a working Typesense instance with indexed repo content.

Prerequisites

  • Railway project access (already have)
  • OpenAI API key with access to text-embedding-3-small
  • Typesense admin key (generated during setup)

Tasks

  • Create Typesense Railway service

    • Deploy typesense/typesense:latest Docker image
    • Set custom start command: --data-dir /data --api-port 8108 --api-key $TYPESENSE_ADMIN_KEY
    • Add persistent volume at /data (5GB)
    • Set environment variables: TYPESENSE_ADMIN_KEY, OPENAI_API_KEY
    • Enable internal networking (no public domain)
    • Verify health: curl http://typesense.railway.internal:8108/health

  • Create the uvilo collection

    • Write a script (scripts/create-collection.ts) that sends the schema to Typesense
    • Schema includes: id, title, summary, content, path, department (facet), project (facet), type (facet), source (facet), status (facet), visibility (facet), owner (facet), embedding (float[] with embed config for openai/text-embedding-3-small, from title + summary + content + project + status)
    • Run the script against the Railway Typesense instance
    • Verify collection exists via Typesense API

  • Build the repo indexer (scripts/index-repo.ts)

    • Walk repo tree, find all .md files
    • Parse YAML frontmatter (title, status, owner, visibility)
    • Extract body text (strip frontmatter, keep code fences as plain text)
    • Derive department from root-level folder name
    • Derive type from path heuristics (knowledge, skill, spec, plan, state, project)
    • Derive project from path — folder name inside Projects/ (e.g., Forge/Projects/TypeSense/ → TypeSense), empty string for non-project files
    • Extract status, visibility, owner from frontmatter (default to empty string if absent)
    • Generate AI summary: call gpt-4o-mini with file content, prompt for ≤100-word summary. One summary per file — chunks inherit the parent file’s summary.
    • Set source=repo on all documents
    • Chunk files >800 words into overlapping segments (~400 words, ~50-word overlap)
    • Generate unique IDs: relative path for whole files, path#chunk-N for chunks
    • Upsert into Typesense via POST /collections/uvilo/documents/import
    • Support --incremental flag: git diff --name-only HEAD~1 to find changed files, only re-index those
    • Add npm run index:repo script to package.json

  • Test the indexer

    • Run full index against the repo
    • Verify document count matches expected file count
    • Test search via Typesense API: keyword query, semantic query, hybrid query
    • Test facet filtering: by department, project, type, source, status, visibility, owner

  • Add sidebar entries for Spec and Plan pages

    • Add Typesense_Spec and Typesense_Plan_Phase_1 to .internal/astro.config.mjs sidebar

  • Update project INDEX and README

    • Add new files to INDEX.md
    • Update README.md folder structure

  • Commit and push to dev

Verification

  • curl http://typesense.railway.internal:8108/health returns {"ok":true}
  • GET /collections/uvilo returns the schema
  • Full index completes without errors
  • Search query memory architecture returns Forge/Knowledge/Memory_Architecture.md as a top result
  • Incremental re-index processes only changed files