Skip to content
Uvilo OS
draft Visibility internal Owner erik@uvilo.com Approver _ Created _ Updated _

TypeSense Index Research

1. GitHub Action Path Filtering for DocSearch Scraper

R1: Refine GitHub Action path filtering

1.1 Finding

The index-search.yml workflow runs the Typesense DocSearch scraper against the deployed website. It currently triggers on **/*.md, .internal/docsearch.config.json, .internal/src/**, and the workflow file itself. The **/*.md glob is too broad — it triggers on every markdown change, including project State files, Eval reports, and WIP files that never appear on the website.

The website is built from repo content via Vercel, so only .md files that are part of the built site matter. The current site build uses .internal/src/ templates and the VitePress config to determine which pages are generated.

GitHub Actions path filters support glob patterns but not negation. To narrow triggers, we need to specify only the paths that affect the website build.

1.2 Options

**A. Use department-level path prefixes instead of **/\*.md**

Replace **/*.md with explicit department paths that are deployed to the website (e.g., Uvilo/**/*.md, Forge/Knowledge/**/*.md). Non-website departments (like project subdirectories) would be excluded.

  • Pros: Simple, declarative, no code changes
  • Cons: Must be maintained when new website-visible paths are added; may miss some paths

B. Add a path-ignore filter using paths-ignore

Use paths-ignore to exclude known non-website patterns (e.g., **/Projects/**, **/*_State.md, **/*_Eval.md).

  • Pros: Explicit exclusions; catches most noise
  • Cons: paths-ignore and paths are mutually exclusive in GitHub Actions; cannot use both together

C. Use a custom trigger condition in a job step

Trigger on **/*.md still, but add an early job step that diffs changed files against a known set of website paths and exits successfully (skipping the scraper) if no relevant files changed.

  • Pros: Most precise; can use any logic
  • Cons: Adds complexity; the workflow still “runs” (consuming a runner) even if it exits early

1.3 Recommendation

Option A — replace **/*.md with explicit website-visible path prefixes. The website is built from a known set of content directories. This is the simplest approach that avoids false triggers. We should verify which paths are actually part of the VitePress build.

1.4 Decision

Option C — trigger on **/*.md (no path narrowing), but only index the .md files that changed since the last push. The filtering logic moves from the workflow trigger to the indexer itself, which diffs against the last push to determine changed files and only processes those. This simplifies the trigger config and makes the indexer incremental by default.

2. Concurrent OpenAI API Calls in the Indexer

R2: Concurrent OpenAI API calls in index-department.py (will be rewritten in TypeScript per R4)

2.1 Finding

The current index-department.py processes files sequentially: for each file, it makes an openai_summary() call, then for each chunk it makes an openai_embedding() call. Both are HTTP requests with 60-second timeouts. There is a manual rate-limit pause of 0.5s every 3 files.

In TypeScript, we can use Promise.all with a concurrency limiter (e.g., p-limit or a custom semaphore) to run multiple API calls in parallel. OpenAI’s API supports concurrent requests within rate limits (Tier 1: 500 RPM for chat, 3000 RPM for embeddings).

Key constraints:

  • Must respect OpenAI rate limits (concurrency should be configurable)
  • Summary and embedding calls for the same file can run in parallel (they’re independent)
  • Multiple files can be processed in parallel
  • Batch upsert to Typesense should still batch by BATCH_SIZE (40 docs)

2.2 Options

A. p-limit npm package

Simple concurrency limiter: const limit = pLimit(5); const results = await Promise.all(files.map(f => limit(() => processFile(f))));

  • Pros: Minimal dependency, well-known, clean API
  • Cons: Adds an external dependency

B. Custom semaphore using native APIs

Implement a simple concurrency limiter using a queue and Promise resolution.

  • Pros: No external dependency
  • Cons: More code to maintain, re-implements what p-limit does

C. Use OpenAI SDK with built-in batching

The OpenAI Node.js SDK supports batch requests. Could batch multiple summary/embedding calls into a single API request using the Batch API.

  • Pros: Fewer HTTP connections, potentially more efficient
  • Cons: Batch API has different latency characteristics (responses may be delayed); more complex to implement; overkill for our scale

2.3 Recommendation

Option A (p-limit). It’s the simplest solution, well-tested, and the dependency is small. Concurrency of 5-8 should comfortably stay within rate limits while providing 5-8x speedup over sequential processing. Since this is a bundled esbuild project, the dependency is compiled into the output — no runtime node_modules needed.

2.4 Decision

Option A — use p-limit for concurrency. Concurrency of 5-8, configurable.

3. Full Reindex as GitHub Action

R3: Full reindex via GitHub Action

3.1 Finding

Currently, the full department reindex runs on the Railway container via the Reindex Typesense skill. It frequently times out because Railway containers have limited resources and execution time.

GitHub Actions provide:

  • ubuntu-latest runners with 2 vCPUs, 7 GB RAM
  • No execution time limit for normal workflows (6-hour job timeout)
  • Access to GitHub Secrets for API keys
  • Can run the TypeScript indexer directly via npx tsx

The indexer needs: OPENAI_API_KEY, TYPESENSE_URL, TYPESENSE_ADMIN_KEY, and a checkout of the repo. All of these are available in a GitHub Action.

3.2 Options

A. New index-department.yml workflow triggered manually (workflow_dispatch)

A separate workflow that runs the TypeScript indexer. Can accept parameters (department name, incremental flag, since ref) via workflow_dispatch inputs.

  • Pros: Clean separation from DocSearch scraper; explicit trigger; parameterizable
  • Cons: Requires manual trigger or another workflow to call it

B. Add a reindex job to the existing index-search.yml

After the DocSearch scraper, add a second job that runs the department reindex.

  • Pros: Single workflow file
  • Cons: Couples two different concerns; reindex always runs when scraper runs; can’t trigger independently

C. New index-department.yml triggered on push + manual

Trigger automatically on relevant .md changes (using the same path filter as R1) AND support manual workflow_dispatch.

  • Pros: Automatic incremental reindex on content changes + manual full reindex
  • Cons: Adds compute cost for every content push (incremental reindex is fast though)

3.3 Recommendation

Option A — a new index-department.yml workflow with workflow_dispatch trigger. This gives explicit control over when reindexing happens. The incremental reindex is fast enough that it can be triggered manually when needed, and the full reindex can be run department-by-department. We can add push-based triggering later if desired.

The workflow should support inputs: mode (incremental/all/department), department (optional), since_ref (optional, defaults to HEAD~1).

3.4 Decision

Option A — new index-department.yml workflow with workflow_dispatch trigger. Supports inputs: mode (incremental/all/department), department (optional), since_ref (optional). This runs as a GitHub Action on ubuntu-latest runners, eliminating Railway timeout issues.

4. TypeScript Migration Approach

R4: Migrate indexing scripts to TypeScript; R5: Migrate Typesense MCP server to TypeScript

4.1 Finding

Two Python artifacts need TypeScript migration:

  1. index-department.py — the indexing script (R4). Currently at .internal/index-department.py. Should become a standalone TypeScript script or bundled project.
  2. typesense-mcp.py — the MCP server (R5). Currently at Forge/Configs/MCP_Servers/typesense-mcp.py. Should become a bundled TypeScript MCP server following the Write TypeScript skill conventions.

The forge-discovery MCP server provides a working reference implementation of the TypeScript MCP server pattern.

For the indexer, the Write TypeScript skill defines two patterns:

  • Bundled project: for long-running services
  • Standalone script: for one-shot tasks, run via npx tsx

The indexer is a one-shot task (run, process files, exit), so a standalone script is the natural fit. However, it has external dependencies (p-limit for concurrency, openai SDK for API calls), so it should be a bundled project instead (per the Write TypeScript skill rule: “If deps are needed, it should be a bundled project instead”).

4.2 Options

A. Separate projects: index-department (bundled) + typesense-mcp (bundled)

Two independent TypeScript projects under appropriate locations:

  • Forge/Typesense/Maintenance/index-department/ — bundled CLI for repo markdown indexing

  • Forge/Typesense/Maintenance/create-typesense-collections.ts — collection setup (moved from Forge/Configs/)

  • Forge/Configs/MCP_Servers/typesense-mcp/ — bundled project for the MCP server

  • Pros: Clean separation of concerns; each project has its own dependencies; follows existing patterns

  • Cons: Two projects to maintain; potential code duplication (Typesense API calls, env var reading)

B. Monorepo: single typesense project with both indexer and MCP server

One project with src/indexer.ts and src/mcp.ts, sharing utilities.

  • Pros: Shared code (Typesense client, env var reading, types); single dependency set
  • Cons: Bundles two different things together; the indexer and MCP server have different run patterns; doesn’t follow the existing MCP server convention

C. Shared utility package + two separate projects

Extract shared Typesense utilities into a shared module, then have two separate projects import from it.

  • Pros: Best separation with no duplication
  • Cons: Overengineered for two scripts; adds complexity to the build

4.3 Recommendation

Option A — two separate bundled projects. The indexer and MCP server are fundamentally different tools (CLI script vs. long-running stdio server). Shared code is minimal enough to duplicate (a few HTTP helpers and env var reading). The MCP server follows the established pattern in Forge/Configs/MCP_Servers/. The indexer and create-typesense-collections.ts live under Forge/Typesense/Maintenance/ (operational tooling, not .internal/ site build).

4.4 Decision

Option A — two separate bundled projects: index-department (at Forge/Typesense/Maintenance/index-department/) and typesense-mcp. Minimal shared code is acceptable to duplicate.

5. search_knowledge Default Status Filter

R7: Default search_knowledge to published-only results

5.1 Finding

The current search_knowledge tool accepts an optional status parameter with no default. When no status is specified, no filter is applied, so all documents regardless of status are returned.

The change is straightforward: set status default to "published" in the tool schema. When a caller explicitly passes a different status (e.g., "draft", "archived") or "all", the filter should respect that.

5.2 Options

A. Default status parameter to "published"

Change the Zod schema to status: z.string().optional().default("published").

  • Pros: Simple; backward-compatible for callers who don’t specify status; explicit override works
  • Cons: "all" is not a real Typesense status value — needs special handling

B. Default status to "published", special-case "all" to omit filter

When status === "all", omit the status filter from the filter_by string entirely.

  • Pros: Clean API; "all" is intuitive; no need to know the actual status values
  • Cons: "all" is a magic string

5.3 Recommendation

Option B — this is the most intuitive API. "published" as default, "all" to remove the filter, and any specific status value to filter explicitly. The implementation is trivial: check if status === "all" and skip the filter, otherwise include status:={value}.

5.4 Decision

Option B — default status to "published"; "all" omits the filter entirely. Any explicit status value filters normally.

6. Typesense MCP Skill Documentation Update

R6: Update Typesense MCP skill for current environment

6.1 Finding

The current skill documentation and code contain several outdated references:

  1. “no curl in container” — the base image is now debian-slim, which includes curl
  2. Python dependency references — the skill and MCP server will be in TypeScript after R4/R5
  3. /proc/1/environ env var pattern — this is a Railway-specific hack that reads from PID 1’s environment. In a GitHub Action, environment variables are available via process.env directly. The TypeScript version should use a cleaner approach (check process.env first, fall back to /proc/1/environ for Railway compatibility).

6.2 Options

A. Update after R4/R5 migration is complete

Wait until the TypeScript migration is done, then update the skill doc to reflect the new reality.

  • Pros: No intermediate updates; doc is accurate to final state
  • Cons: R6 is a separate requirement; delaying it couples it to R4/R5

B. Update incrementally

Fix the known inaccuracies now (remove “no curl” references), then update again after migration.

  • Pros: Fixes known misinformation immediately
  • Cons: Double work; the doc will change again after migration

6.3 Recommendation

Option A — update the skill doc as part of the final implementation after R4/R5 are complete. Since R4/R5 are part of the same project, they’ll be done in the same implementation cycle. The skill doc update is naturally the last step.

6.4 Decision

Option A — update skill documentation after R4/R5 TypeScript migration is complete, as part of the same implementation cycle.