Test Tools

When to Use

Operator asks to smoke-test, regression-test, or audit every tool in the bot’s ToolSet.

Prerequisites

• Forgentic chat with a bot whose ToolSet includes the tools under test
• spawnAgent and checkAgentJob on the orchestrator bot (required — each tool group runs in its own sub-agent convo)
• toolSearch, loadTool, and unloadTool in ToolSet when deferred tools are included
• forge-filesystem in ToolSet when testing filesystem MCP tools
• playwright in ToolSet when testing browser MCP tools (always exercised when present)
• High maxSteps on the orchestrator and worker bots (100+)
• Operator available to answer ask*Question UI tools once if those groups are included
• Continuation bot (botGroup + botHandle) when switchAgent is in the ToolSet

Procedure

Orchestrator (this convo)

1. Prepare run — set runId to UTC YYYY-MM-DD-HH-mm-ss, sandboxPrefix to tool-test-{runId}, tempDir to .tool-test/{runId}/, reportIdentifier to {sandboxPrefix}-report, reportPath to Forge/Output/Tool_Test_{runId}.md. Use these for all inputs and created entities. Create the report file immediately in Forge/Output using the header from references/report-template.md (with {runId}, {sandboxPrefix} substituted, and 0 pass, 0 fail, 0 skip as placeholder counts). This ensures results survive a crash.

2. Discover tools — build pending[]:

   • Add every eager tool in the session tool list.
   • For each deferred group/server: toolSearch with maxResults: 50 and queries list, search, read, create; collect unique toolKey values into pending[] — do not call loadTool here.

3. Partition into groups — assign each tool in pending[] to exactly one group (skip groups with zero tools):

   Group id	Tools
   meta-deferred	toolSearch, loadTool, unloadTool
   native-readonly	search*, find*, read* (except convo read used only after trash)
   native-mutating	create*, update*, trash*, restoreConvo, convo trash/restore
   mcp-readonly	forge-discovery__*, forge-typesense__*
   mcp-mutating	forge-filesystem__*, forge-bash__*
   playwright	playwright__*
   ui	artifacts, ask*Question, getWeather, searchTheWeb, extractWebPage
   external	composio__* and other external MCP tools
   agent-meta	spawnAgent, checkAgentJob, abortAgent

   Do not include switchAgent in any group — orchestrator handles it in step 7.

4. Spawn workers — for each non-empty group, in table order:

   1. spawnAgent with mode: 'async', same bot as orchestrator, overrides.maxSteps: 100, and a message containing: groupId, runId, sandboxPrefix, tempDir, reportPath, the tool list for that group, and instruction to follow references/multi-test-cases.md + references/tool-categories.md for that group only.
   2. Record jobId + groupId.
   3. Poll checkAgentJob until terminal (completed, failed, or aborted) before spawning the next group — preserves create-before-search ordering across native-mutating and earlier read-only baselines.
   4. On worker failure, append a ## Failures note for the group and continue with remaining groups.

   The orchestrator does not call tools under test directly (except discovery in step 2). All tool smoke tests run in worker convos.

5. Finalize report — after all groups finish, read reportPath, verify every tool in pending[] has a row (or SKIP with reason). Add ## Incomplete for any missing tools.

6. Report — artifacts with identifier = reportIdentifier, type: text/markdown, display: panel. Body is the current content of the report file at reportPath. Summarize final counts in chat.

7. switchAgent (last) — if in ToolSet, call last (after the report). Hand off to the continuation bot. reason: update artifact {reportIdentifier} with final status, summarize in chat, and stop — do not continue testing. Append the switchAgent row to the report file first. SKIP if no continuation bot is configured.

8. Partial runs — if the orchestrator exhausts steps before all groups spawn or poll, update the report file with ## Incomplete listing remaining groups/tools, then switchAgent if available.

Worker (each sub-agent convo)

Test only the tools assigned in the spawn message — one tool (or variant) at a time:

• Per call: plan → call → assert (count matches array length on list/search tools) → cleanup → append row to report file (forge-filesystem__edit_file on reportPath, adding a table row and updating summary counts) → log row. Retry once; then FAIL and continue. Every test result is persisted immediately — if the worker crashes, the report file already contains all results up to that point.
• Multi-test: Tools with nested params, optional relations, list/search filters, MCP cycles, or agent job modes need extra variants — see references/multi-test-cases.md. Log each variant as its own row (Tool (variant)).
• native-readonly group: pre-create empty baseline only.
• native-mutating group: full CRUD chain; after creates, re-run applicable search* / find* / read* with sandboxPrefix (post-create path); complete todo chain; invoke every trash* per multi-test-cases.
• playwright group: one headless session; skip browser_screenshot and browser_pdf_save (log SKIP); call every other playwright__* tool on safe URLs.
• agent-meta group: test spawnAgent sync, checkAgentJob, and abortAgent on jobs started in that worker convo only — do not respawn orchestrator workers.

Category rules: references/tool-categories.md.

Rules

• All names/handles/titles/queries of created database items use sandboxPrefix; never update or trash non-prefixed entities except convo trash/restore (see multi-test-cases).
• Convo trash/restore: use the oldest non-trashed convo in the DB (not the current convo); trash then restore it.
• Filesystem writes only under tempDir; trash after test; use the same workspace_root for trash, list, and untrash.
• Shell: read-only only (pwd, ls, echo, simple pipes); no rm, git push, installs.
• Composio / external MCP: smallest read-only call; skip disconnected integrations.
• Playwright: test every in-ToolSet playwright__* tool except browser_screenshot and browser_pdf_save (always SKIP those); safe URLs only (about:blank, https://example.com); end with browser_close. Never log in or hit production app URLs.
• Call switchAgent only once, on the orchestrator, after the report (step 7); never mid-run.
• One tool variant at a time per worker; no secrets or full file bodies in the report.
• Nested-object/array updates (updatePrompt, updateBot, updateTodoList, updateBotPrompts): always run scalar and nested variants per references/multi-test-cases.md.