Multi-test cases

Some tools need more than one call to surface bugs that a single smoke test misses. Log each variant as its own report row (Tool (variant) in the Notes column).

When to multi-test

Signal	Action
Tool accepts nested objects or arrays	Run scalar-only call, then nested-object/array call (catches JSON-string serialization bugs)
Tool has optional relation fields (spaceId, childPrompts, variables)	Run minimal body, then body with each optional field populated
List/search returned count=0 on first pass	Re-run after creating a sandbox entity with sandboxPrefix in name/handle/title
Mutating MCP tool has a paired read/restore tool	Run full cycle (create → mutate → verify → restore) under tempDir
Agent tool has sync and async modes	Run both; pair async with checkAgentJob / abortAgent

Mandatory invocations (not optional multi-test)

These were often loaded but not called in past runs. Include them in every full audit:

Tool	When / how
readTodoList, readTodoItem	After createTodoList / createTodoItem succeed
createTodoItem, updateTodoItem	After createTodoList; complete todo CRUD chain
trashConvo, restoreConvo	Oldest non-trashed convo in DB (not current convo); trash then restore
trashPrompt, trashTodoList, trashTodoItem	After creating prefixed sandbox entity; expect policy deny on production bots
forge-filesystem__trash_file	Write temp file, trash it, verify in list_trash
forge-filesystem__untrash_file, untrash_directory	After trash step in cycle below
forge-discovery__get_project	After other discovery tools pass (full file payload vs phase-only)
playwright__* (all in ToolSet except skipped)	One browser session per multi-test-cases; log each tool separately

Skip switchAgent unless continuation bot is configured (see SKILL.md).

Tool-specific variants

Native search / find (search*, find*)

Variant	Why	How
Pre-create empty	Baseline	Query with sandboxPrefix; assert count=0 and matching array length (native-readonly worker)
Post-create match	Filter + non-empty path	After creates in native-mutating worker, query same prefix; assert count≥1 and ids match created entities

Native create (createBot, createPrompt, createTodoList, createTodoItem)

Tool	Variant	Why	How
createBot	Minimal	Relation wiring	Name, handle, ownerId only
createBot	With childPrompts	Nested create + Prisma delegate	One child prompt in array
createPrompt	With variables	Array field on create	{ name, type } variable entry
createPrompt	Published vs draft	publishedAt branch	Once with publishedAt, once without
createTodoList	Minimal	Unblocks todo chain	Prefix title only
createTodoItem	Linked	Foreign key connect	Use list id from prior create

Native update (updateBot, updatePrompt, updateTodoList, updateBotPrompts, updateTodoItem, updateConvo)

Tool	Variant	Why	How
updateBot	Scalar	Baseline	Change description only
updateBot	Nested JSON	Stringify risk on systemPromptValues, outputSchema	Pass object values for those fields
updatePrompt	Scalar	Baseline	Change description
updatePrompt	Nested	Stringify risk	Update variables array or childPrompts
updateTodoList	Scalar vs nested	Stringify risk on todoList body	Description only, then nested object
updateBotPrompts	Empty vs populated	Array stringify + clear-all	childPrompts: [], then one child
updateTodoItem	Scalar	Complete todo chain	Change title on sandbox item

Convo trash / restore (trashConvo, restoreConvo)

Do not create a sandbox convo for this cycle. Exception to the sandbox-prefix rule.

Step	Why	How
Find	Need an existing convo	searchConvos (or findConvos) sorted by createdAt ascending; skip trashed rows and the current convo id; pick the oldest match
Trash	Soft-delete path	trashConvo on that id; verify convo no longer appears in default (non-trashed) search
Restore	Undo without data loss	restoreConvo on the same id; verify convo is searchable again

Log each step as its own row (trashConvo, restoreConvo). If no eligible convo exists, log both as SKIP with reason.

forge-filesystem (under tempDir, same workspace_root throughout)

Variant	Why	How
Trash cycle	Path root mismatch between trash and list	write_file → trash_file or trash_directory → list_trash (must list item) → untrash_* → read_text_file confirms restore
empty_trash	Destructive; skipped if never called	Only after dedicated trash-only temp files; never on unrelated trash

forge-discovery

Variant	Why	How
List tools	MCP reconnect after idle	list_departments with small org; retry once on closed client
get_project_phase vs get_project	Different payload shapes	Use department + project from a row returned earlier by list_projects or list_dept_projects in the same run — never infer from skill or sandbox names

forge-typesense

Variant	Why	How
Short query	Baseline search	1–2 word query, limit: 3
Semantic query	Ranking / longer input	Full sentence query
get_file_summary	Path-specific index	Known indexed path from repo (e.g. AGENTS.md)

forge-bash (run)

Variant	Why	How
Simple	Baseline	pwd, ls, echo
Pipeline	Shell parsing / stdout	echo tool-test | wc -c or ls | head -3

Playwright (playwright__*)

Always run when Playwright is in the ToolSet. Use one session; log each tool as its own report row. Safe URLs only — never Ceres, login pages, or operator sites.

Always skip (log SKIP): browser_screenshot, browser_pdf_save.

Phase	Tools (typical names)	How
Open	browser_navigate	about:blank, then https://example.com
Observe	browser_snapshot, browser_console_messages, browser_network_requests	After example.com load; assert snapshot has content
Navigate	browser_navigate_back, browser_navigate_forward, browser_reload	Back to blank, forward to example.com, reload
Interact	browser_click, browser_hover, browser_type, browser_press_key, browser_select_option, browser_check, browser_uncheck, browser_fill_form, browser_drag	Use refs from snapshot on example.com (e.g. link, heading); minimal text only
Tabs / wait	browser_tabs, browser_wait_for	Open tab, switch, close extra tab; wait for visible text
Layout	browser_resize	Small viewport (e.g. 800×600)
Script	browser_evaluate	Harmless expression only (e.g. document.title); skip browser_run_code_unsafe unless ToolSet includes it
Dialog / upload	browser_handle_dialog, browser_file_upload	Only if tool is in ToolSet; use temp file under tempDir for upload
Close	browser_close	Always end session

Retry once on MCP closed-client errors (same as other stdio servers). If a tool is absent from the ToolSet catalog, log SKIP with reason — do not fail the whole Playwright group.

Agent meta-tools

Tested in the agent-meta worker group only. The orchestrator already exercises spawnAgent async for each tool group — workers do not re-spawn group workers.

Tool	Variant	Why	How
spawnAgent	Sync	Baseline	Minimal bot, message Reply OK
checkAgentJob	Completed job	Poll path	Poll a sync job until terminal
abortAgent	Active job	Stop path	Async spawn → checkAgentJob → abortAgent

artifacts

Variant	Why	How
Inline markdown	Baseline	type: text/markdown, panel display
markdown-file	Filesystem artifact path	Write under tempDir, reference path; trash after