Loading ToolBench
Remote MCP server for Firm workspaces, deployed on Cloud Run. Wraps FirmMcpServer with git commit/push on writes, OAuth authentication, and comprehensive workspace management tools.
Firm Remote MCP Server demonstrates solid definition quality with 14 well-named, verb-led tools operating on a DSL-based entity system. Tool names are clear and action-focused (list, get, query, build, add_entity, replace_source, etc.). Descriptions are comprehensive and contextual, ranging 150-400+ characters with explicit guidance on when to use each tool and how they compose. All tools have explicit input schemas with typed parameters and descriptions. However, several critical patterns are incomplete: (1) No output schemas documented, users/LLMs cannot predict response structure or chain tools reliably. (2) Tool annotations (readOnlyHint/destructiveHint/idempotentHint) are absent despite clear risk categorization in the rubric (READ_ONLY vs WRITE vs DESTRUCTIVE). (3) Error handling and recovery guidance are minimal, tools return success/failure but lack actionable recovery hints. (4) Pagination is not visible in query/list tools despite potentially large result sets. (5) No per-parameter constraints (enums, ranges, formats) beyond descriptions. The tools form a coherent composition (discovery → read → write → git commit), and naming conventions are consistent. Overall: above-average definitions hampered by missing output schemas, error handling, and safety annotations.
Add a new entity to the workspace. Provide the entity type, ID, and a map of field values (JSON types). The entity is added to the workspace and committed/pushed to git.
Sync with remote and rebuild the workspace. Fetches the latest state from the remote mcp branch (or main if mcp was deleted after a PR merge). Returns the current status: number of entities and schemas if valid, or validation errors if the workspace is broken. Call this before starting work to ensure you have the latest data.
Delete a .firm source file. Provide the relative file path. The deletion is committed and pushed to git.
Get reference documentation for the Firm DSL syntax and query language. Use 'topic' parameter: 'dsl' for DSL syntax (entities, schemas, field types), 'query' for query language (from, where, related, order, limit, aggregations), or 'all' for both (default). Call this before writing or modifying .firm files to understand the correct syntax.
Find the source file path for an entity or schema. Returns the relative path to the .firm file containing the definition. Use this to locate where an entity or schema is defined before reading or editing the source file.
No output schemas documented for any tool. LLMs cannot predict response structure, field names, or types. This forces LLMs to reason about response shape from context alone, increasing hallucination risk and preventing reliable tool composition.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are absent. Tools are classified as READ_ONLY, WRITE, and DESTRUCTIVE in the rubric, but the MCP server definition does not expose these safety hints. LLMs cannot distinguish which tools are safe to retry, which modify state, and which cause irreversible changes.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 78 | <=2025-11-25 | v2 |
Get full details of a single entity or schema. For entities: provide the entity type (e.g., 'person') and ID (e.g., 'john_doe'). For schemas: use type='schema' and id=<schema_name> (e.g., id='person'). Returns all fields and their values. Use 'list' first to discover available IDs.
List all entity IDs of a given type, or all schema names if type is 'schema'. Returns only IDs/names for discovery purposes. Use 'get' to retrieve full details for a specific entity or schema, or use 'query' to fetch details for multiple entities matching search criteria.
Query entities using the Firm query language. Returns full details for all matching entities, or an aggregated result when an aggregation clause is used. Examples: 'from person', 'from task | where is_completed == false', 'from task | where is_completed == false and priority > 5', 'from invoice | where status == "draft" or status == "sent"', 'from person | where name contains "John" | limit 5', 'from task | count', 'from invoice | where status == "sent" | sum amount', 'from task | where is_completed == false | select @id, name, due_date'. Use 'list' for a simple ID overview, or 'get' for a single entity's details.
Read the raw DSL content of a .firm source file. Provide the relative path to the file (e.g., 'schemas/person.firm', 'core/main.firm'). Use 'find_source' first to locate the file path for a specific entity or schema.
Get IDs of entities related to a specific entity. Returns entity IDs that reference or are referenced by the given entity. Use 'direction' to filter: 'incoming' (entities that reference this one), 'outgoing' (entities this one references), or omit for both.
Replace the entire content of a .firm source file with new DSL. Provide the relative file path and the new DSL content. Use 'read_source' first to see current content. Changes are committed and pushed to git.
Search for a text string across all .firm source files. Returns matching lines with file paths and line numbers. Case-insensitive by default. Use this to find where entities, fields, or values are defined or referenced.
Show the file tree of all .firm source files in the workspace. Use this to understand the file layout before reading, writing, or organizing source files.
Create or append to a .firm source file. Provide the relative file path and DSL content. If the file exists, content is appended. If not, it is created. Changes are committed and pushed to git.
Destructive tool (delete_source) lacks confirmation or dry-run capability. Agents can irreversibly delete .firm files without a safety gate. No recovery guidance in error responses.
Error handling and recovery guidance are minimal. Tool descriptions do not explain what to do if a call fails (e.g., 'If entity not found, try list() first'). Error responses from the implementation likely return raw status without actionable recovery hints.
No pagination support visible in list and query tools. Descriptions mention examples and queries returning multiple entities, but no limit, offset, or cursor parameters are documented. Large result sets could blow context windows.
Parameter descriptions lack formal constraints (enums, ranges, regex). The 'query' parameter in query() and search_source() accept free-form strings with no format guidance. The 'type' parameter in list/get/related/find_source appears to accept any string with no enum of valid entity types.
add_entity 'fields' parameter is typed as generic 'object' with no schema for field values. LLMs cannot validate which fields are required, their types, or constraints. This invites hallucinated field names and invalid values.
dsl_reference tool description does not return actual reference content, it only lists available topics. The response schema is unclear (does it return markdown? structured sections?). Users cannot get help during tool invocations without an extra round-trip.