Loading ToolBench
MCP Agents to help with ATLAS physics analysis, providing tools to query and manage ATLAS physics analysis datasets, Monte Carlo samples, and metadata through the CentralPage catalog.
The server provides 6 well-named read-only tools with consistent descriptions and proper input schemas. Naming follows verb_noun convention (get_*, search_*) and is clear. All descriptions exceed the 20-character minimum and explain purpose. Input schemas are well-formed with type declarations and parameter descriptions. However, output schemas are not explicitly documented (tools return JSON strings without structure specification), error handling guidance is minimal, and there is no pagination or result-limiting strategy despite tools that could return unbounded datasets. The implementation is solid for basic usage but lacks the robustness expected of production-grade agent tools.
Searches the PMG group's Standard Model Monte Carlo datasets for a hashtag that contains `keyword`. Only hashtags in `scope` are considered. Full 4-tuples hashtags are returned. These tuples can be passed to other methods to return datasets associated with them. The hashtags specify categories of datasets. They often have easily understandable english names and so make for a great place to start a Standard Model dataset search. The third returned tag indicates whether the dataset is 'Baseline', 'Systematic', or 'Alternative'. By default only hashtag combinations with `Baseline` are returned. If one needs samples that are alternative for for systematic comparisons, change the `baseline_only` parameter. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json
Returns a list of allowed scopes/data-taking-periods for the CentralPage MC Sample catalog. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json.
Searches AMI for all EVNT tier datasets whose name contains `search_str`. If you know you are looking for a standard model dataset, ``is_central_page`` should be set to ``True``. If you are looking for something exotic, then ``is_central_page`` should be ``False``. If ``is_central_page`` is ``True``, only datasets with PMG / central-page hashtags are returned. If ``is_central_page`` is ``False``, all matching EVNT datasets whose name contains ``search_str`` are returned, regardless of PMG / central-page hashtags. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json string
Output schemas are not explicitly documented. All tools return JSON strings (via json.dumps()) but the structure of returned data is not formally specified. LLMs cannot reliably parse or compose the results without trial-and-error, and cannot plan downstream tool chains.
No pagination, result limiting, or cursor support. Tools like get_addresses_for_keyword and get_dataset_with_name could return unbounded result sets, blowing the context window. Descriptions do not mention result limits or pagination options.
Error handling lacks recovery guidance. Tools raise ValueError (e.g., 'hashtags must be a list of 4 strings') and call underlying cp.* functions whose error modes are not visible. No error descriptions guide the LLM on what to do next or how to self-correct.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 42 | - | v1 |
Returns a list of event generator (evtgen) sample names for a given CentralPageAddress. These will be rucio dataset names, for datasets that contains the output of the MC generation step. All samples for this address are returned. Parse the sample names to find the ones required. Sample names often contain decay channels, etc. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json
Returns metadata for a given dataset as JSON. This includes cross section, generator filter efficiency, physics short name, etc. If ``use_top_of_provenance`` is True, the server will first resolve the provenance chain and fetch metadata for the top (last) dataset, typically the EVNT. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json
Returns a list of rucio dataset names of a particular data_tier for a given EVTGEN sample and scope. evtgen_sample should be a valid rucio ID, with EVNT as the data tier. data_tier should be "PHYSLITE", "PHYS", "DAOD_LLP1", etc. Default to PHYSLITE unless otherwise requested. Returns the datasets and the ATLAS MC Campaigns. Those without a MC campaign should probably be ignored. `ignore_cache` can be set to True to bypass the cache - but should only be done if explicitly requested (or for testing). Returns json
Parameter 'run_number' in get_samples_for_run is documented as 'Run number or EVTGEN sample identifier' but is cast to int(run_number). If a user passes an EVTGEN sample name (a string identifier), the int() cast will fail silently or raise ValueError. The parameter type should clarify whether it accepts only numeric IDs or also string identifiers.
No tool annotations (readOnlyHint, idempotentHint, etc.). While all 6 tools are read-only (safe), this is not formally declared in the tool schema, forcing LLMs to infer safety from the description alone.
Dependencies between tools are not documented. For example, get_evtgen_for_address expects 'hashtags' from get_addresses_for_keyword, but this is not stated in descriptions. Discovery hints ('Call get_addresses_for_keyword first to get hashtags') would guide multi-step planning.