Office of Kaiju Affairs

The Agent Arena

The kaiju left. The paperwork did not.

Mira Tanaka's bakery has been damaged. She needs compensation, a rebuilding permit, and a straight answer about what she owes. Unfortunately, she now faces something almost as intimidating as a kaiju: a government website.

We gave agents three jobs — find a buried fee rule, determine Mira's eligibility, and complete her permit application without submitting it. One agent navigates the website like any visitor. The other uses seven WebMCP tools generated from the site's published content.

We recorded 95 attempts across two model sizes. WebMCP did not win every comparison. That is what made the experiment useful.

Help Mira with a task See what happened

Checking whether this browser can use WebMCP…

All three tasks at a glance

Each row summarizes five attempts using the same model, task, website, and scoring checklist. Time and actions show how much work the agent performed. Score shows how much of the expected answer it gave Mira. Open any task below to inspect the recordings and answer key.

Loading the recordings…

Help Mira

The scenarios are fictional. The bureaucracy is alarmingly plausible. Copy a task into your agent while this page is open, then compare its answer with the recorded browsing and WebMCP attempts.

Loading tasks…

What this project is testing

Can a CMS publish an agent map?

The CMS generates a manifest describing the pages, services, sources, forms and selected rules published on this site. One browser runtime turns that manifest into seven WebMCP tools.

The manifest is a lightweight map of what this publisher has encoded. It is not a magical government-wide source of truth — those tend to live in at least six systems, three contracts, and one spreadsheet nobody is allowed to delete.

Can we test the agent experience like a public service?

We recorded the same citizen tasks with and without WebMCP. Each recording includes the agent's actions, tool calls, elapsed time, final answer and score. The Arena calculates its figures from those files.

The test found two real design problems: an eligibility tool that returned a verdict without the explanation Mira needed, and a form tool that knew fields were missing but failed to mention them. We fixed both tools and kept the earlier recordings.

Can a tool create a real human handoff?

The permit tool validates and fills Mira's application, then stops. It exposes no submit operation. This is a boundary of the WebMCP capability — not a claim that no agent could ever interact with the website through another mechanism.

The tool prepares. Mira decides.

Can private facts stay private?

Eligibility decisions may involve income, property, claims history and other sensitive information. The current tool evaluates those facts inside the static page. A stronger design would publish the rules in a form the agent could evaluate locally, without giving the website Mira's facts at all.

Public rules should be public. Private circumstances should remain private whenever possible.

Two civic-technology lessons we did not expect

Executable policy needs editorial review

WebMCP can reduce the need for an agent to interpret policy prose. But someone still has to translate that policy into executable rules, and that translation can be wrong.

A citation may point to the correct page while an AND, OR, threshold or exception is encoded incorrectly. The result can be consistent, sourced, and confidently wrong — an uncomfortable combination for any public service. Our three scenarios test the rules demonstrated here. They do not prove that every machine-readable rule means exactly what every public sentence intends.

The next step is not asking policy editors to inspect JSON. It is giving them:

  • a plain-language explanation of what the machine will do;
  • automatically generated example decisions;
  • boundary cases;
  • supporting evidence from the cited policy; and
  • a human approval step before publication.

If policy is going to become executable, reviewing the machine interpretation must become part of publishing it.

One website cannot represent the whole government

The Office of Kaiju Affairs fits neatly inside one CMS because it is fictional. Real public services do not. The policy may live in one department, the explanation in another CMS, the application in a SaaS form product, the case history in a legacy platform, and the exception in legislation everyone assumed somebody else had encoded.

A WebMCP tool can be perfectly correct about its own system and still give a citizen an incomplete answer. The manifest is therefore not “the truth”. It is a machine-readable map of one publisher's portion of the truth.

At institutional scale, those maps need to connect. Each publisher should identify:

  • what organization and policy version it represents;
  • what services and rules it covers;
  • what it does not cover;
  • where related manifests can be found; and
  • when the right answer is a referral rather than a confident verdict.

The goal is not one enormous government database. It is a network of accountable publishers whose boundaries agents — and citizens — can see.