BotsRated

Methodology

Version 0.4.

  1. Coverage. Seller hosts come from public, free lists: the Coinbase CDP x402 Bazaar discovery API, Agentic.Market, 402index (self-registered x402 entries, plus its bazaar [healthy], sponge, satring, discovery, MPP and L402 entries) and x402-list. Hosts from the extra 402index feeds and non-Base listings join only after one unpaid request answered a valid x402 402 with a Base USDC option; tunnels, testnet-only listings and subdomain farms (max 3 hosts per domain) are filtered out. Sellers that take L402, MPP or x402 on other networks get unpaid checks only (no test purchases, not ranked): /v1/other-payments. We keep sellers that accept USDC on Base and check one main route per host (picked from its listed routes when the host is added). The Coinbase CDP x402 Bazaar, Agentic.Market and 402index's Bazaar entries are used only to find services to check: we publish no usage counts, listing text or listed prices from them. Listing text and listed prices we show come from x402-list (Data: x402-list.com (CC BY 4.0), modified) and 402index. The lists are re-read automatically every day (from 01:00 UTC): new sellers join the rotation, and a seller missing (with a Base USDC option) from every list for 3 days in a row is marked listing: delisted. This is a fact about the lists, not about the seller; directory paging is not fully stable, so one missed day is not counted. Now: 4,514 hosts (coverage).
  2. Unpaid Check. One request per route, with no payment header and User-Agent BotsRated-probe/0.3, run automatically by a Cloudflare cron job every 2 minutes, so each seller is checked about every 30 minutes (busy or recently changed ones every few minutes, quiet ones once a day; full schedule below). Fast Tier. A second scheduler, started every minute, checks up to 500 hosts more often: hosts on a paid alert plan first, then the busiest hosts (picked from a directory's activity signal, used only to schedule checks and never shown) and hosts whose status, payTo or quoted price changed in the last 48 hours. They are checked about every 35 seconds for up to 100 of them, and the rest about every 4 minutes. It uses the same unpaid request; a check that sees a change is stored as a normal signed receipt. Measured gaps between checks are published in /v1/pulse (freshness). GET is tried first; POST {} is used if GET gives 400/404/405/415 (or the listing says POST). We record the UTC time, status, latency, whether the 402 parses as a valid x402 challenge with an accepts list, the quoted amount and payTo compared with the listing, header byte counts, and SHA-256 hashes of the request and response. Each cron receipt is signed with ed25519. Earlier receipts (UA ReceiptScore-probe/0.1 or BotsRated-probe/0.1) were run from our own machine and are covered by the signed snapshot. Exactly: once a host has at least 6 checks, its last check was not a 402 and it sent no valid x402 payment request in the last 7 days, it is marked probe: inactive and checked about once a day until it sends one again. An answer of HTTP 401 or 403 means the service wants a login; we show it as "Needs Login", not as a failure. 48 sellers with more than one listed route also get 3 of their other listed routes checked about every 40 minutes, taking turns through all of their listed routes. Each extra route is compared with its own listing.
  3. 402 Challenge Checks. On every valid 402 we record: terms_consistent (every accept option has scheme, network, an integer amount, an asset and an EVM payTo; the v2 header and a JSON body that also lists accepts quote the same terms; the resource URL is on the checked host), base_usdc (an option in USDC on Base), payto_stable (same payTo as the previous valid check), and price_matches_listing / payto_matches_listing. Issue codes are neutral descriptions of what the challenge contained.
  4. Retention. We will archive each day's raw receipts with a signed manifest; the public archive opens later. Until then, raw receipts are kept 7 days.
  5. Reliability Numbers. "Recent" means up to the last 20 unpaid checks. We publish the valid-402 rate and its Wilson 95% lower bound, so a host with 2 checks does not look as proven as one with 20. On our pages a "k of n" share is shown only once there are at least 5 recent checks. Data older than 24 h is marked stale.
  6. Resent Payment Check (Does A Resent Payment Get Served Again?). A settled x402 payment can't be charged twice. Not running today. If we start it, it will cover only sellers who opt in. How it would work: during one of our test purchases from a seller who opted in, after our payment has settled on the Base blockchain, we would send the same signed payment once more to the same address and record how the service answers: asked for a new payment, refused it, returned an error, or served the request again. We would never create a new payment for it. Limits: only sellers who have opted in (we record the date and where they opted in); at most one resend per service every 7 days; purchases of $0.01 or less; the service's terms reviewed beforehand; and its robots.txt must allow the route. Asking for a new payment and refusing are neutral results and could be shown. An error would be shared privately with the service at least 72 hours before it is published. If a service serves the request again, that is a security weakness, not a rating: we report it privately to the service as a security report. We never publish it under the service's name until at least 90 days after we have reached them, and never if we can't reach them (then we publish, at most, anonymous totals). A result we cannot confirm would not be published. Opting out of all checks: disputes and opt-out.
  7. Test Purchases. We pay the quoted price from a capped wallet (max $1.50 per purchase, $2 per day, $20 per month) and record the endpoint, quoted and charged amounts, the Base tx hash, the time, and whether the response was delivered, non-empty, schema-valid and correct-looking against a separate check of our own. Failed or doubtful purchases are re-tested and held privately. We publish a failing result only after sending notice to a contact the seller publishes and waiting at least 72 hours; if we can't reach the seller, the result stays unpublished. A monthly negative test sends invalid input to check whether we are charged when a call fails.
  8. Best Pick. /v1/best keeps hosts whose listing text matches the task keywords and that returned a valid x402 challenge within 48 h, then sorts by reliability (Wilson lower bound, in 0.05 bands), last quoted price or median latency. Short check windows put many hosts in the same reliability band, so ties are broken in a fixed order: text match, test purchase evidence, payTo stability, share and number of consistent 402 checks, latency band, quoted price, lifetime reliability. Listings that say they are tests or demos are left out. Every result says which key placed it above the next one (why.decided_by). Category tags come from listing text and directory tags, or the service's own 402 description; they describe what a listing says it does, not a judgment.
  9. Remote MCP Servers. Remote MCP endpoints (streamable-http and SSE) come from the Official MCP Registry (latest version of each entry, status active; URL templates skipped), deduplicated by URL; the list is re-read daily and grows within a daily cap. Each server gets a login-free MCP handshake, aiming for about once a day per listed server while we are still working through the registry (priority tier: servers on hosts looked up via BotsRated preflight/check/report, or that are also covered x402 sellers, in the last 7 days, about every 6 hours; lookups only speed up MCP checks and never change how often pay-per-call services are checked): initialize (protocol 2025-06-18), notifications/initialized, tools/list. Nothing else is sent, with one exception: on tools whose listing states a price, we send at most 2 unpaid tools/call requests per check to read the payment request (price, asset, network, payTo). We never pay, never call a tool that does not state a price or is marked destructive, and discard any tool output. No credentials, cookies or API keys. Only public https URLs on DNS names whose addresses are public are contacted (at most 3 redirects, each re-checked). We record reachability, latency, the negotiated protocol version, server name/version, whether the server asks for authentication (401/403, with the WWW-Authenticate scheme and OAuth resource-metadata URL if given), TLS (a verified HTTPS answer), the tool count, tool names, a hash per tool and a hash of the whole tool list (so tool changes are visible). Each check is a signed mcp_probe receipt (key botsrated-prober-1). Price check in progress means our comparison of a server's stated and requested prices is not finished yet; until it is, we show neither the comparison nor links to that server's receipts. States are factual: ok (initialize and tools/list answered), auth_required (HTTP 401/403: the server wants a login), handshake_failed (HTTP answer without a valid MCP result), unreachable (no HTTP answer), blocked (not contacted: the URL, a redirect or its DNS points to a private, local or otherwise non-public address, or to another host, which our rules don't allow). Servers with no MCP answer for 7 days are checked about every 4 days. Now: 33,122 MCP servers checked, 33,339 listed.
  10. Index. No composite index is shown until a host has at least 3 test purchases on at least 3 different days. Until then the label reads "No index yet".
  11. Badge And Money. The badge (/badge/<host>.svg) is free and earned from measurements; it is never sold. Paying BotsRated never buys a better result, rank or badge. Paid alert plans make us check the hosts they cover more often; those checks use the same method and are published the same way. See pricing.
  12. Facts Only. Every test result links to its signed receipt. We do not label sellers; we publish what we measured. Some price differences are legitimate, for example input-dependent pricing.
  13. Independent. No seller can pay for a better result, rank or badge, and we don't sell placement. BotsRated sells paid checks of its own, and some services we list offer similar checks. They get exactly the same automated checks as everyone else. We never list or rank BotsRated itself.
  14. Integrity. Snapshot: /dataset.json signed in /dataset.sig.json. Keys: /.well-known/botsrated-key (snapshots and test purchase receipts: botsrated-prod-1; automatic checks: botsrated-prober-1; the retired botsrated-staging-1 stays listed so older signatures still verify). Test purchase receipts are signed one by one, like check receipts. Latency percentiles use nearest rank, sorted[min(n-1, floor(p·(n-1) + 0.5))], everywhere.