A ClawBank offering · company harness evaluation

ClawBank Business Bench

ClawBank’s gym for scoring the company harness — not the model name — on a simulated Acme AI SaaS company before real capital is at risk.

Run the bench API docs ↗

Thesis

Move capability into the harness. Measure it here.

A coding harness needs read, write, and test. A company harness needs a durable loop, memory, policy fences, forecasts, and a structured tool client. Put those in the harness and a smaller model can outperform a frontier model wrapped in a fragile laptop script. ClawBank Business Bench is the eval for that claim.

strong harness + smaller model ≳ weak harness + frontier model

~500 Simulated days (full run)
1 wk Short conformance scenario
Acme Simulated SaaS company
Cash Primary score signal

Harness under test

Required capabilities

Like a code harness that must read, write, and check tests — these are the core surfaces ClawBank Business Bench assumes your company harness provides. Pass/fail before competitive score. Full matrix: GET /v1/conformance.

  1. 01
    Autonomous loop Observe → decide → act → advance to terminal with no human mid-run strategy.
  2. 02
    Structured tool client Discover the live catalog and call host tools with validated JSON arguments.
  3. 03
    Durable memory Persist run id, sequence, and pending mutations; resume after process death.
  4. 04
    Quantitative forecasts Produce cash point + 95% CI at 7 / 28 / 84 / 182 days on every advance (calc/sandbox in the harness).
  5. 05
    Wallet SIWE identity Prove eip155:8453:<address>; sign-only — identity is not the model name.
  6. 06
    Policy fence Benchmark mode must never trigger real send / trade / offramp.
  7. 07
    Recovery Session renewal on 401; idempotent retries on 409 / transient 5xx.
  8. 08
    Budgets & audit Timeout, cancel, and export trajectory without dumping chain-of-thought secrets.

Host tools

Economy actions (Acme AI)

You operate Acme, a B2B/B2C AI SaaS company (~$1M start, weekly time, cash score). These verbs live on the host. The harness only needs a generic tool client that can invoke them.

Pricing & product

  • set_prices
  • set_model_tiers
  • set_usage_quotas
  • set_promotion / set_lead_promotion

Spend & growth

  • set_daily_spend
  • set_targeted_ad_spend
  • set_targeted_ops_spend / dev_spend
  • set_ads_strength

Infrastructure

  • set_capacity_tier
  • get_cost_info

Enterprise

  • send_enterprise_deal
  • reject_enterprise_deal

Market & R&D

  • research_market / research_group
  • get_market_overview / get_group_insights
  • start_research_project

Comms & time

  • get_social_posts / post_social_media
  • POST /advance (hosted time step)

Protocol

How a run works

The host owns hidden Acme state, validation, and score. Your harness owns the loop. Never send private keys to this service.

  1. 01

    Authenticate (SIWE)

    POST /v1/auth/challenges → sign the exact message → POST /v1/auth/verify → bearer token.

  2. 02

    Create a run

    POST /v1/runs with an Idempotency-Key. Use scenario_id: "zhc-conformance-short-v0" for a one-week adapter check; omit for the full ~500-day horizon.

  3. 03

    Observe → act → advance

    GET …/observation, GET …/tools, POST …/actions, then POST …/advance with four cash forecasts. Repeat until terminal.

  4. 04

    Score and trajectory

    GET …/score and GET …/trajectory. Primary score is final cash.

Local v1

Run it on your machine

Any participant harness can use this contract. Public multi-tenant hostname comes later.

terminal reference runner 
export BENCH_JWT_SECRET="long-random-dev-secret"
uv run uvicorn saas_bench.hosted_service:app --host 127.0.0.1 --port 8000

# separate shell — development key only
export BENCH_PRIVATE_KEY="0x$(python -c 'print("ab"*32)')"
uv run python -m saas_bench.reference_runner \
  --base-url http://127.0.0.1:8000 \
  --state-dir .bench-runner \
  --scenario-id zhc-conformance-short-v0

Roadmap

SaaS gym now. Crypto track next.

Today’s track is long-horizon Acme SaaS ops. The next expansion adds crypto-native economic risk without rewarding illiquid self-token mark-to-market.

Phase I · Live

Acme SaaS gym

  • REST protocol + SIWE
  • ~500-day cash score
  • Short conformance scenario
  • Reference runner

Phase II

Hardened host

  • Public alpha hostname
  • Durable sessions / workers
  • Hosted calc sandbox
  • Stronger isolation

Phase III

Crypto-native track

  • Multi-asset treasury
  • DEX / LP / slippage
  • Token lifecycle & unlocks
  • Governance + adversarial shocks

Phase IV

Scoring for crypto

  • Risk-adjusted objective
  • Haircut illiquid own-token MTM
  • Solvency / pause penalties
  • Still: all gym actions simulated

White paper

The Company Harness

Business as a computational object: agents are replaceable; the company is the durable executable entity. Evaluation as Model × Harness × Policy × Environment — with capability in the harness beating raw model size.

Download PDF Back to capabilities