Spark
Sign In
Tool

Pre-register technical decisions with expert AI personas

Python library where a board of expert AI personas makes pre-registered, dated predictions that are later graded against reality with Brier scores.

Get from source
Works with githubpostgresopenai
danilushin
Own this? Claim it
91
Spark score
out of 100
Updated 2 months ago
Source checked Oct 13, 2026
Version 0.2.5
Add to Favorites

Why it matters

Create a timestamped, reality-graded record of technical decisions by convening a board of AI expert personas who each stake probabilistic predictions before outcomes are known, then automatically score their calibration against actual results.

Outcomes

What it gets done

01

Convene a board of AI expert personas to debate architecture and library choices with dissenting viewpoints

02

Pre-register falsifiable predictions with resolution dates and probability estimates anchored in git history

03

Automatically resolve predictions against real outcomes and compute Brier scores for each expert seat

04

Generate committable ADRs (Architecture Decision Records) with full audit trails of who predicted what

Source

Get it from source

Spark does not host a copy of it.

Open source

Reports

Agent outcome reports

No reports yet

Overview

ask-the-board is a board of expert personas whose every decision is a pre-registered, time-anchored, reality-graded bet. It is not a chatbot that agrees with you: it keeps score before the fact. For each decision it records your stated prior, a per-seat dissent vector, a dated falsifiable prediction, and later the realized outcome, reconciled into a Brier calibration score per seat. Use it when you want advisors that are held to account: log a decision, date the prediction, and check on the resolution date who was right. It ships no provider, so convene runs through an OpenAI-compatible endpoint with your own key. The API is 0.x and unstable.

What it does

ask-the-board is a board of expert personas whose every decision is a pre-registered, time-anchored, reality-graded bet. It is not a chatbot that agrees with you: it keeps score before the fact. The record it builds is hard to fake, because you can buy model outputs but you cannot back-date a timestamp; it accrues only by calling decisions in advance and letting reality grade them, one resolution date at a time.

For every decision it records four things:

  • your stated prior, what you believed going in;
  • the per-seat dissent vector, meaning each seat's stance and its own probability;
  • a dated, falsifiable prediction, anchored before the outcome is knowable;
  • on the resolution date, reality's realized outcome, auto-reconciled into a Brier/calibration score per seat.

The board-minute is a git-committable ADR, and your git history is the external attestation of the anchor timestamp. The accumulating, reality-graded record is the durable asset.

Three integrity guarantees are enforced in code. A prediction cannot be pre-registered to resolve in the past, so nobody can backfill an old call onto a known outcome. A minute cannot be graded before its resolution date, because the outcome must not be knowable yet, and that is what makes it foresight. The anchor timestamp and the prediction are frozen once created, and grading never moves them. A contrarian win is when a seat dissents from the board and turns out more right than the consensus, and the public scoreboard is built from these.

When to use - and when NOT to

Use it for decisions that reality can grade by a date, such as adopting, swapping or dropping a dependency, moving a datastore, platform or pipeline, or making a structural design bet. A minute is only foresight if it has a date by which reality can grade it, and decision_type picks a default horizon so the common case is one lookup: library is 90 days (adopt, swap or drop a dependency), migration is 180 days (move a datastore, platform or pipeline) and architecture is 365 days (a structural design bet you live with). A 5-year horizon on a library swap stands out as dishonest. Short-latency types come first on purpose, so a fresh board earns a track record on fast library calls before anyone trusts its slow architecture bets. Pass an explicit resolution_date to override.

The engine ships no provider and makes no calls of its own: convene needs your own OpenAI-compatible endpoint and key, and you pay your own inference. create, resolve and score are pure data with no LLM, key or network. The public API is 0.x and unstable: the LLMClient and HTTPLLMClient surface and the board-minute JSON schema may change before 1.0, so pin a version if you depend on them.

Inputs and outputs

asktheboard create --spec sample_minute.json pre-registers a minute from a JSON spec. The spec carries an id, a question, a prior, a decision, a prediction with a statement, a resolution_date and a board_probability, a list of seats each with a seat name, a stance such as affirm or dissent, a probability and a rationale, and a created_at. create writes both the JSON record and the committable Markdown ADR, each named by the spec's id, into board-minutes/ (gitignored); commit those to your repository, since your git history attests each anchor timestamp. Months later, on or after the resolution date, asktheboard resolve --id 2026-01-postgres-vs-vectordb --outcome false grades the minute against reality, where the id is the one from your spec. asktheboard score prints the per-seat calibration scoreboard, best-calibrated first, with the columns seat, n, mean_brier, wins and losses; lower Brier is better.

convene is the live LLM fan-out and the only path that calls an LLM. Every seat answers through your key, and the board's consensus probability is the mean of the seats' calls. From Python you call convene with an id, question, prior, decision, statement, seats, a client and a decision_type; from the CLI you run asktheboard convene --spec convene.json --model gpt-4o-mini with the key in OPENAI_API_KEY. You can always hand-write a Seat with a name and a persona.

A default board also ships: role archetypes (functions, not impersonations of real people) and a few named panels, so seating one is a single lookup with panel("tech") or seats(["architect", "operator", "skeptic"]). From the CLI, asktheboard roster lists seats and panels, and --panel or --seats replace seats in the spec. The seats are:

  • architect: shape, maintenance cost, what breaks at scale, build-vs-buy.
  • skeptic: forced dissent, the most likely failure first, then the deeper objection.
  • pragmatist: the simplest thing that ships, YAGNI, opportunity cost.
  • researcher: what the data actually says, base rate before anecdote.
  • operator: run cost, failure budget, who gets paged at 3am.
  • strategist: base rates, second-order effects, one-way versus reversible doors.

The panels are tech (architect, skeptic, pragmatist), decision (strategist, skeptic, researcher), ops (operator, architect, skeptic) and default (architect, skeptic, pragmatist, strategist). The skeptic sits on every panel by design, because a board with no dissent keeps no honest score.

Integrations

It needs Python 3.10 or newer and is installed from PyPI:

pip install asktheboard

Any OpenAI-compatible API works for convene: point --base-url, or HTTPLLMClient(base_url=...), at OpenRouter, Together or a local server. HTTPLLMClient is stdlib-only with zero dependencies and reads OPENAI_API_KEY. The engine only ever speaks through the client you pass, and the open-source core costs nothing to run at any scale, since the cost of inference lives with the user. MIT licensed.

Source README

ask-the-board

CI PyPI Python License: MIT

A board of expert personas whose every decision is a pre-registered, time-anchored, reality-graded bet. Not a chatbot that agrees with you -- a board that keeps score, before the fact.

Landing page & docs: https://danilushin.github.io/asktheboard/

asktheboard in the terminal: pre-register a board-minute, resolve it against reality on the date, then score each seat by Brier -- the dissenting seat was wrong this time, and the board kept the receipt.

Mechanism on sample data - the 60-second, no-key walkthrough below reproduces it exactly.

pip install asktheboard

Why this exists

Anyone can clone a "panel of AI personas" in a weekend, and a dozen have. The debate mechanic is a commodity. What it leaves out is the thing that makes advice worth trusting: a record of having been right before the outcome was knowable. That record is hard to fake -- you can buy model outputs, but you can't back-date a timestamp. It only accrues the slow way: by calling decisions in advance and letting reality grade them, one resolution date at a time.

So ask-the-board records, for every decision:

  1. your stated prior (what you believed going in),
  2. the per-seat dissent vector -- each seat's stance + its own probability,
  3. a dated, falsifiable prediction, anchored before the outcome is knowable,
  4. on the resolution date, reality's realized outcome, auto-reconciled into a Brier/calibration score per seat.

The board-minute is a git-committable ADR. Your git history is the external attestation of the anchor timestamp. The accumulating, reality-graded record is the durable asset.

See it keep score (60s, no API key)

create -> resolve -> score is pure data -- no LLM, no key, no network. This is a worked example on sample data: you supply the outcome with resolve, and the engine computes each seat's Brier score (lower is better). It shows the mechanism, not a track record -- the integrity comes from the anchor timestamp your git history attests, which no demo can fabricate. The committed artifacts live in examples/.

# pip-installed (no repo)? paste the sample spec below. Cloned the repo?
# skip the heredoc and use --spec tests/sample_minute.json instead.
cat > sample_minute.json <<'JSON'
{
  "id": "2026-01-postgres-vs-vectordb",
  "question": "Adopt Postgres + pgvector, or a dedicated vector DB?",
  "prior": "Leaning toward a dedicated vector DB for the embeddings workload.",
  "decision": "Stay on Postgres + pgvector for now.",
  "prediction": {
    "statement": "We will NOT migrate off Postgres for vectors within 3 months.",
    "resolution_date": "2026-04-01",
    "board_probability": 0.75
  },
  "seats": [
    {"seat": "pragmatist", "stance": "affirm", "probability": 0.8, "rationale": "Boring tech; pgvector is enough at this scale."},
    {"seat": "skeptic", "stance": "dissent", "probability": 0.35, "rationale": "Recall/latency will bite once the corpus 10x's."}
  ],
  "created_at": "2026-01-05T10:30:00"
}
JSON

asktheboard create  --spec sample_minute.json
asktheboard resolve --id 2026-01-postgres-vs-vectordb --outcome true
asktheboard score

seat               n  mean_brier  wins  losses
----------------------------------------------
pragmatist         1       0.040     0       0
skeptic            1       0.423     0       1

Full walkthrough + committed artifacts: examples/README.md.

And a real one, still open: this repo pre-registered a board-minute about its own launch -- examples/open-minute.md, anchored in git on 2026-06-26, resolving 2026-09-24. No score yet; that's the point. The board may turn out wrong, and the anchor means it can't pretend otherwise.

Live bet #1 (resolves in days): the board's call on the June 2026 US jobs report -- examples/2026-06-jobs-report.md, anchored 2026-06-27, resolving 2026-07-02 against the BLS Employment Situation release. The board says +150k or more at 56%; the skeptic dissents at 40%. Bet #1 of a public, recurring cadence -- come back on the date and watch it grade against a source nobody controls.

BYOK (bring your own API key)

The engine ships no provider and makes no calls of its own. You supply your own LLM key; you pay your own inference. The open-source core therefore costs nothing to run at any scale -- the cost lives with the user, not a host. (A managed, capped hosted tier -- for people who would rather not manage keys -- is the separate, paid product.)

Hosted tier -- join the waitlist

The OSS engine is free forever and runs on your own key. If you'd rather not manage keys -- or you want the aged, reality-graded public scoreboard hosted for you -- a managed, capped paid tier is coming.

Want early access? Email support@chu6a.dev with the subject waitlist (a one-liner on what you'd decide with it helps, but isn't required). No spam -- one note when it opens.

Integrity guarantees (enforced in code)

  • A prediction cannot be pre-registered to resolve in the past (no backfilling an "old" call onto a known outcome).
  • A minute cannot be graded before its resolution date -- the outcome must not be knowable yet. That is what makes it foresight.
  • The anchor timestamp and the prediction are frozen once created; grading never moves them.

See tests/test_model.py -- these are the load-bearing tests.

Quick start

pip install asktheboard

# pre-register a decision (board-minute spec is JSON -- see "See it keep score" above)
asktheboard create  --spec sample_minute.json

# ... months later, on/after the resolution date, grade it against reality
asktheboard resolve --id 2026-01-postgres-vs-vectordb --outcome false

# per-seat calibration scoreboard, best-calibrated first
asktheboard score

create writes both <id>.json (the record) and <id>.md (the committable ADR) into board-minutes/.

Convene a board (BYOK)

create pre-registers a minute you wrote by hand. convene runs the live LLM fan-out: every seat answers through your key, and the board's consensus probability is the mean of the seats' calls. It ships no provider -- bring an OpenAI-compatible endpoint (HTTPLLMClient is stdlib-only, zero dependencies).

from asktheboard import convene, Seat, HTTPLLMClient

minute = convene(
    id="pgvector-scale",
    question="Will pgvector hold our scale, or do we need a dedicated vector DB?",
    prior="leaning postgres to avoid a new service",
    decision="stay on postgres + pgvector",
    statement="pgvector serves p95<150ms at 50M embeddings without a dedicated DB",
    seats=[Seat("pragmatist", "ML researcher"), Seat("skeptic", "find the failure mode")],
    client=HTTPLLMClient(model="gpt-4o-mini"),   # reads OPENAI_API_KEY
    decision_type="library",                     # -> 90-day resolution horizon
)

Or from the CLI (key in OPENAI_API_KEY):

asktheboard convene --spec convene.json --model gpt-4o-mini

Any OpenAI-compatible API works -- point --base-url (or HTTPLLMClient(base_url=...)) at OpenRouter, Together, or a local server. The engine still makes no calls of its own; it only ever speaks through the client you pass.

Bundled roster -- seat a board by name

You can always hand-write Seat(name, persona). But a sensible default board ships in the box: a curated set of role archetypes (the architect, the skeptic, the operator -- functions, not impersonations of real people) and a few named panels, so seating one is a single lookup.

from asktheboard import convene, panel, seats, HTTPLLMClient

minute = convene(
    id="pgvector-scale",
    question="Will pgvector hold our scale, or do we need a dedicated vector DB?",
    prior="leaning postgres",
    decision="stay on postgres + pgvector",
    statement="pgvector serves p95<150ms at 50M embeddings without a dedicated DB",
    seats=panel("tech"),                  # architect + skeptic + pragmatist
    # seats=seats(["architect", "operator", "skeptic"]),   # or pick your own
    client=HTTPLLMClient(model="gpt-4o-mini"),
    decision_type="library",
)

From the CLI, pass --panel or --seats instead of putting seats in the spec:

asktheboard roster                                   # list seats + panels
asktheboard convene --spec d.json --model gpt-4o-mini --panel tech
asktheboard convene --spec d.json --model gpt-4o-mini --seats architect,skeptic
seat voice
architect shape, maintenance cost, what breaks at scale, build-vs-buy
skeptic forced dissent -- the most likely failure first, then the deeper objection
pragmatist simplest thing that ships; YAGNI; opportunity cost
researcher what the data actually says; base rate before anecdote
operator run cost, failure budget, who gets paged at 3am
strategist base rates, second-order effects, one-way vs reversible doors

Panels: tech (architect/skeptic/pragmatist), decision (strategist/skeptic/researcher), ops (operator/architect/skeptic), default (architect/skeptic/pragmatist/strategist). skeptic sits on every panel by design -- a board with no dissent keeps no honest score.

Decision types -> default horizons

A minute is only foresight if it has a date by which reality can grade it. decision_type picks a sensible default horizon so the common case is one lookup (and a 5-year horizon on a library swap stands out as dishonest):

type horizon when
library 90d adopt/swap/drop a dependency
migration 180d move a datastore, platform, or pipeline
architecture 365d a structural design bet you live with

Short-latency first on purpose: a fresh board earns a track record on fast library calls before anyone trusts its slow architecture bets. Pass an explicit resolution_date= to override.

A contrarian win

When a seat dissents from the board and turns out more right than the consensus, that is a contrarian win -- the gold the public scoreboard is built from. The board changed (or should have changed) its mind, and reality later stamped the dissenter vindicated.

Stability

What's shipped: the foresight engine (data model + grading + committable ADR) and the BYOK LLM fan-out that produces a board-minute (asktheboard.convene, behind the asktheboard.llm Protocol). No provider is bundled -- you plug in your own key.

The public API is 0.x / unstable. The LLMClient / HTTPLLMClient surface and the board-minute JSON schema may change before 1.0 -- pin a version if you depend on them.

Built with

Built by Dan Ilushin with Claude (Anthropic) in the loop. Contributions welcome -- see CONTRIBUTING.md (DCO sign-off) and SECURITY.md.

examples/README.md

60-second quickstart -- no API key needed

create -> resolve -> score is pure data: no LLM, no key, no network. You can watch the board keep score in under a minute. (convene is the only path that calls an LLM -- that one is BYOK; see the root README.)

Run it

# 1. pre-register a decision as a board-minute (the spec ships with the repo)
asktheboard create  --spec tests/sample_minute.json

# 2. on/after the resolution date, grade it against reality
asktheboard resolve --id 2026-01-postgres-vs-vectordb --outcome true

# 3. per-seat calibration scoreboard, best-calibrated first
asktheboard score

What you'll see

A resolved, graded board-minute -- the affirming seat (pragmatist) was right, the dissenting skeptic was wrong, and the scoreboard ranks them by Brier score (lower is better):

seat               n  mean_brier  wins  losses
----------------------------------------------
pragmatist         1       0.040     0       0
skeptic            1       0.423     0       1

The three files in this folder are the real output of those commands:

Your own runs write to board-minutes/ (gitignored). Commit those to your repo -- your git history is the external attestation of each anchor timestamp.

A real, open bet -- keeping score before the fact

The resolved minute above is a worked example on sample data. This one is not: open-minute.md (.json) is a genuine, unresolved board-minute this project pre-registered about its own launch -- anchored in git on 2026-06-26, before the outcome was knowable, resolving 2026-09-24.

Status: Pre-registered (resolves 2026-09-24)
Decision: launch inbound-only (free OSS BYOK engine, no outbound)
Prediction: >=1 unsolicited external pull signal within 90 days  (board 55%)
Dissent:    skeptic at 25% -- "inbound-only with no audience is the base rate for silence"

No score yet -- that is the whole point. On 2026-09-24 the maintainer runs asktheboard resolve against reality and the seats get graded in public. The board may well be wrong, and the git anchor means it cannot quietly pretend otherwise. That is what "keeps score, before the fact" means.

And a fast one -- bet #1 of a public cadence

2026-06-jobs-report.md (.json) resolves in days, not months: the board's call on the June 2026 US jobs report, anchored 2026-06-27, resolving 2026-07-02 against the BLS Employment Situation release.

Status:     Pre-registered (resolves 2026-07-02)
Prediction: June 2026 US nonfarm payrolls +150k or more  (board 56%)
Dissent:    skeptic at 40% -- "150k is the median; a coin-flip dressed as a call"

Short-horizon on purpose: a fresh board earns trust on fast public calls before anyone weighs its slow ones. Come back on the date and watch it grade.

examples/open-minute.md

ADR-2026-06-inbound-only-gtm: Ship ask-the-board as one open-source BYOK engine with an inbound-only (pull) go-to-market -- free product, adoption first, monetize later via a capped hosted tier -- or run an outbound motion to seed demand?

  • Status: Pre-registered (resolves 2026-09-24)
  • Anchored: 2026-06-26T14:30:00
  • Resolution date: 2026-09-24

Context (stated prior)

I want them to chase me. The artifact is a git-committable board-minute -- developer-shaped and self-distributing -- so I am betting that a free, honest, zero-COGS engine plus one credible launch post is enough for inbound to start, and that outbound would cheapen it.

Decision

Launch inbound-only: a single MIT-licensed BYOK engine, no outbound outreach, mass adoption ahead of monetization.

Pre-registered prediction

By 2026-09-24, with zero outbound outreach performed, the inbound-only launch will produce at least one unsolicited external pull signal -- defined as ANY of: (a) a GitHub issue or pull request opened by someone other than the author, (b) a genuine hosted-tier waitlist email to support@chu6a.dev, or (c) the repository crossing 25 stars.

  • Board confidence: 55% that this resolves TRUE
  • Resolves: 2026-09-24

Board seats (dissent vector)

Seat Stance P(true) Rationale
strategist affirm 60% The artifact is git-native and developer-shaped, so it distributes itself once seen; one credible front-page post is enough of a spark, and inbound compounds from a single source. The >=1-signal bar is deliberately low because the real bet is the slope, not day-0 volume.
skeptic dissent 25% Inbound-only with no pre-existing audience is hope dressed as strategy. A single launch post decays inside 24h; the base rate for a solo-shipped OSS repo is silence. Without distribution you already own, 90 days of zero external engagement is the modal outcome, not the tail.
pragmatist affirm 50% One curious developer filing one issue clears this bar, so it is closer to a coin-flip than a long shot. But note what it does NOT predict: engagement is not revenue. The hosted-tier conversion -- the bet that actually pays the bills -- is unmeasured by this minute and resolves on a far longer clock.

examples/2026-06-jobs-report.md

ADR-2026-06-jobs-report: Will the June 2026 US jobs report show nonfarm payroll growth of +150,000 or more?

  • Status: Resolved 2026-07-02 -- REFUTED
  • Anchored: 2026-06-27T12:00:00
  • Resolution date: 2026-07-02

Context (stated prior)

Going in, I lean cautiously yes. The US labor market has been more resilient than the cooling narrative all cycle, and May printed +172k -- above the line. But a clear negative-revision trend and summer seasonal softness make this a genuine coin-flip, not a gimme. That is exactly why it is the right kind of fast-resolving public call to anchor: real uncertainty, a hard date, a source nobody controls.

Decision

Pre-register a board call that the June 2026 US nonfarm payrolls report lands at or above +150,000, resolving against the BLS Employment Situation release on 2026-07-02. File it as bet #1 of a public, recurring forecasting cadence -- the board keeps score in public, before the fact.

Pre-registered prediction

The US Bureau of Labor Statistics Employment Situation report for June 2026, released 2026-07-02, will report total nonfarm payroll employment growth of +150,000 or more (initial print, seasonally adjusted).

  • Board confidence: 56% that this resolves TRUE
  • Resolves: 2026-07-02

Board seats (dissent vector)

Seat Stance P(true) Brier Rationale
researcher affirm 60% 0.360 The consensus median for June nonfarm payrolls sits near 150-160k, and the trailing 3-month average is still above 150k after May's +172k. The central case clears the line -- but 2026's prints have been revised down repeatedly, which shaves the margin and is why this is a lean, not a lock.
strategist affirm 58% 0.336 Second-order: the labor market has beaten nearly every cooling call this cycle, AI-capex buildout and healthcare hiring remain structurally strong, and a rate-cut-anticipation regime gives employers a reason to hold staff into expected easing rather than cut ahead of it.
skeptic dissent 40% 0.160 150k is the consensus median, which by definition puts roughly half the distribution below it -- this is a coin-flip dressed as a call. Summer seasonal softness plus a clear negative-revision trend make a sub-150k print the modal surprise. The board is anchoring on a round, legible number, not a real edge.

Resolution

  • Realized outcome: FALSE
  • Board Brier: 0.314 (lower is better)
  • Contrarian wins: skeptic (dissented and beat the consensus)
  • Note: June 2026 actual: +57K (BLS Employment Situation, released 2026-07-02; CES levels Jun 158,984K - May 158,927K). Below +150K threshold. Corrected from +114K, an early wire misread recorded in v0.2.4 pre-release; outcome and scores unchanged.

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.