Bay Area Engineer Bicheng (Kenneth) Wang Builds the Missing Receipt for AI Decisions

SAN FRANCISCO, Oct. 23, 2026 — Picture a product team late on a Friday. An A/B test has been running for two weeks. A dashboard shows the new version ahead, a model summarizes the result as a clear win, and someone clicks ship. Six months later an auditor asks a simple question: why was that change released, and could the decision be made again from the same evidence?

In most companies the honest answer is no. The dashboard has been recalculated. The model has been updated. The logs say a decision happened, but they do not hold what is needed to recompute it.

Bicheng (Kenneth) Wang, a Bay Area engineer working in machine learning, has spent the past two years building the piece that is missing from that story. His open-source project ai-decision-system produces what amounts to a receipt for an AI-assisted decision: a record that a third party can take offline, months later, and use to check the outcome against the original inputs and rules.

The project's documentation opens with five words that summarize the whole effort: "Model output is not authority."

From suggestion to decision

For most of the last decade, machine learning in business made suggestions. A model ranked products, flagged a transaction, or drafted a reply, and a person decided what to do. That boundary is moving. AI systems now approve support escalations, score experiments, route requests between teams, and recommend whether a new model version is safe to release.

Each of those moments is a go/no-go decision with real consequences. Wang's view, written into the repository, is that such decisions need more than a log line. "Promotions, approvals and releases need a record that a third party can re-check against frozen inputs and rules," the README states. "Dashboards and logs can describe what happened; re-verifiable evidence also preserves what is needed to recompute the decision."

The GitHub repository dates to November 2024 and carries research history back to 2020. It now holds more than 800 commits, has been forked a dozen times, and had 237 stars on GitHub as of Oct. 23, 2026.

Four checkpoints along the path of a decision

The repository's decision-record work centers on four independent components. Each sits at a point where an AI-assisted choice becomes a commitment, and each emits its own checkable record.

The first checkpoint is the model's answer itself. A tool called Jev decision asks the model for a typed answer instead of free text: a yes or no, a choice from a fixed list, or a score. It then gates that answer against labeled quality requirements and reports one of three outcomes, PASS, FAIL or UNGATED. It also defines what happens when a model refuses to answer or reports no usable confidence, two cases that ordinary pipelines often let slip through silently. The project is careful to note that confidence reported by a provider "is not a calibrated safety guarantee."

The second checkpoint is human approval. decision-control freezes the routing policy in force when a decision is proposed, records the proposer's label, and adds the human review to the same record. For decisions that need review, the proposer and approver labels must differ; confirming who is behind each label is left to the host system. In its unreleased 0.2.0 version, the tool can capture a decision made by the widely used policy engines OPA and Cedar, and it ships an approvals example for LangGraph, a popular framework for building AI agents. The resulting record can be replayed later with a single command.

The third checkpoint is the experiment. Experiment gate reads the exported summary of an A/B test and returns one of four verdicts: SHIP, HOLD, ROLLBACK or INVALID. It does not run the experiment and does not deploy anything. Its job is to turn the numbers into a decision file that anyone can verify offline. The development version can import supported results exports from GrowthBook, an open-source experimentation platform, using an explicit metric mapping.

The fourth checkpoint is the model release. RELIA runs release and update gates for models in a pinned environment, designed so that two fresh runs in the same pinned environment produce byte-identical results files. Unlike the other three, which have 0.1.0 releases on PyPI, RELIA is an unreleased candidate, marked human-gated, and its folder grants no license yet.

The three smaller packages run on the Python standard library alone, with no outside dependencies at runtime. That choice keeps them small enough to audit and easy to adopt inside companies with strict software policies.

The engineering of trust

What sets the project apart is how seriously it treats the record itself.

Records are hashed so that any change can be detected. The current schemas of the three smaller packages use RFC 8785, the JSON Canonicalization Scheme, a standard that makes the same data produce the same bytes on any machine and in any programming language. Each of the three smaller packages carries an identical copy of the canonicalization code, tested against a shared set of reference vectors generated by Node.js. A separate Node.js CI job then re-checks the schemas and digests of those packages' committed conformance vectors, so the format is validated by code that shares nothing with the Python implementation that wrote it.

Records can also be signed. The project supports in-toto statements inside a DSSE envelope, the same formats the software supply-chain community uses to prove how a build was produced, with Sigstore signing available through its cosign tool.

Just as notable is what the documentation refuses to claim. It states that "hashes detect change, not forgery." It states that "a signature shows the holder of a key signed the bytes, not who that person is." It describes its bundled examples as software checks on synthetic data and states that no outside use is claimed. In a field where tools routinely promise more than they deliver, the documentation separates software checks on synthetic data from evidence of adoption, model quality or production safety. For a system whose whole purpose is evidence, that restraint is the point.

Wang positions the work as a complement to the platforms teams already run. Experiment platforms such as GrowthBook, Statsig and Eppo compute the statistics. Policy engines such as OPA and Cedar decide who may do what. Evaluation tools such as promptfoo and OpenAI Evals score prompts and model behavior. ai-decision-system adds the layer those tools leave to the user: retained records whose inputs, rules and outcome can be re-checked offline.

A second question: do AI teams actually help?

In April 2025, Wang started a second project that applies the same habit of mind to a different question.

Coding agents such as Anthropic's Claude Code and OpenAI's Codex can now write, test and revise software on their own. A fast-growing group of tools runs several of them together, on the theory that a team of agents will outperform one. Wang's project multi-agents is built around a sentence from its roadmap: dozens of tools run several coding agents in parallel, and "none of them tells you whether the team beat a single agent, or what the extra agents cost."

The published research does not settle it. A study of 260 configurations across six benchmarks found multi-agent setups 2 to 15 percent worse, in relative terms, than single agents on a 20-task subset of SWE-bench Verified, a standard coding benchmark. Other teams reported gains: one design with a coordinator, researcher, engineer and reviewer reached 72.2 percent on the same benchmark, but only 0.4 points above the strongest GPT-5-family single-agent baselines in that comparison. And the benchmark itself has a problem. OpenAI stopped reporting SWE-bench Verified in February 2026 because of contamination and flawed tests; contamination means models may have seen the answers during training.

multi-agents answers the question by measuring it. A developer points the tool at a repository and a task and chooses a team. The options are a single agent as the baseline, a coder paired with a reviewer, several workers in parallel with a judge who picks the best patch, or a planner who splits the task into subtasks, hands them to workers and merges the results. Workers can be Claude Code, Codex, or any model through the OpenAI Agents SDK. Each agent works in its own copy of the code, a git worktree, so the developer's files are untouched until they choose to apply the result.

A completed run returns a patch and a per-call record of tokens, cost and seconds. In one run shown in the documentation, a two-worker team with a judge produced a patch for a failing-test task in 85 seconds for about three and a half cents. When a cost cannot be determined, the run is labeled unpriced rather than guessed. A budget flag stops new agent work once a dollar limit is reached. With --issue and --pr, a GitHub issue goes in and a draft pull request comes out; adding --check runs the project's tests against the final patch and records the result.

A benchmark built to resist contamination

The benchmark at the center of multi-agents is designed around the weaknesses of earlier ones. It draws tasks from SWE-bench-Live and lets researchers keep only issues created after each model's training cutoff. The current filtered pool covers Rust, Java, C, C++, C# and JavaScript. Each task runs in a fresh cloud sandbox with git history stripped, so an agent cannot find the original fix in the repository's history. Patches are graded with the dataset's hidden tests.

The statistics are chosen to answer the real question. Each team is compared with a solo agent on the same tasks, task by task, using an exact McNemar test. Results carry 95 percent confidence intervals, along with cost per task, cost per resolved task and wall time. Teams that time out stay in the comparison and are graded on what they left, rather than dropped, so a slow team cannot make itself look better by finishing only the easy tasks.

As with ai-decision-system, the project is direct about where it stands. Its status table lists the team runner and plugins as working and smoke-tested with real tools, the benchmark as built and tested offline, and the results as pending the first pilot run.

Why it matters

Wang's two projects address one problem from two sides. As AI moves from suggesting to deciding, organizations need two things they mostly lack today: proof of what an automated system decided, and honest measurement of whether a more elaborate AI setup is worth what it costs.

Those needs reach well beyond any single company. Financial institutions answer to auditors, healthcare systems to regulators, and public agencies to the people they serve. Each will need to show how an automated decision was made, using evidence that someone outside the organization can verify. Engineering leaders deciding how much to spend on AI tooling need numbers that were not chosen to flatter the result.

Both projects are public, and apart from the unreleased RELIA candidate they are released under the MIT license, so any company, university or government team can adopt, inspect or extend them. Together they had 262 stars on GitHub as of Oct. 19, 2026. Their larger contribution is a working example of a standard: every automated decision should come with a receipt, and every claim about AI performance should be one a stranger can check.

About Bicheng (Kenneth) Wang

Bicheng (Kenneth) Wang is a Bay Area engineer working in machine learning, AI evaluation and quantitative finance. His open-source work includes ai-decision-system, replayable records for AI-assisted decisions; multi-agents (github.com/BichengWang/multi-agents), a runner and benchmark for teams of coding agents; RL-pricing (github.com/BichengWang/RL-pricing), a reproducibility lab for trading AI; and financial-agent (github.com/BichengWang/financial-agent), a research and paper-trading platform with gated execution. He is the co-author, with Xinyi Zhang, of the arXiv preprint "Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining" (arXiv:2609.31870).

Learn more: github.com/BichengWang