The Fin AI
Open-world evaluation · sessions, accepted →
FinCosmos
The investor signed off.
The backtest was wrong.
Agents now search the Web, write code, and hand back backtests that people act on. FinCosmos leaves the investor's intent, the data, and time open, and then checks what the agent delivers. Every agent, , ran the same ten tasks.
signed off by the investor
reach the accepted orbit
signed offnot signed offno-tool baselinefarther out = higher quality before vetoes
Signing off is not evidence that the work is right.
Replay
The investor signed off. Find what went wrong.
Real sessions from the benchmark, trimmed. In each one the simulated investor was satisfied. Mark what you think the delivery got wrong, then open it.
Leaderboard
Ten tasks, three sessions each, no acceptance yet
Ranked by acceptance rate, then by quality before vetoes. Open a row for the breakdown. Click a heading to rank by it.
Accepted the investor is satisfied and none of the 15 vetoes fails. Signed off the investor is satisfied. Quality pre-veto quality on 0 to 100, with a 95% bootstrap interval over tasks; the five category scores use the same scale. Vetoes failed per session, of 15. Asked share of the details the investor did not state up front that the agent raised. Cost API-equivalent USD per session.
Darker blue means higher.
Silent failures
Graded by sign-off, they pass. Checked, none do.
The same sessions under three graders. Switch the grader and watch what counts as a success.
Why the signed-off sessions fail
Share of the signed-off sessions that fail each group of checks. Program checks catch every one.
Workflow
Agents break at both ends of the work
Sessions passing each step, of 30 per agent. The last column follows the four agents' 120 sessions through every step in turn.
They ask about half of what they were never told
The opening request states about three of the strategy's seven details. The investor answers what the agent asks and corrects what it gets wrong, but volunteers nothing.
Unstated details the agent raised
Never-stated details the delivery gets wrong
Learning from feedback
With FinCosmos feedback, an agent learned all ten
Claude Code with a persistent memory practiced each task until its first acceptance. After every graded session it read the leak-free report, which gives scores and evidence but never reveals the hidden strategy, and edited its memory.
Each taskMedian
Failed vetoes per session: in the first session, averaged over the first and the last five sessions before acceptance, and in the accepted session. Hover a line for the task.
Benchmark
The intent, the data, and time stay open
Existing benchmarks state the task, hand over curated data, or enforce the cutoff for the agent. Each of these removes a silent failure instead of measuring it.
Fifteen vetoes, and one is enough to fail
A session is accepted only when the investor is satisfied and every veto passes. The count shows how many of the signed-off sessions fail each veto.