AIDAN KARASBoston UniversityContact
← Projects

arbiter — can an LLM agent beat a plain model on SEC filings?

2026-09 → now · active · Sole author

A benchmark that gives an agent and a baseline identical point-in-time data. Result so far: no measurable skill from either.

Pythonscikit-learnGitHub Actions

Repository

Question

Does an LLM agent beat a conventional model at forecasting short-horizon equity moves from SEC filings, when both see byte-identical, point-in-time data?

How it works

  • Each filing is frozen into an immutable, content-hashed evidence packet stamped with its acceptance time.
  • Domains: insider trades (Form 4) and 8-K red flags.
  • Four arms: baseline (logistic regression), dl, agent, and arbiter (cross-family review).
  • Scored by information coefficient and calibration on a chronological split with an embargo, so nothing from the future leaks in.

Result

On the walk-forward evaluation the information coefficient was +0.029 (t = 1.52): not distinguishable from zero. I report it as a null result. The value of the project is the harness that makes a claim like this checkable.

Status

Active. Python 3.12, MIT licence, CI on GitHub Actions.