Data Stack for AI Trading Agents

An AI trading agent does not “see the market.” It sees whatever your data stack delivered on time, correctly joined, and correctly timestamped. When that pipeline lies, autonomy becomes amplified error.

This article expands the data plane from our architecture and Steps 1–2 of how to build into a practical data stack for continuous crypto markets—feeds, quality, features, and fail-closed behavior.

Why data kills agents first

Models often degrade smoothly. Data fails discontinuously. Five failure modes show up in postmortems more than “bad alpha”:

• Silent gaps — WS reconnects without backfill; the agent treats absence of prints as calm
• Clock skew — exchange event time and local receive time mixed in one feature
• Sequence breaks — dropped L2 diffs applied as if the book were absolute
• Schema drift — vendor renames a field; parser coerces null → 0
• Research lookahead — revised candles, corporate actions, or post-event text in “historical” runs

If you cannot explain what the agent knew and when, you cannot evaluate it—and you should not scale it.

Reference stack layers

Think in stages with hard interfaces. Each stage has an owner, a schema, and a failure mode.

Venues / chains / files / APIs
        ↓
Ingress collectors (WS, REST, pollers)
        ↓
Normalizer → canonical events + dual clocks
        ↓
Quality service  → healthy | delayed | dead
        ↓                    ↓
Online feature store    Historical lake / replay
        ↓                    ↓
Live policy + risk      Research / training / eval
        ↓
Context pack (optional LLM tools) → structured only
        ↓
Decision ledger (inputs + feature versions + quality)

Ingress

Collect what the mandate requires—not everything a vendor sells. Typical live agents need trades and/or books, mark prices, account balances, open orders, and fills. Perps agents usually add funding, open interest, and liquidations. Prefer:

• Idempotent consumers (safe restarts)
• Durable buffers so a process crash does not invent history
• Explicit backfill paths after disconnects

Normalization & instrument master

Map every venue into a canonical event schema and a single instrument master (internal symbol → venue symbols, contract size, tick size, quote currency). Without a master, multi-venue agents double-count or mis-size.

Minimal event fields worth standardizing:

{
  "event_type": "trade|bbo|book_diff|funding|order|fill",
  "instrument_id": "BTC-USDT-PERP",
  "venue": "example_ex",
  "seq": 18482910,
  "exchange_ts": "2026-07-11T12:00:00.123Z",
  "receive_ts": "2026-07-11T12:00:00.141Z",
  "payload": { }
}

One schema beats twelve ad-hoc dicts when you add the thirteenth venue—or when you try to replay last month honestly.

Quality service

Continuously score feeds per venue and instrument. Minimum states: healthy, delayed, dead. Useful inputs:

• Heartbeat / last message age
• Gap counters and sequence jumps
• Book integrity checks (crossed market, empty book after snapshot)
• Cross-check vs a secondary source when available

Expose flags to both policy and risk. Thresholds are product choices—document them. Example starting points (tune per venue): mark delayed if BBO age > 2s in a liquid pair; dead if age > 10s or sequence recovery fails.

Online features vs historical lake

Online features power live decisions: imbalance, short-horizon volatility, microprice, inventory, spread, funding z-score. They must be computable at decision latency with the same definition used offline.

A historical lake stores raw and curated events for research, evaluation, and incident replay. Shared schemas and versioned feature code prevent train/serve skew—the silent cousin of lookahead bias.

Context for planners and LLMs

Tool-using agents need curated context—not raw firehoses. Build structured tool outputs (events, metrics, doc snapshots) as in LLMs for crypto market analysis. Validators sit between text and anything that can influence size.

Feed catalog by need

Map feeds to agent type: a market-making agent without books is cosplay; an LLM research agent without citations is a story generator.

Worked example: what the agent “knows” at T

Suppose decision time T for a perps inventory policy. A honest stack might assemble:

• BBO and last 5s trades with exchange_ts ≤ T and quality = healthy
• Position and open orders from the account stream reconciled to the same clock
• Funding rate as-of last published print before T
• Feature vector version feat_v3.2 (hash of code + params)
• Quality snapshot: venue A healthy, secondary mark delayed → policy already size-capped

That bundle—not “the model”—is what you must log under a decision ID if you want learning loops instead of folklore.

Clocks, alignment, point-in-time

Point-in-time (PIT) means: a feature at decision time T uses only information available at or before T under your published rules.

• Store exchange time and receive time separately; never overwrite one with the other.
• Default as-of rule for market features: include events with exchange_ts ≤ T (document if you use receive time for latency studies).
• For labels with horizon H, embargo samples so post-horizon information cannot leak into features.
• When replaying LLM context, freeze the document set available at T—no modern post-mortems in a “2024 agent.”

Bad: train on daily bars adjusted with future revisions, deploy on live unadjusted prints.
Good: same raw event stream for research and live, PIT joins, versioned transforms.

PIT discipline is how data engineering and evaluation stay honest. See leakage traps in the evaluation guide.

Quality monitors and fail-closed behavior

Quality is a control input, not a vanity dashboard. Wire flags to explicit actions:

Always attach the quality snapshot to the decision ledger. When something blows up, the first question is not “what did the model output?”—it is “what did we believe the market was?”

Minimal vs production stack

Minimal (research / single venue)

• One WS trades + BBO, one REST snapshot/reconcile path
• Canonical schema + dual timestamps
• Staleness timer → no new risk
• Local feature compute; append-only log for replay

Production (desk-grade)

• Redundant collectors, durable bus, tested backfill
• Multi-venue normalizer + instrument master
• Quality service with SLOs and paging
• Online store + versioned feature definitions
• Lake/replay with PIT joins for research
• Per-decision input manifests (features, quality, raw refs)

Grow the stack when measurement demands it—not when a slide deck demands “AI platform.”

Signal Desk data stack checklist

Use this as a promotion gate before limited live:

1. Canonical event schema + instrument master documented
2. Exchange vs receive clocks stored and used consistently
3. Quality states wired to risk/policy actions (not only graphs)
4. Online features reproducible from the lake with the same code version
5. Disconnect → backfill path tested
6. Decision ledger joins fills to inputs used
7. PIT rules written down for research and live

Anti-patterns

• One giant CSV as the system of record for live trading
• Training features that cannot be computed online at the same latency
• Ignoring funding/OI while trading perps “on price only”
• Feeding raw social firehose into an LLM with order rights
• No join keys between fills and the signals that caused them
• Assuming historical vendor files match live WS semantics
• Treating “last non-null” as a valid book during outages

Frequently asked questions

What data does an AI trading agent need?

At minimum: reliable market data (trades and/or books), account and order state, and clocks you trust. Many agents also need funding, open interest, on-chain events, and unstructured sources for research agents—always behind quality flags and point-in-time rules.

What is a market data quality flag?

A structured status on a feed or symbol—typically healthy, delayed, or dead—computed from gaps, latency, sequence breaks, and heartbeats so the agent can refuse to trade when data is untrustworthy.

What is the difference between online features and a historical lake?

Online features are low-latency values served to live decisions. A historical lake stores raw and curated events for research, training, and replay. Both must share schemas and point-in-time semantics or evaluation lies.

How does bad data create lookahead bias?

If features use future revisions, full-sample normalization, misaligned timestamps, or post-event text in historical replays, offline metrics inflate. Point-in-time joins and frozen document sets prevent that class of leakage.

What should I log for each trading decision?

At least: decision ID, timestamp, feature vector version, key feature values or references, quality flags, account snapshot IDs, and the resulting intent. Without that bundle you cannot debug or improve the agent.

---

Related: Architecture · How to Build · Evaluation · LLMs · Risk