Methodology: Pipeline Architecture and Signal Lifecycle

Design philosophy

Market Synthesis operates on a single thesis: when market narrative and structural reality diverge, the eventual convergence creates a tradeable repricing event. I detect these gaps using adversarial LLM debate, validate them against market data, and express them as directional trades.

The design principles are:

1. Adversarial over consensus. A single model can sound convincing while missing the best counter-argument. Structured debate forces internal disagreement before any conclusion.
2. Market-data gates over hard-coded rules. Every filter uses observable market signals. No arbitrary thresholds or time limits.
3. Precision over recall. Bad trades cost real money; missed trades only cost opportunity.
4. Debate decides exits. No mechanical P&L gates or time limits. The same adversarial structure that opens a trade also decides when to close it.

Data sources

The pipeline ingests 632,000+ headlines across 6,155 indexed days (2014-2026) from five sources:

Supporting market data:

All data is cached locally and synced to S3 (s3://market-synthesis-cache-ap). The pipeline runs entirely from cache — no live API calls during backtest.

Pipeline architecture

End-to-end pipeline: feed ingestion through adversarial debate to trade execution and exit

Signal lifecycle

Each signal passes through a defined lifecycle:

1. Discovery. The feed processor identifies a cluster of headlines that suggest a narrative-reality divergence. Narrative collapse logic prevents re-emission of ongoing stories.

2. Classification. A Sonnet 4.5 classifier routes the signal to one of seven domain categories. This is a high-stakes decision — miscategorization cascades through the entire pipeline.

3. Entry gate check. Market-data-driven gates verify that the move is not already priced in. Each gate uses observable signals (bond momentum, FX levels, equity drawdowns).

4. Adversarial debate. Three domain-specialized agents (Prosecutor, Defender, Expert) argue the signal in parallel. The Adjudicator synthesizes their arguments into a directional verdict with confidence score.

5. Calibration. A calibration agent adjusts the raw confidence using ECE optimization — correcting for known overconfidence patterns.

6. Instrument selection. An LLM-based selector picks the tradeable expression (futures, ETFs, FX pairs). A coherence gate verifies the instrument matches the signal direction.

7. Position entry. The signal enters the signal store with full provenance (debate transcripts, gate checks, instrument rationale).

8. Periodic exit review. The same adversarial debate structure reviews open positions weekly. The debate determines whether the narrative-reality gap is still open.

9. Exit. Three outcomes: debate decides gap closed (planned exit), volatility stop-loss triggered (mechanical override), or gap widens further (hold continues).

Adversarial debate protocol

The debate is the core intellectual contribution. Each signal receives a full adversarial hearing:

Agent roles

Why mixed models work

Haiku advocates commit firmly to their position. This gives the Sonnet adjudicator clean, differentiated signal to work with. Homogeneous Sonnet debates produce more equivocal arguments that are harder to adjudicate.

Domain specialization

All three debate agents (Prosecutor, Defender, Expert) use category-specific prompt templates. Seven categories times three agents yields 21 specialized prompts, each tuned over 38 iterations to the failure modes of that category.

For example:

• Commodity Prosecutor is prompted to look for physical supply/demand data contradicting market pricing
• CB Defender is prompted to argue that central banks can maintain impossible positions longer than expected
• Sovereign Debt Expert focuses on debt sustainability arithmetic and contagion mechanics

Entry gates

The v38 gates are market-data-driven filters. The design principle: if observable market data already reflects the thesis, the trade is late.

Each gate was added only after identifying a systematic loss pattern and validating that the gate eliminates losses without filtering winners. The thresholds are derived from empirical backtest analysis, not theoretical priors.

Instrument selection

The instrument selector uses a 3-layer approach:

1. Disk cache — previously validated instrument mappings
2. Static registry — known mappings for common signals (e.g., oil supply disruption maps to CL=F)
3. LLM fallback — Sonnet selects the instrument with reasoning, subject to coherence gate

The coherence gate verifies that the selected instrument actually moves in the direction the signal predicts. A short signal paired with a long-only ETF is caught and corrected before entry.

Reproducibility

The entire pipeline is deterministic (temperature 0.0 across all LLM calls). Same input produces same output. All intermediate artifacts are preserved:

• Signal store: development/results/signal_store.jsonl (append-only)
• Debate transcripts: preserved per signal
• Gate decisions: logged with market data at decision time
• Backtest results: development/results/adversarial_exit_partial.jsonl

To reproduce:

cd development
python -u run.py --mode feed --from 2014-01-01 --to 2026-06-01 --workers 10
python -u tests/validate_lifecycle.py
python -u tests/evaluate_store.py
python tests/test_pipeline_invariants.py  # 27/27 tests pass

Cost model

The pipeline costs approximately $0.04 per emitted signal (8 API calls: 5 Haiku + 3 Sonnet via AWS Bedrock). A full 12-year backtest with 140 trades costs under $6 in API fees.

For the exit system in detail, see Exit System. For backtest results, see Findings.