You write the thin venue-specific historical sources; the host composes the platform's caching, gap-filling and re-aggregation chain around them.

Who builds what

Historical loading has two halves, and the seam draws a hard line between them. You write the thin venue-specific parts: "give me time bars for this instrument between these two instants", "give me raw ticks". The host composes the platform's backfill chain around them: on-disk caching, gap filling, bar building from ticks, re-aggregation of non-native periods, order-flow ladder construction.

This is what lets a venue plugin be contracts-only: it hands the host its raw sources and receives a finished, chart-facing IBackfillProvider. The caching policy is platform knowledge and stays in one place, so every venue gets the same gap-filling and re-aggregation behaviour for free.

graph LR
  V["Venue raw sources
time bars · ticks · ladders"] -->|VenueBackfillParts| C["IVenueBackfillComposer
(host)"]
  C -->|caching · gap fill · re-aggregation| B["IBackfillProvider"]
  B --> CH["Chart / backtest / strategy session"]

The raw sources you write

All in namespace SabrTrader.Pipeline.Storage. Implement the ones your venue actually serves. Every request arrives as one HistoryRequest: the instrument, the exchange the user picked (or null for your default routing), the UTC window, and a progress sink (or null).

The raw historical sourcespublic sealed record HistoryRequest(
    string InstrumentId,
    string? Exchange,                          // a picked feed (delayed data), or null
    DateTime FromUtc,
    DateTime ToUtcExclusive,
    IProgress<BackfillProgress>? Progress = null);

// Native time bars for a UTC range. The workhorse: nearly every venue has this.
public interface IHistoricalTimeBarSource
{
    Task<HistoryServed<Bar>> LoadTimeBars(
        HistoryRequest request, TimeSpan period, CancellationToken ct = default);
}

// Raw ticks for a UTC range. Feeds the tick cache, from which Range / Renko /
// volume / Footprint specs are built. Every source states ProvidesHistoricalTicks.
public interface IHistoricalTickSource
{
    Task<HistoryServed<Tick>> LoadTicks(HistoryRequest request, CancellationToken ct = default);

    bool ProvidesHistoricalTicks { get; }
}

// EXACT venue-built footprint ladders, only for venues whose trades carry the
// aggressor side. Null instead means footprints derive from cached ticks.
public interface IHistoricalOrderFlowBarSource
{
    bool ProvidesProfiledMinuteBars { get; }
    Task<HistoryServed<OrderFlowBar>> LoadProfiledMinuteBars(
        HistoryRequest request, CancellationToken ct = default);
}

// Venue-built NON-time bars (tick / volume / range served server-side).
public interface INativeBarSource
{
    bool SupportsNativeBars(BarSpecification spec);
    Task<HistoryServed<Bar>> LoadNativeBars(
        BarSpecification spec, DateTime fromUtc, DateTime toUtcExclusive,
        IProgress<BackfillProgress>? progress, CancellationToken ct = default);
}

A venue with no historical trades API hands the composer NoHistoricalTicks.Instance. It reports ProvidesHistoricalTicks = false, so the chain never asks it for ticks and tick-derived specs build forward from the live stream instead.

Saying what you served

The platform caches history on disk a day at a time, and a day it has cached is never asked for again. So every answer says exactly how much of the window the venue served, and the cache keeps only the whole days inside that span. Anything outside it stays missing and is asked for on the next load.

Items are oldest first. A page that overshoots the window is fine; the chain clips it.

Composing the chain

In ConnectAsync, hand your parts to context.BackfillComposer and publish the result as the data port's Backfill.

IVenueBackfillComposer.cspublic interface IVenueBackfillComposer
{
    ComposedVenueBackfill Compose(VenueBackfillParts parts);
}

public sealed record VenueBackfillParts(
    string ProviderKey,                    // stable key, also the cache namespace
    string VenueDisplayName,               // for backfill-activity labels
    IHistoricalTimeBarSource TimeBars,
    IHistoricalTickSource Ticks,           // empty source when the venue has no tick history
    IInstrumentMetadata Instruments,       // tick size and more for bar building
    IBackfillProvider DirectBackfill,      // the venue's own uncached backfill
    ITimeBarPeriodSupport? PeriodSupport,  // native period truth; null means any period
    TimeSpan DefaultLookback,
    Action<string>? Log = null,
    IHistoricalOrderFlowBarSource? OrderFlowBars = null,
    INativeBarSource? NativeBars = null);

public sealed record ComposedVenueBackfill(
    IBackfillProvider Backfill,
    bool SupportsTickDerived);             // true when the cached tick path is wired

MyDataProvider.csvar composed = (context.BackfillComposer ?? DirectVenueBackfillComposer.Instance)
    .Compose(new VenueBackfillParts(
        ProviderKey:      "myvenue",
        VenueDisplayName: "My Venue",
        TimeBars:         _historicalBars,
        Ticks:            _historicalTicks,
        Instruments:      _instruments,
        DirectBackfill:   _directBackfill,
        PeriodSupport:    MyVenuePeriods.Instance,
        DefaultLookback:  TimeSpan.FromDays(30),
        Log:              context.Notices.Log));

_backfill            = composed.Backfill;          // publish as IDataProvider.Backfill
_supportsTickDerived = composed.SupportsTickDerived;

SupportsTickDerived tells you whether tick-derived specs (Range, Renko, volume, Footprint) can build for this session. Gate SupportsBarSpec on it rather than guessing, so a chart never asks for a spec the chain cannot serve.

The direct composer

context.BackfillComposer is nullable, because a bare tooling host or a test rig can run with no backfill plane. DirectVenueBackfillComposer.Instance is the composer for that case: it returns your uncached direct backfill verbatim, so native time bars serve exactly as they do in the app and SupportsTickDerived reports false. Defaulting to it is what lets the same venue code run unit tests without a host.

Native periods & re-aggregation

ITimeBarPeriodSupport is how the venue states its period truth, so the host can re-aggregate the rest instead of asking for periods the venue would silently approximate.

ITimeBarPeriodSupport.cspublic interface ITimeBarPeriodSupport
{
    bool SupportsNatively(TimeSpan period);

    // The largest natively-served period that divides `period` evenly; the host
    // loads that and re-aggregates up. Null = nothing divides it.
    TimeSpan? LargestNativeDivisor(TimeSpan period);
}

Progress & activity

A long paged load should report progress so the chart's backfill indicator moves. Report each page to request.Progress when it is not null (native bars get the sink as the progress parameter); the host turns those reports into the venue-labelled activity row. A source that cannot report progress simply never calls it.

The offline condition

When a historical request arrives while the venue is not connected, or the venue cannot be reached, throw BackfillSourceUnavailableException from the source. The chain serves what it already has on disk and leaves the rest missing, so the next load after the venue comes back fills it. Any other exception fails the load instead.

MyHistoricalSource.cspublic async Task<HistoryServed<Bar>> LoadTimeBars(
    HistoryRequest request, TimeSpan period, CancellationToken ct = default)
{
    if (!_client.IsConnected)
        throw new BackfillSourceUnavailableException("My Venue is not connected.");

    var bars = await _client.LoadBarsAsync(
        request.InstrumentId, period, request.FromUtc, request.ToUtcExclusive, ct);
    return HistoryServed<Bar>.Complete(bars, request);
}

Exchange-routed history

A venue that can serve the same instrument from more than one exchange or feed reads the exchange from request.Exchange. Null means your default routing, so a venue with a single feed ignores it.

Opt-in source extensions

Four optional interfaces let a venue tell the chain something it cannot otherwise know. Each is detected on the source you already supply: implement it and the chain uses it.

Using the ladder// OANDA-style granularities: everything on the ladder is native, a period an entry
// divides exactly is re-aggregated from the largest such entry, the rest is refused.
private static readonly ITimeBarPeriodSupport Periods = new LadderTimeBarPeriodSupport(new[]
{
    TimeSpan.FromSeconds(5),  TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(30),
    TimeSpan.FromMinutes(1),  TimeSpan.FromMinutes(5),  TimeSpan.FromMinutes(15),
    TimeSpan.FromHours(1),    TimeSpan.FromHours(4),    TimeSpan.FromDays(1),
});

Live history

A chart that goes live after its backfill asks the provider through ILiveHistoryProvider (namespace SabrTrader.Pipeline.Runtime). It passes a LiveHistoryRequest whose AsOfUtc is the one instant it read before the load and hands over to live ticks at. The answer, a LiveHistory, holds the bars and, when the last bar is still forming, its FormingReach: a LiveHandoffPoint saying which live ticks that bar already holds. Because the load and the handoff use the same instant, they always agree about which bar is open.

The chain the composer returns implements this for you, building the forming bar from its caches. You only touch it when you wrap a backfill provider yourself. The rule for a wrapper: it serves live history exactly when the provider it wraps does, and reports that through ProvidesLiveHistory. A wrapper that transforms bars transforms LiveHistory.Bars (and the footprint) the same way its Load does, and keeps FormingReach only while the forming bar is still the last bar.

TimedBackfill.csusing SabrTrader.Pipeline.Bars;
using SabrTrader.Pipeline.Runtime;

// Pass-through wrapper that logs how long each load takes.
public sealed class TimedBackfill : IBackfillProvider, ILiveHistoryProvider
{
    private readonly IBackfillProvider _inner;
    private readonly Action<string> _log;

    public TimedBackfill(IBackfillProvider inner, Action<string> log)
    {
        _inner = inner;
        _log = log;
    }

    public async Task<IReadOnlyList<Bar>> Load(BarSpecification spec, CancellationToken ct = default)
    {
        var started = DateTime.UtcNow;
        var bars = await _inner.Load(spec, ct);
        _log($"{spec.DisplayName}: {bars.Count} bars in {DateTime.UtcNow - started}");
        return bars;
    }

    public bool ProvidesLiveHistory => _inner is ILiveHistoryProvider { ProvidesLiveHistory: true };

    public async Task<LiveHistory> LoadLive(
        BarSpecification spec, LiveHistoryRequest request, CancellationToken ct = default)
    {
        if (_inner is not ILiveHistoryProvider { ProvidesLiveHistory: true } live)
            throw new InvalidOperationException("The wrapped provider serves no live history.");

        var started = DateTime.UtcNow;
        var history = await live.LoadLive(spec, request, ct);
        _log($"{spec.DisplayName} as of {request.AsOfUtc:O}: {history.Bars.Count} bars, " +
             $"forming reach {history.FormingReach?.At:O}, in {DateTime.UtcNow - started}");
        return history;   // bars untouched, so the reach stays valid
    }
}

When you serve the request yourself, let the request do the window arithmetic: request.Validate() rejects a request that cannot describe a window, request.ResolveWindow(defaultLookback) gives the concrete (FromUtc, ToUtcExclusive) with your default depth filled in, TryGetExplicitWindow tells an absolute window from a "last N days up to now" one, and ToOrderFlowRequest() turns a request that asked for order flow into the matching OrderFlowBackfillRequest for an IOrderFlowAwareBackfillProvider. The member details are in the providers reference.

Continuous futures

One optional capability covers everything continuous-contract related, probed on the data port with AsCapability<IContinuousFuturesSource>(). Every member has a default, so a venue implements only what it actually offers.

A product has one roll. A venue whose server-stitched series also says which dated contract it is on decides that roll: the platform reads the series while a chart is live on that contract, and the product's live month follows the venue. A venue that cannot say which contract its series is on should not offer the series here; the platform then stitches the venue's dated contracts on its own roll dates, and live data follows the same dates.

IContinuousFuturesSource.cspublic interface IContinuousFuturesSource
{
    // A server-stitched continuous symbol for a root ("ES" -> "ES.v.0"), if the venue has one.
    // A venue that returns one must answer ContinuousSeriesContractAsync for it.
    string? ContinuousBackfillSymbol(string root) => null;

    // The dated contract that series is on right now, in the venue's own spelling ("ESZ6"), asked of
    // the venue at call time. Null when the venue names none; throw when it cannot be asked. The
    // platform bounds the wait with ct, so stop at its cancellation.
    Task<string?> ContinuousSeriesContractAsync(string root, CancellationToken ct)
        => Task.FromResult<string?>(null);

    // True once proven: the series also serves tick-level history (tick, volume, range and Renko
    // bars, and order flow). Default false; a series that serves time bars only leaves it false.
    bool ContinuousSeriesServesTickData => false;

    // Cached per-contract tick source for continuous tick replay (backtests).
    IRawTickRangeProvider? ContinuousTickContracts => null;
}