Processing Financial Market Data in Real Time with Apache Flink

Financial markets generate an enormous volume of data every second. A single liquid equity can produce thousands of ticks per second during peak trading. Processing that data in real time, rather than in hourly or end-of-day batches, changes what you can build: live risk dashboards, automated trading signals, real-time compliance checks, and portfolio analytics that actually reflect current market conditions.

Apache Flink is built for exactly this kind of workload. Its event-time processing model, stateful computation, and low-latency execution make it well suited for financial data pipelines where ordering, accuracy, and speed all matter at once.

This article walks through the practical side of building these pipelines: how to ingest tick data, compute common financial metrics like VWAP and moving averages, aggregate order book state, and meet the latency requirements that financial applications demand.

Why Financial Data Is Different

Most stream processing tutorials use examples like clickstreams or IoT sensor readings. Financial market data shares some characteristics with those domains, but it introduces several additional challenges that shape how you design your pipeline.

Volume and velocity. During active trading hours, a feed covering US equities alone can produce millions of events per second. Your pipeline has to keep up without falling behind, because even a small backlog means your calculations are stale.

Ordering matters. A trade that arrived at 10:00:00.001 and one at 10:00:00.002 are distinct events with potentially different implications. Unlike a web analytics pipeline where a few seconds of reordering is fine, financial pipelines need event-time semantics to maintain correctness.

Multiple event types. A market data feed includes trades, quotes, order book updates, and sometimes auction signals. These are structurally different messages that often need to be joined or correlated within the same pipeline.

Latency expectations vary by use case. High-frequency trading operates in microseconds and typically uses specialized hardware, not Flink. But for risk monitoring, compliance, portfolio analytics, and mid-frequency signal generation, sub-second to low-second latency is both sufficient and achievable with Flink.

Ingesting Tick Data into Flink

Before Flink can compute anything, you need to get market data into a format and transport layer it can consume. The standard pattern is to land tick data in Kafka topics, then have Flink read from those topics.

There are several ways market data reaches Kafka:

• Direct feed handlers that connect to exchange feeds (SIP, direct, or consolidated) and produce to Kafka.
• CDC from a trading database. If your order management system or trade capture system writes to PostgreSQL or another database, Streamkap can capture every insert and update via CDC and deliver it to Kafka in real time. This is especially useful for internal trade data, position updates, and reference data changes.
• Third-party market data APIs wrapped in a Kafka producer that normalizes the data before producing.

Regardless of the source, the Kafka topic becomes the boundary between ingestion and processing. Flink reads from it using the Kafka connector.

Here is a basic Flink source configuration for consuming tick data:

KafkaSource<String> tickSource = KafkaSource.<String>builder()
    .setBootstrapServers("kafka-broker:9092")
    .setTopics("market-ticks")
    .setGroupId("flink-market-processor")
    .setStartingOffsets(OffsetsInitializer.latest())
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

DataStream<String> rawTicks = env.fromSource(
    tickSource,
    WatermarkStrategy
        .<String>forBoundedOutOfOrderness(Duration.ofMillis(200))
        .withTimestampAssigner((event, timestamp) -> extractEventTime(event)),
    "Market Tick Source"
);

A few things worth noting here. We use OffsetsInitializer.latest() because for a live market data pipeline, you typically want to start from the current position rather than replaying history. The watermark strategy allows 200 milliseconds of out-of-orderness, which accounts for minor delays in the feed without holding windows open too long.

The extractEventTime function parses the exchange timestamp from the tick message. This is the timestamp the exchange assigned when the trade or quote occurred, not the time Kafka received the message. Using exchange timestamps as event time is what keeps your calculations correct even when network jitter reorders events.

Computing VWAP in Real Time

Volume-Weighted Average Price (VWAP) is one of the most commonly used metrics in trading. It represents the average price of a security weighted by volume over a given period, typically the trading day. Traders use it as a benchmark: executing a large order at or below VWAP is generally considered good execution.

The formula is straightforward:

VWAP = sum(price * volume) / sum(volume)

In a batch world, you would compute this at the end of the day. With Flink, you can maintain a running VWAP that updates with every tick.

Here is how to implement it using a keyed process function:

public class VwapCalculator extends KeyedProcessFunction<String, Trade, VwapResult> {

    private ValueState<Double> cumulativePriceVolume;
    private ValueState<Long> cumulativeVolume;

    @Override
    public void open(Configuration parameters) {
        cumulativePriceVolume = getRuntimeContext().getState(
            new ValueStateDescriptor<>("priceVolume", Double.class, 0.0));
        cumulativeVolume = getRuntimeContext().getState(
            new ValueStateDescriptor<>("volume", Long.class, 0L));
    }

    @Override
    public void processElement(Trade trade, Context ctx, Collector<VwapResult> out) throws Exception {
        double pv = cumulativePriceVolume.value() + (trade.price * trade.volume);
        long vol = cumulativeVolume.value() + trade.volume;

        cumulativePriceVolume.update(pv);
        cumulativeVolume.update(vol);

        double vwap = pv / vol;
        out.collect(new VwapResult(trade.symbol, vwap, vol, ctx.timestamp()));
    }
}

The stream is keyed by symbol, so each ticker maintains its own independent VWAP state. Every incoming trade updates the running totals and emits the current VWAP value. Downstream consumers, whether a dashboard, an alerting system, or a trading algorithm, receive an updated VWAP with every single trade.

To reset the VWAP at the start of each trading day, you can register a timer that fires at the market open time and clears the state:

@Override
public void processElement(Trade trade, Context ctx, Collector<VwapResult> out) throws Exception {
    // Register a timer for the next market open if not already set
    long nextOpen = calculateNextMarketOpen(ctx.timestamp());
    ctx.timerService().registerEventTimeTimer(nextOpen);

    // ... VWAP calculation as above
}

@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<VwapResult> out) {
    cumulativePriceVolume.clear();
    cumulativeVolume.clear();
}

This pattern gives you a continuously updating intraday VWAP that resets cleanly at the session boundary.

Moving Averages with Sliding Windows

Moving averages smooth out price noise and are used for trend detection, crossover signals, and volatility estimation. A simple moving average (SMA) over the last N minutes computes the mean of all trade prices in that window.

Flink’s sliding windows map directly to this concept:

DataStream<MovingAverage> sma = trades
    .keyBy(trade -> trade.symbol)
    .window(SlidingEventTimeWindows.of(
        Time.minutes(5),   // window size
        Time.seconds(10))) // slide interval
    .aggregate(new AverageAggregator());

This creates a 5-minute window that slides every 10 seconds. Every 10 seconds, Flink emits the average price for the preceding 5 minutes of trades for each symbol.

The AverageAggregator is a standard Flink AggregateFunction:

public class AverageAggregator implements AggregateFunction<Trade, Tuple2<Double, Long>, Double> {

    @Override
    public Tuple2<Double, Long> createAccumulator() {
        return Tuple2.of(0.0, 0L);
    }

    @Override
    public Tuple2<Double, Long> add(Trade trade, Tuple2<Double, Long> acc) {
        return Tuple2.of(acc.f0 + trade.price, acc.f1 + 1);
    }

    @Override
    public Double getResult(Tuple2<Double, Long> acc) {
        return acc.f0 / acc.f1;
    }

    @Override
    public Tuple2<Double, Long> merge(Tuple2<Double, Long> a, Tuple2<Double, Long> b) {
        return Tuple2.of(a.f0 + b.f0, a.f1 + b.f1);
    }
}

For exponential moving averages (EMA), which give more weight to recent prices, a KeyedProcessFunction works better because EMA is defined recursively rather than over a fixed window:

EMA_today = price * k + EMA_yesterday * (1 - k)
// where k = 2 / (N + 1)

You store the previous EMA value in Flink’s keyed state and update it with each incoming trade. This is another case where Flink’s stateful processing model fits the domain naturally.

Order Book Aggregation

An order book represents the current set of outstanding buy and sell orders for a security at various price levels. Maintaining a real-time aggregated order book view requires handling a high-frequency stream of incremental updates: new orders, modifications, cancellations, and fills.

Each update affects a specific price level on one side of the book. The aggregated view at any point in time shows the total quantity available at each price level.

In Flink, you can model this using MapState keyed by symbol:

public class OrderBookAggregator extends KeyedProcessFunction<String, OrderBookUpdate, AggregatedBook> {

    private MapState<Double, Long> bidLevels;
    private MapState<Double, Long> askLevels;

    @Override
    public void open(Configuration parameters) {
        bidLevels = getRuntimeContext().getMapState(
            new MapStateDescriptor<>("bids", Double.class, Long.class));
        askLevels = getRuntimeContext().getMapState(
            new MapStateDescriptor<>("asks", Double.class, Long.class));
    }

    @Override
    public void processElement(OrderBookUpdate update, Context ctx,
                               Collector<AggregatedBook> out) throws Exception {
        MapState<Double, Long> levels = update.side.equals("BID") ? bidLevels : askLevels;

        if (update.quantity == 0) {
            levels.remove(update.priceLevel);
        } else {
            levels.put(update.priceLevel, update.quantity);
        }

        out.collect(buildSnapshot(ctx.getCurrentKey(), bidLevels, askLevels));
    }
}

A quantity of zero signals that a price level has been fully consumed and should be removed. Otherwise, the quantity at that level is replaced with the new value.

The buildSnapshot method iterates over the current state of both sides and produces a sorted snapshot. In practice, you might not want to emit a full snapshot on every single update. Instead, you could use a processing-time timer to emit snapshots at a fixed interval (say every 100 milliseconds), which reduces downstream load while keeping the view fresh.

One thing to watch out for: order book state can grow large for instruments with many active price levels. Setting a TTL on levels that have not been updated recently, or applying a depth limit (top 10 levels only), keeps memory usage predictable.

Meeting Latency Requirements

For financial data pipelines, latency is not just a performance metric. It is a functional requirement. A risk calculation that arrives 30 seconds late might violate regulatory thresholds. A trading signal that arrives after the opportunity window has closed is worthless.

Here are the levers you have in Flink to control latency:

Buffer timeout. Flink batches records internally before sending them between operators. The default buffer timeout is 100 milliseconds. For latency-sensitive pipelines, you can reduce it:

env.setBufferTimeout(10); // flush every 10 milliseconds

Setting this lower means Flink flushes partial buffers sooner, reducing latency at the cost of slightly higher network overhead.

Checkpoint interval and duration. Checkpoints cause brief pauses in processing. For a financial pipeline where you need consistent sub-second latency, configure checkpoints at longer intervals (say 60 seconds) and enable unaligned checkpoints to reduce the impact:

env.enableCheckpointing(60000);
env.getCheckpointConfig().enableUnalignedCheckpoints();

Parallelism tuning. Each Kafka partition maps to a Flink parallel instance. If your tick data topic has 16 partitions, setting parallelism to 16 ensures each partition is consumed by a dedicated thread with no contention.

Async I/O. If your pipeline needs to enrich ticks with reference data from an external service (security master, risk parameters), use Flink’s async I/O operator instead of a synchronous call that blocks the processing thread:

DataStream<EnrichedTick> enriched = AsyncDataStream.unorderedWait(
    tickStream,
    new ReferenceDataLookup(),
    500, TimeUnit.MILLISECONDS,
    100  // max concurrent requests
);

This lets Flink fire off lookup requests without stalling the main processing pipeline.

Putting the Pipeline Together

A complete financial data pipeline typically has multiple computation branches reading from the same ingested stream. Here is what the overall topology looks like:

1. Ingestion. Streamkap captures trade and position data from your databases via CDC and delivers it to Kafka. Market data from exchange feeds lands in separate Kafka topics.
2. Parsing and validation. A Flink map operator deserializes raw messages, validates required fields, and routes malformed events to a dead-letter topic.
3. Branching. The validated stream is split. One branch computes VWAP. Another computes moving averages. A third maintains order book state. Each branch is keyed by symbol and operates independently.
4. Output. Results are written to downstream systems. A VWAP stream might feed a real-time dashboard (via Kafka to a WebSocket server). Moving average crossover alerts might go to a notification service. Aggregated order book snapshots might land in a low-latency data store like Redis.

Streamkap fits into this architecture at both ends. On the ingestion side, it handles CDC from your databases and delivers clean, schema-managed data to Kafka without you having to run Debezium or manage Kafka Connect clusters. On the processing side, Streamkap’s managed Flink service handles cluster provisioning, checkpoint management, and scaling so you can focus on the computation logic rather than infrastructure.

Monitoring Financial Flink Jobs

Financial pipelines need tighter monitoring than most workloads. Here are the metrics to watch:

• Event-time lag. The difference between the current watermark and wall-clock time. If this grows, your pipeline is falling behind real-time. For a financial pipeline, even a few seconds of lag warrants investigation.
• Checkpoint duration. If checkpoints start taking longer, your state may be growing unbounded. This is common with order book aggregation if stale levels are not cleaned up.
• Backpressure. If any operator shows backpressure, it means a downstream stage cannot keep up. This is your first indicator that you need to increase parallelism or optimize an expensive operation.
• Records per second per operator. This helps you identify which branch of your pipeline is the bottleneck. Compare against the expected tick rate for the instruments you are processing.

Set alerts on these metrics with thresholds appropriate for your latency requirements. A risk monitoring pipeline might alert at 5 seconds of event-time lag. A compliance pipeline might have a stricter threshold.

From Batch to Streaming

Many financial firms still compute VWAP, risk metrics, and order book analytics in batch jobs that run at end-of-day or on hourly schedules. The shift to streaming does not have to happen all at once. A practical migration path is to run the Flink pipeline alongside the existing batch job, compare results, and gradually shift consumers to the streaming output once you have confidence in its accuracy.

Flink’s exactly-once processing guarantees mean you can trust the streaming output to match what the batch job would have produced, with the added benefit of continuous updates throughout the day. With Streamkap managing the ingestion and Flink infrastructure, the operational cost of running the streaming pipeline is significantly lower than maintaining your own Kafka Connect and Flink clusters.

Financial data processing is one of the domains where stream processing delivers the most immediate value. The data is already generated in real time. The consumers already want real-time results. The missing piece has traditionally been a processing layer that could handle the volume, maintain correctness, and operate reliably without a large dedicated infrastructure team. Flink fills that gap, and a managed platform like Streamkap removes the operational overhead that has historically kept teams from adopting it.