Data Preparation

hftbacktest · Data Preparation · Working lab

The hftbacktest docs open their tutorials with the least glamorous page in the set, and they are right to. Every result a backtester produces is a claim about what you could have known and when, and that claim is made entirely by the data preparation step. Get it wrong and nothing breaks: you get a run, a P&L curve and a Sharpe ratio, and they are all fiction.

The tutorial page itself is short: two calls, a snapshot, and a second data source. What it does not explain is the interesting part. Why one message becomes eighty-two rows. What 3758096385 is. Why convert prints Correcting the event order and then hands you back more rows than you gave it. Why a venue that never closes needs a snapshot at all, and why the trades file goes in before the depth file.

This lab is that page with the machinery exposed and running. Everything below is computed from the five feed lines the docs print and from a port of the library’s own conversion functions, so the ev values are assembled from the constants, the fan-out is counted off the real message, and the latency correction lands on the tutorial’s printed timestamp to the nanosecond.

Tick-by-tick full order book data is not something you download. There is no Yahoo Finance for the book, and the docs are blunt about it. For crypto your realistic options are to collect the raw feed yourself with the project’s Data Collector, or to buy it. Either way what lands on disk is a gzipped log of messages, and the first job is to turn it into the eight-column array the backtester eats.

# the whole Binance Futures conversion
from hftbacktest.data.utils import binancefutures

data = binancefutures.convert(
    'usdm/btcusdt_20240808.gz',
    combined_stream=True
)
# Correcting the latency
# local_timestamp is ahead of exch_timestamp by 1272156851
# Correcting the event order

Three things in that strip are worth stopping on, and all three are things the tutorial’s output does not say out loud.

The fan-out is the shape of the whole file. 5 lines and 2697 bytes go in; 83 rows come out, and 3 of the five lines produce nothing at all. One depthUpdate carrying 53 bid levels and 29 ask levels becomes 82 rows, every one of them stamped with the same exch_ts and the same local_ts, because they arrived in one message about one moment.

And the three lines that vanish are worth more attention than the two that survive. They are bookTicker messages, and convert’s default opt='' skips them without printing a word. Pass opt='t' and each becomes two rows with the custom ids 103 and 104. Those rows carry no DEPTH_EVENT and no side flag, because best-bid-and-offer is a separate channel and not depth. The sample output of a working conversion is mostly the stream it is throwing away, and nothing tells you so.

Eight columns, in this order: ev, exch_ts, local_ts, px, qty, order_id, ival, fval. Six of them are obvious. order_id is only ever set by a level-3 market-by-order feed, and ival and fval are reserved: an integer and a float you can put anything in, which is how the custom data tutorial later attaches your own signals to the same array.

That leaves ev, which looks like one bitfield and is really two things sharing a word: a small event type in the low byte, and four independent flags in the top nibble. Twenty bits in the middle are unused. That is why a depth update on the bid side is 3758096385 and not 5.

The nine types, and what each one does to the book:

It is not, of course. It is a clock offset: the machine doing the recording and the exchange’s matching engine disagree about what time it is, and no amount of care with the recording will fix that after the fact. So the conversion does the only honest thing available. It finds the most negative latency in the whole file and shifts every local_ts forward by that much, which preserves every gap between rows and moves the file just far enough that the worst row stops being impossible.

# validation.py, the whole correction
latency = sys.maxsize
for row_num in range(len(data)):
    latency = min(latency, data[row_num].local_ts - data[row_num].exch_ts)

if latency < 0:
    offset = -latency + base_latency
    print('local_timestamp is ahead of exch_timestamp by', -latency)
    for row_num in range(len(data)):
        data[row_num].local_ts += offset

The default base_latency=0 is the setting to be suspicious of, and the last column is why. After the shift, the worst row in the file has a latency of exactly zero. It now claims you learned about an event at the instant it happened at the matching engine, and a backtest will cheerfully fill you on that information. The library’s own docstring says as much: the conversion “may still produce zero latency cases,” and base_latency exists so you can put a floor under them. The floor is a claim about your own infrastructure, and zero claims nothing.

What survives the correction is the shape of the distribution, and that is what you actually came for:

Four milliseconds to fifty-two, in four minutes, on one symbol. Not an average anybody could have told you in advance, and not a constant. This is the number a backtest is either respecting or quietly ignoring, and it is the reason the whole two-timeline apparatus in the next section exists.

Here is the problem, stated plainly. Two feeds, a throttled depth stream and a trade stream, are two queues, and they do not preserve each other’s order. So it happens constantly that event A happened before event B at the matching engine, and yet B reached your socket first. Both facts are true. Neither is a bug.

Now try to put those two rows in one array. Sorted by exch_ts the array is a lie about what you knew. Sorted by local_ts it is a lie about what happened. There is no third option, because no single ordering of two rows can ascend in two clocks that disagree.

# the call that does the splitting
data = correct_event_order(
    tmp,
    np.argsort(tmp['exch_ts'], kind='mergesort'),  # walk 1
    np.argsort(tmp['local_ts'], kind='mergesort')  # walk 2
)
validate_event_order(data)

So hftbacktest stops trying to find one. correct_event_order walks the rows twice: once in exchange order, once in reception order. Then it merges the two walks. Where both walks agree on a row it emits that row once, carrying both bits. Where they disagree it emits the row twice: once with EXCH_EVENT at the moment the exchange had it, and once with LOCAL_EVENT at the moment you did. Two interleaved timelines in one array.

The two verdicts under it are validate_event_order, the check the library runs last, applied twice: once to the array as convert emitted it and once to the corrected one. Push the slider negative and the first check fails while the second passes. That failure is the entire justification for the function: without it the array is not ascending in exchange time, and an exchange simulator stepping through it would process a trade it had already processed the consequences of.

The collector fetches a REST snapshot when it opens its socket and never again. So the first day of a recording has a real starting book and every day after it has nothing, and a depth update only tells you about the levels that changed. Levels nobody touches stay invisible. Deletions arrive for prices your book has never heard of and do nothing at all.

# build 20240808's end-of-day book, for 20240809 to start from
from hftbacktest.data.utils.snapshot import create_last_snapshot

_ = create_last_snapshot(
    ['usdm/btcusdt_20240808.npz'],
    tick_size=0.1,
    lot_size=0.001,
    output_snapshot_filename='usdm/btcusdt_20240808_eod.npz'
)

Two details in that call are easy to skim past. The first is that create_last_snapshot is not a parser. It builds a BacktestAsset, calls _goto_end() to run the entire day through the real replay engine, and dumps depth.snapshot(). The end-of-day book is produced by the same machinery your strategy will run against, which is the only way it is guaranteed to agree with it.

The second is that it will not run without a tick size and a lot size, because prices in hftbacktest are integer numbers of ticks rather than floats.

Choose a tick coarser than the venue’s and the best bid and the best ask land on the same integer. The book loses its spread, nothing raises an error, and the backtest runs to completion. tick_size is not a display setting; it is the resolution at which prices exist, and it is a property of the venue that you have to look up, because nothing in the feed states it.

# 20240809, starting from where 20240808 left off
_ = create_last_snapshot(
    ['usdm/btcusdt_20240809.npz'],
    tick_size=0.1,
    lot_size=0.001,
    output_snapshot_filename='usdm/btcusdt_20240809_last.npz',
    initial_snapshot='usdm/btcusdt_20240808_eod.npz',
)

# or replay both days instead of chaining. same answer
_ = create_last_snapshot(
    ['usdm/btcusdt_20240808.npz', 'usdm/btcusdt_20240809.npz'],
    tick_size=0.1,
    lot_size=0.001,
    output_snapshot_filename='usdm/btcusdt_20240809_last.npz'
)

The trade-off is in the timestamps, and the library’s own docstring is refreshingly direct about it: for Binance futures, Tardis uses E, the sending time, rather than T, “so the latency is slightly less than it actually is.” Slightly, and always in the flattering direction. You are also inheriting somebody else’s network path and somebody else’s clock for local_ts, which makes it a usable relative ordering and not a measurement of your own latency at all.

# trades first. the order is the whole point. see below
from hftbacktest.data.utils import tardis

data = tardis.convert(
    ['BTCUSDT_trades.csv.gz', 'BTCUSDT_book.csv.gz']
)
# Reading BTCUSDT_trades.csv.gz
# Reading BTCUSDT_book.csv.gz
# Correcting the latency
# Correcting the event order

This is the sharpest lesson on the tutorial page and it is stated in half a sentence. convert reads its input files in order into one buffer and sorts that buffer with kind='mergesort', which is stable, so two rows sharing an exchange timestamp come out in whichever order their files were read. A trade and the depth decrease it caused share that timestamp exactly, because they are one matching-engine event described twice.

Get the order backwards and nothing looks wrong. The rows are identical, the count is identical, the timestamps are identical. But the depth decrease is applied before the trade that explains it, both reduce your queue position, and 1.197 lots of volume moves you 2.394 lots forward. Every simulated fill arrives a little early and every simulated edge comes out a little better than it was. A data preparation bug does not crash a backtest. It flatters it.

# tardis incremental_book_L2: one row per level already
exchange, symbol, timestamp, local_timestamp, is_snapshot, side, price, amount

# tardis trades
exchange, symbol, timestamp, local_timestamp, id, side, price, amount

# 27.5M rows for one symbol-day. the default buffer is 100M,
# and the snapshot buffer is a separate 1M
_ = tardis.convert(
    ['BTCUSDT_trades.csv.gz', 'BTCUSDT_book.csv.gz'],
    output_filename='btcusdt_20200201.npz',
    buffer_size=200_000_000
)

Every step is a place where a plausible shortcut produces a backtest that runs. Read E instead of T and you get results, flattering ones. Skip the latency correction and rows with negative latency sort into the past. Skip the event-order correction and one of the two timelines runs backwards. Skip the snapshot and your book has holes you will never be told about. Pass the depth file first and your queue moves twice as fast as the tape.

None of those raise an exception. That is the actual subject of this tutorial, and it is why it comes first.

Data Preparation in one breath. hftbacktest eats a numpy structured array of eight columns, and the only interesting one is ev, an event type in its low byte and four flags in its top nibble, two of which decide whether the exchange or your strategy is allowed to see the row. Getting there from a raw feed means four things. Parse each line, taking the 19-digit nanosecond prefix as local_ts and the matching-engine timestamp T as exch_ts, and fan a depth message out into one row per level. Correct the latency by shifting every local_ts until the most negative feed latency in the file reaches zero. Then think hard about base_latency, because zero latency on any row is a licence to trade on information nobody had. Correct the event order by merging an exchange-time walk with a reception-time walk, splitting any row the two disagree about into an EXCH_EVENT copy and a LOCAL_EVENT copy, which is the only way one array can ascend in two clocks at once. And because a crypto venue has no start of day, build the book by replaying the previous day to its end and hand that snapshot to the next one.

Buying the data instead moves the problem rather than removing it: a vendor’s local_ts is a stranger’s machine, its exch_ts may be a sending time, and its files must go into the converter trades before depth or the queue position you simulate moves twice for every trade. Every one of these failures produces a complete, plausible, profitable-looking backtest. The point of this page is that the errors are silent, and the only defence is knowing what each column means.