Introducing Pulsora: A Rust Time-Series Database for Trading Data

We had a firehose and nowhere good to put it.

The internal problem was specific: store and query a large, continuous stream of market data — ticks, quotes, bars across a lot of symbols — fast enough that ingestion never became the bottleneck and time-range queries came back without us holding our breath. Market data is the cleanest, meanest version of a time-series workload. It arrives in order, it arrives constantly, and it never stops. The volume isn't interesting because it's big once. It's interesting because it's big every single second, forever.

We tried the obvious thing first: reach for an existing time-series database. That's usually the right call, and for a while it was fine. Then the write throughput requirements crept up, the on-disk footprint crept up faster, and the time-range queries we actually ran started to feel the index weight. At some point the storage layer was doing more bookkeeping per row than work on the row.

So we built Pulsora — a time-series database in Rust, optimized for market data and time-ordered datasets. It's open source on GitHub under Apache-2.0. This post is an introduction to what it is, why it exists, and how it actually stores and queries data — grounded in the code, not the brochure.

Why build instead of adopt

Building a database is not a thing you do casually. The bar to clear was: what do we specifically need that we weren't getting cheaply enough?

Three things kept coming up.

Write throughput that scales with blocks, not rows. A tick stream is millions of small, nearly-identical rows. Most general-purpose stores want to index every one of them. That's an index entry per row, and at our row counts the index becomes the dominant cost — in write amplification, in disk, in compaction. We wanted the per-row cost of storage to approach zero and the index to scale with the number of blocks we write, not the number of rows.

Compression that understands the data. Market data compresses beautifully if you treat it as columns of a known type. Timestamps arrive at near-regular intervals. Prices change slowly. Volumes are small integers. Generic row-oriented compression leaves most of that on the table. We wanted type-aware columnar compression — the kind of thing the Facebook Gorilla paper laid out — built in, not bolted on.

A boring, format-flexible interface. We ingest from a few different producers and consume into a few different tools. We didn't want a bespoke wire protocol. We wanted to POST CSV from a script, stream Apache Arrow from a pipeline, and read back JSON, Arrow, or CSV depending on who's asking. HTTP and well-known columnar formats, nothing exotic.

None of these is novel on its own. The combination — for our workload, with our footprint constraints — was enough to justify writing it. And once you're building it anyway, building it in Rust to get compile-time correctness on the hot path is the easy part. We've written before about why we build our own tools; Pulsora is squarely in that lineage.

What Pulsora is today

Let me be honest about scope: Pulsora is 0.1.0. It is an engine that does its job well, not a clustered, replicated, multi-tenant platform with a query planner. It is a single-node, embedded-RocksDB, columnar time-series store with a REST API. That's the whole thing, and the whole thing is the point.

Here's the shape of it:

  • Storage backend: RocksDB, with a custom columnar layer on top.
  • Ingestion formats: CSV, Apache Arrow IPC stream, and Protocol Buffers.
  • Query output formats: JSON (default), Arrow IPC stream, Protobuf, and CSV — negotiated by the Accept header.
  • Schema: inferred automatically on first ingest. You don't declare tables; you POST data and Pulsora figures out the columns and types.
  • Interface: an HTTP/REST API served by Axum on Tokio.
  • Durability: an optional write-ahead log so buffered rows survive a crash.

The whole thing is a single binary named pulsora. There's one CLI flag.

# Run with defaults
cargo run

# Or point it at a config file
cargo run -- -c pulsora.toml

That -c/--config flag is the entire command-line surface. Everything else lives in TOML. This is deliberate — the configuration is the API for operating it, and the runtime API is HTTP.

How it stores data on disk

This is the part we cared about most, so it's the part worth explaining.

Columnar blocks, not rows

Rows come in, but they don't get stored as rows. Pulsora accumulates incoming rows in an in-memory buffer, and when the buffer hits its size threshold (buffer_size, default 1000) or its time threshold (flush_interval_ms), it groups them into a ColumnBlock — a chunk of rows stored column-by-column, each column compressed independently with an algorithm chosen for its type.

The block is the unit of storage. A ColumnBlock is roughly:

pub struct ColumnBlock {
    pub row_count: usize,                        // rows in this block
    pub columns: HashMap<String, Vec<u8>>,       // compressed column data
    pub null_bitmaps: HashMap<String, Vec<u8>>,  // null tracking per column
}

Storing by column instead of by row is what makes the compression work, because every value in a column has the same type and similar magnitude. That's the whole reason columnar stores exist, and time-series data is the workload it was invented for.

Type-specific compression

Each column type gets a compression strategy built for its shape. Per the repo's documentation:

Data type Compression Typical ratio (repo's figures) Why it works
Timestamp Delta-of-delta + varint 5–10x Near-regular intervals
Float XOR (Gorilla) + varfloat 2–5x Slowly changing values
Integer Delta + varint 3–8x Sequential / counter data
String Dictionary encoding 2–4x Repeated values (symbols)
Boolean Run-length encoding 10–50x Sparse data

The timestamp path is the textbook case. Ticks arrive at intervals that are almost constant, so the second-order difference (delta-of-delta) is usually zero or tiny, and a varint encodes a tiny number in one byte. Floats use XOR-against-previous-value bit packing from the Gorilla algorithm — when a price barely moves, most of the XOR'd bits are zero and pack away to nothing. Strings like ticker symbols repeat endlessly, so a dictionary maps each distinct symbol to a small integer once.

Underneath all of that, RocksDB applies a second layer of generic block compression — lz4 by default, with none, snappy, and zstd available. So you get type-aware encoding first, then a general-purpose pass on top.

These ratios are the repo's own benchmark claims, measured with the Criterion suites under benches/. I'm reporting them as the project's numbers, not as an independent verdict — run cargo bench against your own data and trust that instead.

The index that doesn't grow per row

This is the design decision the whole thing turns on.

Most time-series stores keep a per-row index so they can find any single row by time. Pulsora doesn't. It keeps a block-level index only — metadata about each block, never about individual rows. The on-disk key layout, per the architecture docs, looks like this:

Block index: [table_hash:u32]['B'][min_ts:i64][block_id:u64]
             value: [block_id][min_ts][max_ts][rows][min_id][max_id]   — one entry per block
Block data:  [table_hash:u32]['D'][block_id:u64]                       — the compressed columnar block
Overrides:   [table_hash:u32]['O'][block_id:u64]                       — dead row positions

The table_hash is a 4-byte FNV-1a hash of the table name, used as a key prefix so tables are isolated without dragging string names through every key. After the prefix, a single byte ('B', 'D', 'O') separates the index, the data, and the override set.

The consequence: storage cost scales with the number of blocks, not the number of rows. Write a billion rows and you add no index entries beyond the blocks that hold them. That was the requirement we couldn't buy cheaply, and it's the core of why Pulsora exists.

Row updates are handled without rewriting blocks. A REPLACE marks the superseded copy's position in the old block's override set — a per-block set of dead row positions, merged via a RocksDB merge operator (set union) in the same write batch as the new data. Liveness is one point read per block, not a lookup per row.

The write path

Ingestion is a POST. Schema is inferred on the first one.

curl -X POST http://localhost:8080/tables/stocks/ingest \
  -H "Content-Type: text/csv" \
  --data-binary @ticks.csv

That CSV's header row becomes the schema. Pulsora samples the values, detects each column's type (Integer, Float, String, Boolean, Timestamp), and identifies the timestamp column automatically — it understands RFC 3339, YYYY-MM-DD HH:MM:SS, date-only, and Unix timestamps in both seconds and milliseconds. From then on, that table's schema is fixed and incoming data is validated against it.

For higher-throughput pipelines, skip CSV parsing entirely and stream Apache Arrow:

curl -X POST http://localhost:8080/tables/stocks/ingest \
  -H "Content-Type: application/vnd.apache.arrow.stream" \
  --data-binary @ticks.arrow

Arrow comes in with its schema already attached, so there's no inference step and no text parsing — the columns are already columns. Protobuf ingestion works the same way for producers that already speak it.

Internally, the flow is: parse the input, validate or infer the schema, buffer rows in memory (deduplicating on the way in, latest-write-wins), and — if the WAL is enabled — append each row to a .wal file before it touches memory so a crash doesn't lose buffered data. When the buffer flushes, the rows become a compressed ColumnBlock, the block and its index entry are written to RocksDB in a single batch, and the WAL is truncated.

The read path

Queries are GETs with a time range and pagination.

# Everything (capped by the default limit)
curl "http://localhost:8080/tables/stocks/query"

# A time window
curl "http://localhost:8080/tables/stocks/query?start=2024-01-01T09:30:00&end=2024-01-01T16:00:00"

# Paginated
curl "http://localhost:8080/tables/stocks/query?limit=1000&offset=0"

start and end accept the same flexible set of timestamp formats as ingestion. limit defaults to 1000, with no enforced maximum; offset skips rows for pagination.

The query engine resolves the request against the block index first: it builds a binary key range from the time bounds and iterates only the blocks whose [min_ts, max_ts] overlaps the window. It collects the candidate blocks, fetches each unique block once, decompresses it, caches the decompressed result so repeated access within a query doesn't re-decompress, applies the override set to skip dead rows, extracts the rows it needs, and serializes them in whatever format the Accept header asked for.

Want the result back as Arrow for a downstream pipeline instead of JSON? Change one header:

curl "http://localhost:8080/tables/stocks/query?limit=10000" \
  -H "Accept: application/vnd.apache.arrow.stream" > out.arrow

curl "http://localhost:8080/tables/stocks/query?limit=10000" \
  -H "Accept: text/csv" > out.csv

Same query, four possible output formats, no conversion step on your side. There are a handful of other read endpoints — GET /tables to list tables, GET /tables/{name}/schema for the inferred schema, GET /tables/{name}/count for a row count, GET /tables/{name}/row/{id} to fetch a single row by id, and GET /health.

Try it

Pulsora builds with a current Rust toolchain and runs as a single binary.

# Clone and run from source
git clone https://github.com/muvon/pulsora.git
cd pulsora
cargo run

# Or install the binary directly
cargo install --git https://github.com/muvon/pulsora.git

# Ingest a CSV
curl -X POST http://localhost:8080/tables/stocks/ingest \
  -H "Content-Type: text/csv" \
  -d "timestamp,symbol,price,volume
2024-01-01 09:30:00,AAPL,150.00,1000
2024-01-01 09:31:00,AAPL,150.25,1500
2024-01-01 09:32:00,AAPL,149.75,2000"

# Read it back
curl "http://localhost:8080/tables/stocks/query?limit=10"

# Look at the inferred schema
curl "http://localhost:8080/tables/stocks/schema"

Tuning lives in pulsora.toml. The settings worth knowing first:

[storage]
data_dir = "./data"
buffer_size = 1000          # rows buffered before a block is written
flush_interval_ms = 1000    # max time a row waits in the buffer
wal_enabled = true          # buffered rows survive a crash

[ingestion]
batch_size = 10000          # rows per database write batch

[performance]
compression = "lz4"         # none | snappy | lz4 | zstd
cache_size_mb = 256         # block cache for reads

If you care about maximum compression over latency, set flush_interval_ms = 0 for a batch-only mode that holds rows until the buffer fills, producing larger, better-compressed blocks. If you care about read latency on hot data, raise cache_size_mb. The full set of knobs is documented in doc/CONFIGURATION.md.

What it isn't, and what's next

I'll save you the disappointment of discovering these the hard way. Pulsora today is single-node. There's no clustering, no replication, no SQL, no general query planner, no predicate pushdown beyond the time range, no rate limiting. Queries filter by time and paginate; they don't GROUP BY. The doc/ARCHITECTURE.md file lists the directions we're looking at — column pruning and predicate pushdown, a disk-backed block cache, tiered hot/warm/cold storage, and clustering — but those are roadmap, not reality. I'd rather you knew the edges than hit them.

What it is, today, is a focused engine that ingests a firehose of time-ordered data, compresses it hard with algorithms that understand it, and serves time-range queries without an index that bloats per row. That was the problem we had. It's the problem Pulsora solves.

Open source

Pulsora is on GitHub under Apache-2.0. It's Rust — RocksDB for storage, Axum and Tokio for the API, Arrow and Prost for the columnar and protobuf formats — and it builds with a plain cargo build. The benchmark suites are in benches/ if you want to measure ingestion and query performance against your own data rather than trust ours.

If you're drowning in time-ordered rows — market ticks, sensor readings, anything that arrives in order and never stops — clone it, point a stream at it, and tell us where it breaks. That's the fastest way it gets better.

— Don

Pulsora is open source on GitHub at github.com/muvon/pulsora under the Apache-2.0 license. Found a bug or want a feature? Open an issue.