Open two psql sessions.

Session A:

BEGIN;
DELETE FROM users;  -- deletes all 1 million rows
-- don't commit yet

Session B:

SELECT count(*) FROM users;
-- count: 1000000

Session A deleted every row. Session B sees every row. Neither session is waiting on the other. No locks, no blocking, no errors.

So... where are the rows? Deleted, or not?

The answer is both, and by the end of this post you'll have built the machinery that makes that answer make sense — in about 100 lines of TypeScript. This machinery is called MVCC (Multi-Version Concurrency Control), and it's how Postgres, MySQL/InnoDB, Oracle, and SQLite (in WAL mode) all let readers and writers ignore each other.

As always in this series: no hand-waving. We'll build it one problem at a time, and at each step we'll only add the minimum code needed to fix the problem in front of us. Then we'll point our toy at real Postgres and show that the columns match, one to one.

The wrong mental model

Most of us carry around a mental model of a database table as a big mutable array. UPDATE overwrites a slot. DELETE removes one.

If that were true, the demo above would be impossible. The moment Session A deletes a row in place, that row is gone — Session B has nothing left to read. The only way to make in-place mutation safe is locks: readers wait for writers, writers wait for readers. Databases worked this way for years, and it was miserable — one long-running report could freeze every write to the table.

So here is the founding decision of MVCC, the thesis of this whole post:

Nothing is ever updated. Nothing is ever deleted. The database only ever appends.

Every "problem" we hit from here on is a consequence of this decision, and every piece of MVCC is the minimal fix for one of those problems. Let's start hitting them.

Problem 1: If we never delete, what does DELETE even mean?

If data is append-only, an UPDATE can't overwrite and a DELETE can't remove. So a "row" can't be one thing. It has to be a chain of versions:

logical row "alice":
 
  ┌─────────────────────┐    ┌─────────────────────┐
  │ v1: balance = 100   │ →  │ v2: balance = 50    │
  │ (old version)       │    │ (current version)   │
  └─────────────────────┘    └─────────────────────┘

• INSERT appends the first version.

• UPDATE appends a new version. The old one stays.

• DELETE appends... nothing. It just marks the latest version as dead. That solves writing. But it immediately creates the next problem: if both v1 and v2 physically exist, which one do you see when you read?

We need every version to answer two questions:

1. Who created me?

2. Who killed me? So we stamp every version with two transaction IDs. Postgres calls them xmin and xmax, and so will we:

type Txid = number;
 
interface Version {
  xmin: Txid;         // birth certificate: which transaction created this version
  xmax: Txid | null;  // death certificate: which transaction deleted it.
                      // null = no one has (yet). Note: a *stamp*, not a removal —
                      // dead versions stay physically present.
  value: unknown;
}
 
interface Row {
  // A "row" is not a value — it's the full history of values,
  // oldest first. Nothing in this array is ever removed or overwritten.
  versions: Version[];
}

Now the three write operations are almost embarrassingly simple:

• INSERT → push { xmin: me, xmax: null, value }

• DELETE → set xmax = me on the live version

• UPDATE → DELETE + INSERT. That's it. An update is literally a delete and an insert stapled together. (Run SELECT ctid FROM t before and after an UPDATE in Postgres and watch the row's physical address change — same trick.) Transactions themselves are just an incrementing counter:

class Database {
  // A "transaction" is nothing more than a number from this counter.
  // Because it only ever increases, txids double as timestamps:
  // smaller txid = started earlier. We'll lean on that ordering constantly.
  private nextTxid: Txid = 1;
  private rows = new Map<string, Row>();
 
  begin(): Txid {
    return this.nextTxid++;
  }
}

Have we solved MVCC? Let's check against the demo. Session A (txid 2) deletes everything: it sets xmax = 2 on every version. Session B (txid 3) reads: it sees versions with xmax = 2 set and... does what, exactly?

We stamped the versions, but we never said what the stamps mean for a reader. Next problem.

Problem 2: Reading uncommitted data (the dirty read)

Naive rule: "a version is visible if xmax is null."

Watch it fail. Session A has deleted everything but hasn't committed. Under the naive rule, Session B sees an empty table. Then Session A runs ROLLBACK — the delete never happened — and Session B has read a state of the world that never existed. This anomaly has a name: the dirty read.

The fix: a stamp only counts if the transaction that made it committed. Which means we need to track, for every txid, whether it committed, aborted, or is still running. Postgres keeps this in a structure called the commit log — pg_xact, historically clog. Ours is a Map:

type TxStatus = "in_progress" | "committed" | "aborted";
 
// The commit log: the single source of truth for "did txid N really happen?"
// Postgres calls this pg_xact (historically: clog).
type Clog = Map<Txid, TxStatus>;
 
class Database {
  private clog: Clog = new Map();
 
  begin(): Txid {
    const txid = this.nextTxid++;
    this.clog.set(txid, "in_progress");
    return txid;
  }
 
  // Commit touches ZERO rows. One map write flips the meaning of every
  // xmin/xmax stamp this transaction ever made, all at once, retroactively.
  commit(txid: Txid) { this.clog.set(txid, "committed"); }
 
  // Rollback is the same trick: nothing gets undone, because nothing
  // was done in place. The stamps stay — they just stop counting.
  abort(txid: Txid)  { this.clog.set(txid, "aborted"); }
}

Notice what commit is: one map write. It doesn't touch a single row. Session A deleted a million rows, and committing that is O(1) — we just flip 2 → "committed" and instantly, retroactively, all million xmax = 2 stamps start counting. Abort is the same trick in reverse: flip one entry and a million stamps become void. This is why ROLLBACK in Postgres is cheap even after a huge write — nothing gets undone, because nothing was done in place.

Improved rule: a version is visible if its xmin committed, and its xmax is null or belongs to a transaction that didn't commit.

Have we solved MVCC? Better — no more dirty reads. But run the demo again with a twist: Session B reads the count (1,000,000), then Session A commits, then Session B — still inside the same transaction — reads again and gets 0. B's world changed under its feet mid-transaction. That's the non-repeatable read, and it's the next problem.

Problem 3: The world moves while I'm reading it

What Session B actually wants is: "show me the database as it was at the moment I started, and keep showing me that, no matter what anyone else does."

It wants a snapshot.

Here's the moment the magic dissolves, so read this sentence twice: a snapshot is not a copy of the data. Copying a million rows per transaction would be insane. A snapshot is two numbers and a set:

// A frozen view of the world. NOT a copy of any data —
// just the two facts needed to judge any stamp, forever:
interface Snapshot {
  nextTxid: Txid;         // where the counter stood at my begin().
                          // Anything >= this is from my future: invisible.
  activeTxids: Set<Txid>; // who was still running at my begin().
                          // Their fate was undecided in my world — so even if
                          // they commit later, for me they never happened.
}
 
begin(): Txid {
  const txid = this.nextTxid++;
 
  // Photograph the in-flight transactions at this exact instant.
  // This set never changes for the life of the transaction —
  // that immutability IS the "repeatable" in repeatable reads.
  const activeTxids = new Set(
    [...this.clog].filter(([, s]) => s === "in_progress").map(([id]) => id)
  );
 
  this.snapshots.set(txid, { nextTxid: this.nextTxid, activeTxids });
  this.clog.set(txid, "in_progress");
  return txid;
}

That's the entire cost of "freeze the world": record where the txid counter was, and who was in flight. The data doesn't get frozen — the rules for judging stamps do.

The first rule the snapshot gives us: xmin < snap.nextTxid. If a version was created by a transaction that started after mine, it doesn't exist in my universe — even if it has already committed. My world ends at the counter value I saw at begin().

Have we solved MVCC? Almost — and the last gap is the subtle one that separates people who've read about MVCC from people who've built it.

Problem 4: The transaction from the past that commits in the future

Suppose transaction 5 is running when I (transaction 8) begin. Transaction 5 has a txid lower than my snapshot's nextTxid, so the rule from Problem 3 — xmin < 9 — says its writes are in my past.

But at the moment I took my snapshot, transaction 5 hadn't committed yet. If it commits while I'm running, the commit-status check from Problem 2 starts passing, the counter check from Problem 3 already passes, and transaction 5's writes leak into my snapshot mid-flight. The non-repeatable read is back, through the side door.

This is exactly what activeTxids is for. Third rule: a stamp from any transaction that was in flight when I began doesn't count for me — no matter what it does later. It can commit five milliseconds after my begin(); as far as I'm concerned, it never happened.

And with that, we have all three conditions for a stamp to "count," and no more problems left to solve. Time to assemble.

The visibility function: 20 lines that are the whole point

Every question in MVCC reduces to one question: given a version and a snapshot, can this transaction see it? Every rule we derived above becomes one condition:

// The one question all of MVCC reduces to:
// "Given this version and my snapshot, does it exist — for me?"
// Every line below is load-bearing: delete it, and a named anomaly returns.
function isVisible(v: Version, snap: Snapshot, clog: Clog, myTxid: Txid): boolean {
 
  // ── Part 1: was this version BORN, from my point of view? ──
  const xminVisible =
    // My own writes are always visible to me. Remove this and your own
    // INSERT vanishes from your own SELECT. (Not isolation — amnesia.)
    v.xmin === myTxid ||
 
    // A stranger's write counts only if ALL THREE hold:
    (clog.get(v.xmin) === "committed" &&   // it really happened
                                           //   (remove → DIRTY READS: you see
                                           //    data that later rolls back)
      v.xmin < snap.nextTxid &&            // it happened in my past
                                           //   (remove → transactions from your
                                           //    FUTURE appear in your present)
      !snap.activeTxids.has(v.xmin));      // its author wasn't mid-flight at my begin()
                                           //   (remove → a "past" txid commits late
                                           //    and leaks in: NON-REPEATABLE READS)
 
  if (!xminVisible) return false;   // never born (for me) → nothing to see
 
  // ── Part 2: has this version DIED, from my point of view? ──
 
  // No death certificate at all: alive for everyone.
  if (v.xmax === null) return true;
 
  // I stamped it myself → dead to me, instantly. (My DELETE must take
  // effect for ME before I commit — but for no one else. Same stamp,
  // different verdict per observer.)
  if (v.xmax === myTxid) return false;
 
  // A stranger's delete is judged by the SAME three conditions as a birth:
  // really happened, in my past, author not in flight at my begin().
  // Symmetry is the punchline: death plays by birth's rules.
  const xmaxApplies =
    clog.get(v.xmax) === "committed" &&
    v.xmax < snap.nextTxid &&
    !snap.activeTxids.has(v.xmax);
 
  // A delete that (for me) never happened leaves the version alive (for me).
  return !xmaxApplies;
}

Read the comments and notice that this function is our whole journey, compressed: the three conditions on xmin are exactly the fixes for Problems 2, 3, and 4 — really happened, in my past, author not in flight. And the punchline is the symmetry: the xmax half is the same three conditions, judging death instead of birth. A delete from an uncommitted transaction, a delete from your future, a delete from a transaction that was mid-flight when you began — none of them happened, for you.

A version is visible when its birth is inside your snapshot and its death is not.

That sentence is MVCC. Everything else is bookkeeping.

Break it yourself: the best way to convince yourself every condition is load-bearing is to delete one and watch its anomaly reappear. The full repo (linked at the end) has a test per condition — comment out a line, run the tests, and see exactly which guarantee falls over.

The operations become trivial

Here's the payoff for doing the hard thinking in one function — every operation is now a one-liner over version chains:

class Database {
  // ... nextTxid, rows, clog, snapshots from above ...
 
  // Reading = filtering history through a pure function.
  // No locks taken, no waiting, no coordination with writers —
  // which is exactly WHY readers never block anyone.
  read(txid: Txid, key: string): unknown | undefined {
    const snap = this.snapshots.get(txid)!;
    const row = this.rows.get(key);
    return row?.versions.find(v => isVisible(v, snap, this.clog, txid))?.value;
  }
 
  // INSERT = append a version stamped with my birth certificate.
  // Invisible to everyone else until my txid flips to "committed" in the clog.
  insert(txid: Txid, key: string, value: unknown) {
    const row = this.rows.get(key) ?? { versions: [] };
    row.versions.push({ xmin: txid, xmax: null, value });
    this.rows.set(key, row);
  }
 
  // DELETE = find the version *I* can see, stamp its death certificate.
  // One integer write. The data itself is untouched — still fully
  // readable by every snapshot that predates me.
  delete(txid: Txid, key: string) {
    const snap = this.snapshots.get(txid)!;
    const v = this.rows.get(key)?.versions
      .find(v => isVisible(v, snap, this.clog, txid));
    if (v) v.xmax = txid;
  }
 
  // UPDATE doesn't exist. It has never existed. It is a DELETE and an
  // INSERT wearing a trench coat — in our toy AND in Postgres
  // (that's why ctid changes on every UPDATE).
  update(txid: Txid, key: string, value: unknown) {
    this.delete(txid, key);
    this.insert(txid, key, value);
  }
}

read doesn't scan a lock table, doesn't wait, doesn't coordinate with anyone. It filters version chains through a pure function. That's why readers never block writers: reading is just math over stamps.

Replay the impossible demo — on our database

Back to the opening mystery, now against our 100 lines:

const db = new Database();
 
const setup = db.begin();                 // txid 1
db.insert(setup, "alice", { balance: 100 });
db.commit(setup);
 
const sessionA = db.begin();              // txid 2
const sessionB = db.begin();              // txid 3 — snapshot: { nextTxid: 4,
                                          //   activeTxids: {2} }  ← remember this set
 
db.delete(sessionA, "alice");             // alice's version is now:
                                          //   { xmin: 1, xmax: 2, balance: 100 }
                                          // ONE version. Watch it give two answers:
 
db.read(sessionB, "alice");  // → { balance: 100 }
                             //   B judges the death stamp: txid 2 is in B's
                             //   activeTxids → "that delete never happened" → alive
 
db.read(sessionA, "alice");  // → undefined
                             //   A judges the SAME stamp: xmax === myTxid
                             //   → my own delete → dead (to me alone)
 
db.commit(sessionA);         // one clog write: 2 → "committed"
 
db.read(sessionB, "alice");  // → { balance: 100 } — STILL alive for B!
                             //   B's snapshot froze at begin(); txid 2 stays in
                             //   activeTxids forever. A's commit changes the clog,
                             //   not B's snapshot. THIS is repeatable reads.

Look at the state after the delete: alice's version is { xmin: 1, xmax: 2, balance: 100 }. One physical version, two different answers:

• For A (txid 2): xmax === myTxid → rule 5 → dead.

• For B (txid 3): txid 2 was in B's activeTxids at begin → rule 6 fails → the delete never happened → alive. And it stays alive for B even after A commits, because B's snapshot doesn't change. The rows are deleted and not deleted — because "deleted" was never a property of the data. It's a property of the observer. The opening question wasn't a paradox; it was just the wrong question.

Don't take my word for it — ask Postgres

Our toy isn't "inspired by" Postgres. It's the same design, and Postgres will show you its stamps if you ask:

CREATE TABLE t (id int, val text);
INSERT INTO t VALUES (1, 'hello');
 
SELECT xmin, xmax, ctid, * FROM t;
--  xmin | xmax | ctid  | id | val
--   748 |    0 | (0,1) |  1 | hello
 
UPDATE t SET val = 'world';
 
SELECT xmin, xmax, ctid, * FROM t;
--  xmin | xmax | ctid  | id | val
--   749 |    0 | (0,2) |  1 | world

Every column maps to something we built:

Run SELECT pg_current_snapshot(); inside a transaction and you'll see something like 748:752:749,750 — "everything before 748 is my past, everything from 752 on is my future, and 749/750 were in flight when I began." Two numbers and a set. Nothing is magic.

Now the fun one. After the UPDATE above, where did version (0,1) — the 'hello' version — go?

VACUUM VERBOSE t;
-- INFO:  vacuuming "public.t"
-- INFO:  table "t": removed 1 dead item identifiers in 1 pages

It was still there. Dead, invisible to every current snapshot — but physically present until VACUUM reaped it.

What we deliberately punted

An honest accounting of the gap between our 100 lines and production Postgres:

• Write-write conflicts. Two transactions updating the same row: our toy lets the second one clobber the first's xmax. Postgres blocks the second writer until the first commits/aborts, then applies first-updater-wins. It's the one place MVCC does lock — writers against writers, never against readers.

• Command IDs (cmin/cmax). Postgres tracks visibility per statement within a transaction, not just per transaction, so a DELETE doesn't hide rows from itself mid-scan.

• SERIALIZABLE. Snapshot isolation — what we built — still permits an anomaly called write skew. Postgres's fix (SSI) is a genuinely great paper (Cahill et al.) and out of scope here.

• Hint bits and performance. Real Postgres doesn't consult the clog on every visibility check; it caches commit status in bits on the tuple itself. Same logic, heavy optimization.

• Garbage. The big one. Our founding decision — never update, never delete, only append — has a bill attached: dead versions pile up forever. Every UPDATE leaves a corpse. Postgres's answer is autovacuum, easily its most feared subsystem, and the source of half the Postgres war stories you've read. That bill, and how it gets paid, is the next post: MVCC's dirty secret — garbage.

The whole architecture, in one breath

Look back at what happened. We made one decision — append-only — and then fixed the problems it created, one at a time, adding only what each problem demanded:

1. Can't overwrite → version chains with xmin/xmax stamps

2. Stamps from uncommitted transactions → the clog

3. The world moving mid-read → snapshots (two numbers and a set)

4. In-flight transactions committing late → activeTxids

5. All four rules composed → the visibility function

6. Every operation → a trivial wrapper around it At no point did we design "MVCC." We just refused to update data in place and then dealt with the consequences honestly — and the entire architecture of Postgres's concurrency control fell out, in 100 lines.

Nothing is magic.

Previously in this series: Write-Ahead Log Explained: Build One in 30 Lines — where these versions would actually get persisted.

You write a row. The database says OK. A millisecond later, someone trips over the power cable.

When the machine boots back up, is your row still there?

If the answer is "yes," then some code ran before that OK was sent that made your write survive a crash that hadn't happened yet. That code is the write-ahead log. It sounds like the deepest, scariest part of a database. It's an append to a file and one system call.

We're going to derive it. One problem at a time, starting from a Map.

The store that forgets

Here's the simplest key-value store there is:

const store = new Map();
store.set("balance", 100);

Fast. Correct. Completely fake — it lives in RAM.

$ kill -9 <pid>

Every write since startup is gone. SIGKILL can't be caught, so there's no shutdown hook, no flush-on-exit, no goodbye. The process is simply deleted mid-run.

So, the problem, stated plainly:

How does a single write survive a crash?

Everything below is that one question, chased until it's answered properly.

The first step is obvious enough: put the data somewhere that isn't RAM.

Step 1: write it to a file

"Durable" just means "it's in a file," right? So after every change, save the map:

const fs = require("fs");
const store = new Map();

function set(key, value) {
  store.set(key, value);
  fs.writeFileSync("data.json", JSON.stringify([...store]));  // save everything
}

function load() {
  if (!fs.existsSync("data.json")) return;
  for (const [k, v] of JSON.parse(fs.readFileSync("data.json", "utf8"))) store.set(k, v);
}

And this works. Not "sort of" — kill the process, restart, call load(), and every write you made is there. As a statement of intent, "save everything after every change" is exactly right. (It has a nasty hole we'll come back to in Step 8, and another one below.)

It fails on cost, and it fails badly.

But first, the other hole, because it's worse than it looks: writeFileSync opens with "w", which truncates the file to zero before writing a single byte. Crash mid-snapshot and you haven't lost one write — you've lost the entire dataset. The real fix is to write a temp file and rename() it over the original, since rename is atomic. Keep that in mind: our very first attempt is the least crash-safe design in this whole article.

Say the store holds 2 GB and one set() changes a 20-byte value. To persist those 20 bytes, you write out all 2 GB. The ratio of bytes-shipped-to-disk versus bytes-actually-changed is called write amplification, and here it's:

2,000,000,000 / 20  ≈  100,000,000 : 1

A hundred million units of work for one unit of change. On an SSD sustaining 500 MB/s, serializing 2 GB takes ~4 seconds — so a single key update blocks for four seconds before it can return OK.

Worse, the cost is O(N) in your total data, not in the size of the change. Grow to 20 GB and every tiny write takes 40 seconds. You've built a store that punishes you for putting data in it.

The fix has to be: stop writing things that didn't change.

Step 2: only write what changed

If just 20 bytes changed, write 20 bytes. Seek to wherever that key lives on disk and overwrite it in place:

function set(key, value) {
  store.set(key, value);
  const offset = offsets.get(key);              // where this key lives on disk
  fs.writeSync(fd, serialize(value), 0, len, offset);   // overwrite it
}

The write is tiny again, and it stays tiny no matter how big the dataset gets. Problem solved — except we just created two new ones.

Random I/O. Seeking to an arbitrary offset for every write is the slowest thing storage does. On a spinning disk, seek latency caps you around a hundred writes per second. Even on an SSD, a 20-byte write forces the drive to read-modify-write an entire flash page underneath you. We traded one big sequential write for an endless stream of small random ones.

Corruption. This is the fatal one. You are mutating the only copy of data you already committed. If the machine dies halfway through overwriting that record, you haven't lost a pending write — you've destroyed a finished one, and there's no clean version to fall back to. Instead of losing the newest write, you lose an old write that was already safe.

In-place updates and crash safety are fundamentally at war.

The mistake both attempts share

Two attempts, two dead ends. Step back and they have the same root cause:

Both attempts try to keep an up-to-date copy of the data on disk.

Maintaining that copy is what forces the bad choice. Rewrite it wholesale and you pay O(N) per write. Edit it in place and you risk destroying committed data.

So stop storing the data.

Step 3: store the changes instead

Don't maintain the map on disk. Instead, append a one-line description of what happened to a log file — and never touch bytes you've already written:

const fd = fs.openSync("data.wal", "a");    // "a" = append only

function append(record) {
  fs.writeSync(fd, JSON.stringify(record) + "\n");
}

append({ op: "set", key: "balance", value: 100 });

The file now looks like a diary rather than a database:

{"op":"set","key":"balance","value":100}
{"op":"set","key":"balance","value":250}
{"op":"del","key":"temp"}

Look at what that bought us, against both failures at once:

• We only write the change. O(change), never O(N). A 20-byte update writes ~40 bytes of record, whether the store holds 2 MB or 2 TB.

• Every write is a sequential append. No seeking — the fastest access pattern a disk has, on spinning rust and SSDs alike.

• We never overwrite anything. Old records are immutable. A crash can only damage the record currently in flight — it can't reach back and corrupt a write that already succeeded.

That third point is the one that actually resolves Step 2's war. Append-only isn't just faster; it's structurally safer, because committed bytes are never in the line of fire.

One problem: this file is a list of changes, not a store. To read balance you'd have to scan the whole log every time.

Step 4: reads need memory

So keep the Map. It never stopped being useful — it was only ever the durability that was missing. Write to both: the log for survival, the map for speed.

function set(key, value) {
  append({ op: "set", key, value });   // durability
  store.set(key, value);               // speed
}

function get(key) {
  return store.get(key);               // pure memory, O(1)
}

Reads never touch the disk. Writes are one small sequential append. This is the shape almost every real storage engine has: an in-memory structure for serving, an append-only log for surviving.

Which leaves one hole. Restart the process and the Map is empty again.

Step 5: replay to rebuild

The log is a complete, ordered history of every change ever made. So on startup, read it top to bottom and re-apply each record. Whatever state you ended with, you get back:

function replay() {
  if (!fs.existsSync(path)) return;
  for (const line of fs.readFileSync(path, "utf8").split("\n")) {
    if (!line) continue;
    const r = JSON.parse(line);
    if (r.op === "set") store.set(r.key, r.value);
    if (r.op === "del") store.delete(r.key);
  }
}

This is why every record carries an op. A log of values alone can't express absence — "delete temp" is an event, and the only way to record an event is to name it. Replay is just re-running history.

Put it together and the store works — it's not finished, but it runs:

// wal.js
const fs = require("fs");

class WAL {
  constructor(path) {
    this.path = path;
    this.store = new Map();
    this.#replay();                       // rebuild memory from history
    this.fd = fs.openSync(path, "a");     // append handle for new writes
  }

  set(key, value) { this.#append({ op: "set", key, value }); this.store.set(key, value); }
  del(key)        { this.#append({ op: "del", key });        this.store.delete(key); }
  get(key)        { return this.store.get(key); }

  #append(record) {
    fs.writeSync(this.fd, JSON.stringify(record) + "\n");
  }

  #replay() { /* as above */ }
}

module.exports = WAL;

Now look closely at set(). It touches two places — the log and the map. Before worrying about their order, we need to pin down what we actually owe the caller.

Step 6: what exactly did we promise?

When set() returns, we've said OK. That's a promise: this write will survive a crash. Anything before the return is ours to lose freely. Anything after is a lie if we lose it. So the rule is:

The log record for an operation must be durable before the operation is acknowledged.

Walk every crash point against set():

The log always contains at least every acknowledged write. It might contain one extra — a record we logged but hadn't returned from yet — and that's harmless, since replaying it just leaves state slightly newer than promised. What it can never do is lack a write we already confirmed. That would be a lie.

Now, the ordering. Look at the table again and notice something uncomfortable: swapping the two lines inside set() wouldn't break a single row. Our map is volatile — it's gone on crash either way — and the guarantee is anchored to the return, not to the map. So why insist on log-first?

Because that's an accident of our map living in RAM. The moment the structure you're updating is also on disk — real data pages, a B-tree, an SSTable — the order becomes everything. Mutate the page first, crash, and you're left with a half-modified persistent structure and no log record to redo or undo it: unrecoverable corruption. Log first, and recovery can always repair the page from the record.

That's what "write-ahead" means — the log lands ahead of the acknowledgment, and ahead of the data it describes. We'll keep log-first from here on, because that's the invariant that survives contact with a real storage engine.

Step 7: does it survive? kill -9

Enough theory. Let's kill it.

// writer.js
const WAL = require("./wal");
const db = new WAL("data.wal");
let n = 0;
setInterval(() => {
  db.set(`key:${n}`, { n, at: Date.now() });
  console.log("wrote", `key:${n++}`);
}, 500);

$ node writer.js
wrote key:0
wrote key:1
wrote key:2
wrote key:3
# in another terminal:
$ kill -9 $(pgrep -f writer.js)

No cleanup ran. No handler fired. Now read it back:

// reader.js
const WAL = require("./wal");
const db = new WAL("data.wal");
console.log([...db.store.entries()]);

$ node reader.js
[ [ 'key:0', {…} ], [ 'key:1', {…} ], [ 'key:2', {…} ], [ 'key:3', {…} ] ]

Every acknowledged write came back. We murdered the process and lost nothing.

Notice something, though: there is no fsync anywhere in this code. If durability supposedly requires forcing bytes to disk, why did this just work? And if it works without it — what exactly are we still exposed to?

Step 8: surviving the power cut

Here's the part most WAL tutorials get wrong.

fs.writeSync() does not put your bytes on the disk. It copies them into the operating system's page cache — kernel memory — and returns. The kernel flushes to the physical device later, on its own schedule.

That explains the test above. Once the bytes are in the page cache, they belong to the kernel, not your process. Your process can die any way it likes — kill -9, segfault, OOM kill, uncaught throw — and the kernel still writes them out. A process crash cannot lose data that's already in the page cache.

So write() alone buys real durability against your program crashing. That's the majority of crashes, and it's genuinely why the demo in Step 7 passed.

What write() does not survive is the operating system dying with those bytes still in its cache: a power cut, a kernel panic, a yanked plug. The page cache is RAM, and RAM doesn't survive power loss. Anything not yet flushed evaporates — including writes you already answered OK to. That breaks the invariant.

fsync() is the syscall that says do not return until these bytes are physically on the storage device:

  #append(record) {
    fs.writeSync(this.fd, JSON.stringify(record) + "\n");  // → page cache
    fs.fsyncSync(this.fd);                                 // → physical disk
  }

That one line is the border between "safe if my program crashes" and "safe if the building loses power."

It's also, by far, the expensive part. writeSync is a memcpy into kernel memory — nanoseconds. fsyncSync blocks on physical hardware — often a millisecond or more, and it's the reason every serious database gives you a knob to trade durability against throughput. We'll see three of those knobs shortly.

One more thing our 30 lines quietly assume: fs.writeSync returns the number of bytes it actually wrote, and it's allowed to write fewer than you asked for. Production code loops until the buffer is drained. For small appends to a regular file you'll effectively never see it — but "effectively never" is doing real work in that sentence.

Step 9: the half-written record

One crash case from Step 6's table deserves its own section: crashing mid-append.

writeSync is not atomic. Lose power partway through and the last line in the file is a fragment:

{"op":"set","key":"key:40","value":{"n":40,"at":1753500000000}}
{"op":"set","key":"key:41","value":{"n":41,"at":17

That's a torn record, and JSON.parse throws on it. Which means our replay loop crashes on startup — the durability feature has become a boot failure. Impressive own goal.

The fix is one line:

    let r;
    try { r = JSON.parse(line); } catch { break; }   // torn tail — stop replaying

Stop at the first line that won't parse, and keep everything before it.

It's worth checking that this heuristic actually holds rather than assuming it. Take one full record and try to parse every possible truncation of it: of the 57 cut points in {"op":"set","key":"a","value":{"n":41,"at":1753500000000}}, zero produce valid JSON. Truncation always removes a closing brace or quote, so a torn record can't masquerade as a complete one. For this format, the crude fix is genuinely sound — not by luck.

This is safe, and it's safe because of the invariant — not by luck:

1. A torn record can only ever be the last record. Nothing gets appended after a crash, so there's no valid data hiding behind the damage.

2. A torn record was never acknowledged. The fsync inside #append hadn't returned, so set() hadn't returned, so the caller was never told OK. Discarding it breaks no promise.

Truncating at the tear leaves the log as a clean prefix of history — exactly the guarantee we defined in Step 6.

Catching a JSON.parse exception is a crude way to detect corruption, but the idea is exactly right. So let's go see how the grown-ups implement it.

The finished thing

Every patch folded back in:

// wal.js — a crash-safe key-value store
const fs = require("fs");

class WAL {
  constructor(path) {
    this.path = path;
    this.store = new Map();
    this.#replay();                     // rebuild memory from history
    this.fd = fs.openSync(path, "a");   // append-only handle
  }

  set(key, value) { this.#append({ op: "set", key, value }); this.store.set(key, value); }
  del(key)        { this.#append({ op: "del", key });        this.store.delete(key); }
  get(key)        { return this.store.get(key); }

  #append(record) {
    fs.writeSync(this.fd, JSON.stringify(record) + "\n");  // → page cache
    fs.fsyncSync(this.fd);                                 // → physical disk
  }

  #replay() {
    if (!fs.existsSync(this.path)) return;
    for (const line of fs.readFileSync(this.path, "utf8").split("\n")) {
      if (!line) continue;
      let r;
      try { r = JSON.parse(line); } catch { break; }   // torn tail — stop
      if (r.op === "set") this.store.set(r.key, r.value);
      if (r.op === "del") this.store.delete(r.key);
    }
  }
}

module.exports = WAL;

Thirty lines. Append-only for cheap writes, fsync for power loss, replay for boot, break for the torn tail. Every line is there because a specific failure put it there.

Proof: the real thing

Nothing is magic, so let's check our 30 lines against production source.

LevelDB (db/log_writer.cc, db/log_reader.cc) is our toy, hardened. Where we write newline-delimited JSON, it frames each record with a 7-byte header — 4-byte CRC-32C, 2-byte length, 1-byte type — inside fixed 32 KB blocks. On recovery it recomputes the CRC and drops a bad or short final record, exactly where our break gives up: a checksum instead of a thrown exception, same decision. And it's literally write-ahead — DBImpl::Write appends to the log first, then applies to the memtable, which is an in-memory map just like ours. WriteOptions.sync is what turns on the fsync.

PostgreSQL exposes the knobs. synchronous_commit decides whether COMMIT waits for the WAL flush — turn it off and commits return before they're durable, which is faster and means a crash can lose the last few transactions without corrupting anything. wal_sync_method picks the actual syscall (fdatasync, fsync, …). And our torn-record problem gets solved at the page level by full_page_writes: the first change to a page after a checkpoint logs the entire page, so a torn page can be rebuilt from the WAL. (If you've been reading the Postgres 18 io_uring work — this is the write path it feeds.)

SQLite makes the tradeoff a single config line: PRAGMA synchronous = OFF | NORMAL | FULL. That's our fsyncSync, promoted to a user-facing dial.

Same primitive underneath all three: append a record, choose how hard to fsync, replay on boot, tolerate a torn tail.

The one problem we didn't solve

Two loose ends, one small and one large.

The small one: fsync on a file isn't always enough. When you create a new file, the filename itself lives in the parent directory, and that directory entry needs its own fsync to survive a crash — otherwise a power cut can leave you with durable bytes and no name pointing at them. And we do hit this: on first run, fs.openSync(path, "a") creates data.wal. Every append after that is safe; the very first one needs an fsync on the parent directory too. Ours doesn't do it.

The large one: the log grows forever. Every set appends, even overwriting the same key a thousand times. Nothing is ever removed. Replay gets slower every day the process runs, and the file eventually eats the disk.

Which means the log needs a way to say: I've folded everything up to here into the main data structure, so the old records can go. That's a checkpoint — and when the main data structure is a sorted tree on disk, checkpointing becomes compaction, the beating heart of every LSM engine.

That's the next brick. See you there.

If this was useful, the connection pooling post is its older sibling.

I used to think request queuing and connection pooling were deep infrastructure magic — something libraries did that I could never see. Then I built each piece myself in ~30 lines of TypeScript, and the whole thing turned out to be arrays, counters, and one key insight about what a "connection" actually is.

This post builds the idea bottom-up. By the end, connection pooling should feel obvious — not memorized, but inevitable.

Building Block 1: A connection is just a held reference

When a client calls fetch(), the kernel opens a TCP connection identified by a unique 4-tuple: (client IP, client port, server IP, server port). Ten simultaneous requests from the same laptop use up to ten ephemeral client ports — ten distinct sockets. (Browsers cap this at ~6 per origin and queue the rest — yet another queue. Sequential requests reuse one connection via keep-alive, and HTTP/2 multiplexes everything over a single socket.)

When Node accepts one, it hands your callback a (req, res) pair. Here's the insight that unlocked everything for me:

res is not "response data". It is a handle to one specific open socket.

There's no ID matching, no "which user does this response belong to" lookup. The correlation is structural — res is the write-end of that exact connection. As long as you hold the reference, you can reply later — even minutes later — and the bytes go down the right socket. (Not indefinitely: Node kills requests after a timeout, 5 minutes by default, and the client may give up first.)

Which means: a connection is something you can store. Put res in an array and answer when you're ready. That single trick is what queuing is.

Building Block 2: A queue is an array and a counter

Say I only want to process 2 requests at a time, and allow at most 5 to wait in line. I need two limits, a counter, and an array:

const MAX_CONCURRENT = 2;      // how many requests we process at once
const MAX_QUEUE = 5;           // how many can wait before we start rejecting

let active = 0;                // how many are being processed right now
const waiting: Job[] = [];     // everyone else — this array IS the queue

http.createServer((req, res) => {
  const job = { res, enqueuedAt: Date.now() };

  if (active < MAX_CONCURRENT) {
    handle(job);               // free slot → run now
  } else if (waiting.length < MAX_QUEUE) {
    waiting.push(job);         // ENQUEUE: res sits in memory, client keeps waiting
  } else {
    res.writeHead(503).end();  // queue full → shed load
  }
});

And when a job finishes, pull the next one:

async function handle(job: Job) {
  active++;
  await doWork();              // the slow part — say each job takes 2 seconds
  active--;
  job.res.end("done");         // reply NOW — the socket was open this whole time
  drain();
}

function drain() {
  while (active < MAX_CONCURRENT && waiting.length > 0) {
    handle(waiting.shift()!);  // DEQUEUE: FIFO, next in line gets the slot
  }
}

Now run this and fire 10 concurrent requests. Assume each job takes 2 seconds of work: with only 2 slots, the server drains the queue in waves of 2, every 2 seconds:

• t=0s — requests 1–2 grab the free slots and start immediately. Requests 3–7 land in waiting[]. Requests 8–10 find the queue full and get instant 503s.

• t=2s — requests 1–2 finish, freeing both slots. drain() pulls requests 3–4 off the queue. They waited 2s.

• t=4s — requests 3–4 finish. Requests 5–6 start. They waited 4s.

• t=6s — requests 5–6 finish. Request 7 finally starts. It waited 6s.

That's where the climbing wait times come from: every request ahead of you in the queue costs you a share of a work-cycle. In general, the request at queue position p waits about ceil(p / MAX_CONCURRENT) × WORK_MS. This is the arithmetic behind every "slow server" you've ever hit — your wait grows with how deep in the queue you land, in steps of (work time ÷ worker count). It's also why "add more instances" fixes latency under load: double the workers and the same queue position waits half as long.

While a request "queues", nothing mystical happens — its res object sits in an array, its TCP socket stays open, and the client's fetch() promise simply hasn't resolved yet.

(Below this array there are more queues you don't manage: the kernel's accept queue, socket buffers, Node's event loop. Same concept at every layer — bytes parked in a buffer, waiting for capacity. A request is never "floating"; it's always parked somewhere specific.)

Building Block 3: Connections are expensive, requests are cheap

One more fact before the payoff. Opening a database connection costs real time:

1. TCP handshake (a network round trip)

2. TLS negotiation (another round trip — two on older TLS 1.2)

3. Authentication (password/SCRAM exchange)

4. The DB allocating resources

That last step is the killer, and it's a design decision from the 1980s we still live with: Postgres forks an entire OS process per connection. Process isolation was the robust concurrency primitive of that era — but it means every connection costs the database real memory, and a Postgres server comfortably holds hundreds of connections, not tens of thousands.

So connections are slow to create (easily 5–50ms, often more than the query itself) and expensive for the server to hold in bulk. Creating one per query would be like hiring and firing an employee for every task. The obvious move: pay the setup cost once, keep the connection alive, and share it.

But "keep it alive" — where? Same answer as Building Block 1. In an array. An idle DB connection is just an authenticated, open socket wrapped in a JS object, sitting in your process memory doing nothing, ready to be handed out.

The Payoff: A pool is the queue, inverted

Look at what we've built:

• Request queue: work waits in an array for a free slot.

• Connection pool: connections wait in an array for incoming work.

Same machinery, mirrored. And when the pool runs dry, it flips back into our request queue — callers wait in line for a connection:

const MAX_POOL_SIZE = 10;      // most connections we'll ever open

class Pool {
  private idle: Connection[] = [];                     // connections waiting for work
  private waiters: ((c: Connection) => void)[] = [];   // work waiting for connections
  private total = 0;

  async acquire(): Promise<Connection> {
    if (this.idle.length > 0)
      return this.idle.pop()!;              // reuse: no handshake, ~0ms

    if (this.total < MAX_POOL_SIZE) {
      this.total++;
      return await createConnection();      // expensive path, done rarely
    }

    // pool exhausted → the caller queues (Building Block 2 again!)
    return new Promise(resolve => this.waiters.push(resolve));
  }

  release(conn: Connection) {
    const waiter = this.waiters.shift();
    if (waiter) waiter(conn);               // hand straight to next in line
    else this.idle.push(conn);              // nobody waiting → back on the shelf
  }
}

Usage:

const conn = await pool.acquire();
try {
  await conn.query("SELECT ...");
} finally {
  pool.release(conn);   // forget this and you have a "connection leak"
}

That's it. That's connection pooling. Two arrays and a counter.

Everything you've heard about pools now decodes for free:

• "Pool exhaustion" — all connections checked out, the waiters array is growing. Usually a missing release() or slow queries hogging connections.

• "Acquire timeout" — a waiter that gives up (rejects its promise) after N ms instead of queuing forever.

• "Idle timeout" — a janitor that closes sockets sitting unused in idle for too long, so the DB isn't holding a process for nothing.

• "Max pool size" — the bound on total, playing the same role MAX_CONCURRENT did in our request queue. (Some pools also take a min size: connections pre-created and kept warm — our toy has no equivalent.)

Proof: the real thing is the same two arrays

Don't take my word for it — open pg-pool/index.js, the pool used by node-postgres. Its constructor has:

this._clients = []        // every connection that exists
this._idle = []           // connections sitting alive, waiting for work
this._pendingQueue = []   // callers waiting because the pool is full

Our toy pool, with underscores. "Holding a connection alive" is literally an object pushed onto _idle:

class IdleItem {
  constructor(client, idleListener, timeoutId) {
    this.client = client              // wraps a live net.Socket to Postgres
    this.idleListener = idleListener  // if the server dies while parked, evict
    this.timeoutId = timeoutId        // idle-timeout janitor
  }
}

The client holds the underlying socket, the array holds the IdleItem, so nothing closes or garbage-collects the connection. A live socket, held by an object, held by an array — the same trick as parking res in our HTTP queue.

Our drain() is their _pulseQueue(): _pendingQueue.shift() to dequeue the next waiter (FIFO), _idle.pop() to grab a parked connection, newClient() if there's room to grow. Even the production metrics are just array lengths:

get waitingCount() { return this._pendingQueue.length }
get idleCount()    { return this._idle.length }
get totalCount()   { return this._clients.length }

The remaining ~400 lines are exactly the hardening we predicted: idle timeouts, acquire timeouts, maxUses/maxLifetimeSeconds rotation, error listeners on parked clients, double-release protection.

One elegant quirk: reuse is _idle.pop() — LIFO, not FIFO. That looks unfair until you see why: handing out the most-recently-used connection keeps a small hot set busy, while rarely-used connections quietly age toward their idle timeout and close themselves. The pool shrinks to fit actual load. A FIFO pool would keep every connection just-barely-alive forever.

The bigger lesson

Your typical Node service is queues all the way down: HTTP requests queue for the event loop, handlers queue for pool connections, queries queue inside the database, and TCP backpressure queues bytes all the way back to the client when anything fills up. None of it is magic. At every layer it's the same primitive:

Hold a reference in an array. Hand it out when there's capacity.

Whenever infrastructure feels magical, build the naive 30-line version. The real thing is almost always the naive version plus error handling — and the next time someone says "we're seeing pool exhaustion," you won't picture magic. You'll picture an array named waiters, and it's growing.

PostgreSQL is one of the most extensible databases ever built. Its extension system lets you add custom data types, functions, operators, and even entirely new storage engines. Traditionally, this power came with a cost: you had to write C code.

C is fast, but it's also unforgiving. A single buffer overflow or null pointer dereference can crash your entire database server—taking all your connections and transactions down with it. For a production database, that's terrifying.

Enter Rust.

In this guide, I'll show you how to build PostgreSQL extensions using Rust and the pgrx framework. We'll use a real extension—pg_mask—as our working example: a set of data masking functions for GDPR and privacy compliance.

Why Rust Over C?

If you've written C for PostgreSQL before, you know the pain:

• Memory management is manual. Forget to pfree() something? Memory leak. Free it twice? Crash.

• Error handling is tedious. PostgreSQL's ereport() system works, but it's easy to get wrong.

• Tooling is antiquated. No package manager, no standardized testing, dependency management via copy-paste.

Rust solves all of this:

The key insight: Rust gives you C's performance without C's footguns. Your PostgreSQL server won't crash because of a segfault in your extension code.

The pgrx Framework

pgrx is the bridge between Rust and PostgreSQL. It handles:

• FFI bindings: All the unsafe C interop is wrapped in safe Rust APIs

• Memory contexts: Automatically integrates with PostgreSQL's memory management

• Error handling: Rust panics become PostgreSQL errors (no crashes!)

• SQL generation: Your Rust function signatures automatically become SQL function definitions

• Development server: Spin up a PostgreSQL instance with your extension pre-loaded

The magic is that pgrx lets you write idiomatic Rust while producing extensions that are indistinguishable from C extensions to PostgreSQL.

Project Structure: Anatomy of a pgrx Extension

Let's look at pg_mask, a data masking extension. Here's the complete project structure:

pg_mask/
├── .cargo/
│   └── config.toml         # Platform-specific linker settings
├── src/
│   ├── lib.rs              # Your extension code lives here
│   └── bin/
│       └── pgrx_embed.rs   # Required for SQL generation
├── Cargo.toml              # Rust project configuration
├── pg_mask.control         # PostgreSQL extension metadata
└── SETUP.md                # Documentation

That's it. Five files to create a production-ready PostgreSQL extension.

Let's examine each file.

Cargo.toml: Project Configuration

[package]
name = "pg_mask"
version = "0.1.0"
edition = "2024"

[lib]
crate-type = ["cdylib", "lib"]

[[bin]]
name = "pgrx_embed_pg_mask"
path = "src/bin/pgrx_embed.rs"

[features]
default = ["pg18"]
pg18 = ["pgrx/pg18", "pgrx-tests/pg18"]
pg_test = []

[dependencies]
pgrx = "0.16"

[dev-dependencies]
pgrx-tests = "0.16"

Key points:

• crate-type = ["cdylib", "lib"]: Builds both a C-compatible shared library (for PostgreSQL to load) and a Rust library (for testing)

• Feature flags for PostgreSQL versions: pgrx supports multiple PostgreSQL versions. Pick yours with a feature flag.

• Single dependency: Just pgrx. That's all you need.

pg_mask.control: Extension Metadata

Every PostgreSQL extension needs a .control file:

comment = 'pg_mask: Data masking functions for GDPR/privacy compliance'
default_version = '@CARGO_VERSION@'
module_pathname = '$libdir/pg_mask'
relocatable = false
superuser = false

• comment: Shows up in \dx listings

• default_version: pgrx replaces @CARGO_VERSION@ with your Cargo.toml version

• module_pathname: Where PostgreSQL looks for your shared library

• superuser = false: Regular users can use these functions (appropriate for data masking)

Important: The control file name must match your extension name (not necessarily the crate name). Extension name pg_mask → pg_mask.control.

The Embedding Binary

// src/bin/pgrx_embed.rs
::pgrx::pgrx_embed!();

This one-liner is required by pgrx for SQL generation. It generates the PostgreSQL-compatible module initialization code and the SQL schema from your Rust function definitions. Without it, cargo pgrx commands will fail.

macOS Linker Configuration

If you're developing on macOS, you need this in .cargo/config.toml:

[target.aarch64-apple-darwin]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]

Why? PostgreSQL extensions are shared libraries that reference symbols from the PostgreSQL server binary. On macOS, the linker normally requires all symbols to be resolved at link time. The -undefined dynamic_lookup flags tell the linker "trust me, these symbols will exist at runtime."

Without this, your build will fail with undefined symbol errors for PostgreSQL internal functions.

Writing Extension Functions

Now the fun part. Here's the complete src/lib.rs:

use pgrx::prelude::*;

pgrx::pg_module_magic!();

/// Masks an email address, showing only first char and domain
/// Example: "john.doe@example.com" -> "j*******@example.com"
#[pg_extern]
fn mask_email(email: &str) -> String {
    match email.split_once('@') {
        Some((local, domain)) => {
            if local.is_empty() {
                format!("@{}", domain)
            } else {
                let first_char = local.chars().next().unwrap();
                let mask_len = local.len().saturating_sub(1);
                format!("{}{}@{}", first_char, "*".repeat(mask_len), domain)
            }
        }
        None => "*".repeat(email.len()),
    }
}

/// Masks a credit card number, showing only last 4 digits
/// Example: "4111111111111111" -> "****-****-****-1111"
#[pg_extern]
fn mask_card(card: &str) -> String {
    let digits: String = card.chars().filter(|c| c.is_ascii_digit()).collect();
    if digits.len() >= 4 {
        let last_four = &digits[digits.len() - 4..];
        format!("****-****-****-{}", last_four)
    } else {
        "*".repeat(digits.len())
    }
}

/// Masks a phone number, showing only last 4 digits
/// Example: "+1 (555) 123-4567" -> "***-***-4567"
#[pg_extern]
fn mask_phone(phone: &str) -> String {
    let digits: String = phone.chars().filter(|c| c.is_ascii_digit()).collect();
    if digits.len() >= 4 {
        let last_four = &digits[digits.len() - 4..];
        format!("***-***-{}", last_four)
    } else {
        "*".repeat(digits.len())
    }
}

/// Masks a SSN, showing only last 4 digits
/// Example: "123-45-6789" -> "***-**-6789"
#[pg_extern]
fn mask_ssn(ssn: &str) -> String {
    let digits: String = ssn.chars().filter(|c| c.is_ascii_digit()).collect();
    if digits.len() >= 4 {
        let last_four = &digits[digits.len() - 4..];
        format!("***-**-{}", last_four)
    } else {
        "*".repeat(digits.len())
    }
}

/// Generic masking: shows first N and last M characters
/// Example: mask_text("sensitive data", 2, 2) -> "se**********ta"
#[pg_extern]
fn mask_text(text: &str, show_first: i32, show_last: i32) -> String {
    let show_first = show_first.max(0) as usize;
    let show_last = show_last.max(0) as usize;
    let len = text.chars().count();  // Use chars().count() for Unicode safety

    if show_first + show_last >= len {
        return text.to_string();
    }

    let first: String = text.chars().take(show_first).collect();
    let last: String = text.chars().skip(len - show_last).collect();
    let mask_len = len - show_first - show_last;

    format!("{}{}{}", first, "*".repeat(mask_len), last)
}

/// Masks an IP address
/// IPv4: "192.168.1.100" -> "192.168.xxx.xxx"
/// IPv6: Shows first segment only
#[pg_extern]
fn mask_ip(ip: &str) -> String {
    if ip.contains(':') {
        // IPv6
        let parts: Vec<&str> = ip.split(':').collect();
        if parts.is_empty() {
            return "xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx".to_string();
        }
        let first = parts[0];
        let masked: Vec<&str> = std::iter::once(first)
            .chain(std::iter::repeat("xxxx").take(parts.len() - 1))
            .collect();
        masked.join(":")
    } else {
        // IPv4
        let parts: Vec<&str> = ip.split('.').collect();
        if parts.len() >= 2 {
            format!("{}.{}.xxx.xxx", parts[0], parts[1])
        } else {
            "xxx.xxx.xxx.xxx".to_string()
        }
    }
}

Let's break down the key concepts.

pg_module_magic!()

pgrx::pg_module_magic!();

This macro generates the PostgreSQL module magic block—a special struct that PostgreSQL checks to verify the extension was compiled for the right PostgreSQL version. In C, you'd write this manually. In Rust, it's one line.

#[pg_extern]: Exposing Functions to SQL

The #[pg_extern] attribute is where the magic happens:

#[pg_extern]
fn mask_email(email: &str) -> String {
    // ...
}

This single annotation:

1. Makes the function callable from SQL

2. Generates the SQL CREATE FUNCTION statement

3. Handles type conversions between PostgreSQL and Rust

4. Wraps panics in proper PostgreSQL error handling

The function signature fn mask_email(email: &str) -> String automatically becomes:

CREATE FUNCTION mask_email(email TEXT) RETURNS TEXT

pgrx handles the mapping:

Writing Safe Code

Notice how the code handles edge cases:

match email.split_once('@') {
    Some((local, domain)) => {
        // Valid email with @
    }
    None => "*".repeat(email.len()),  // No @ found
}

In C, you'd be doing pointer arithmetic and praying you don't overflow. In Rust, the compiler ensures you handle both cases. If you forget the None branch, your code won't compile.

Building and Running

Initial Setup

# Install the pgrx CLI tool
cargo install cargo-pgrx --version 0.16.1 --locked

# Initialize pgrx (required before first build!)
cargo pgrx init --pg18=$(which pg_config)

The init step downloads PostgreSQL headers and sets up the build environment. You must run this before your first build.

Development Workflow

# Start a PostgreSQL instance with your extension loaded
cargo pgrx run pg18

This command:

1. Compiles your extension

2. Starts a temporary PostgreSQL server

3. Connects you to a psql session

From there:

-- Load the extension
CREATE EXTENSION pg_mask;

-- Test your functions
SELECT mask_email('john.doe@example.com');
-- Returns: j*******@example.com

SELECT mask_card('4111111111111111');
-- Returns: ****-****-****-1111

SELECT mask_phone('+1 (555) 123-4567');
-- Returns: ***-***-4567

SELECT mask_ssn('123-45-6789');
-- Returns: ***-**-6789

SELECT mask_text('sensitive data', 2, 2);
-- Returns: se**********ta

SELECT mask_ip('192.168.1.100');
-- Returns: 192.168.xxx.xxx

Running Tests

pgrx includes a test framework that runs tests inside a real PostgreSQL instance:

#[cfg(any(test, feature = "pg_test"))]
#[pg_schema]
mod tests {
    use super::*;

    #[pg_test]
    fn test_mask_email() {
        assert_eq!(
            mask_email("john.doe@example.com"),
            "j*******@example.com"
        );
    }

    #[pg_test]
    fn test_mask_card() {
        assert_eq!(
            mask_card("4111111111111111"),
            "****-****-****-1111"
        );
    }
}

Run tests with:

cargo pgrx test pg18

Packaging for Production

cargo pgrx package

This creates a distributable package in target/release/pg_mask-pg18/ containing:

• The compiled shared library

• SQL installation scripts

• The control file

Common Gotchas

1. Control File Name = Extension Name

The .control file must match your extension name (not necessarily the crate name):

• Extension name: pg_mask

• Control file: pg_mask.control

2. The pgrx_embed Binary is Required

The embedding binary is required for SQL generation. Without it, cargo pgrx commands will fail:

// src/bin/pgrx_embed.rs
::pgrx::pgrx_embed!();

And referenced in Cargo.toml:

[[bin]]
name = "pgrx_embed_pg_mask"  # Must be pgrx_embed_<extension_name>
path = "src/bin/pgrx_embed.rs"

3. macOS Needs Special Linker Flags

On macOS, you must add -undefined dynamic_lookup linker flags in .cargo/config.toml. Without these, linking fails with undefined symbol errors for PostgreSQL functions.

4. Must Run cargo pgrx init Before First Build

Before your first build, initialize pgrx:

cargo pgrx init --pg18=$(which pg_config)

This sets up PostgreSQL headers and the build environment. Skip this and you'll get cryptic build errors.

5. PostgreSQL Version Must Match Feature Flag

Your feature flag must match your target PostgreSQL version:

[features]
default = ["pg18"]  # Must match your PostgreSQL installation
pg15 = ["pgrx/pg15", "pgrx-tests/pg15"]
pg16 = ["pgrx/pg16", "pgrx-tests/pg16"]
pg17 = ["pgrx/pg17", "pgrx-tests/pg17"]
pg18 = ["pgrx/pg18", "pgrx-tests/pg18"]

If you're targeting PostgreSQL 16 but have default = ["pg18"], you'll get version mismatch errors at runtime.

Beyond Simple Functions

pg_mask demonstrates simple scalar functions, but pgrx supports much more:

Custom Types

#[derive(PostgresType, Serialize, Deserialize)]
pub struct Point {
    x: f64,
    y: f64,
}

Aggregate Functions

#[pg_aggregate]
impl Aggregate for MySum {
    type State = i64;
    type Args = i32;
    // ...
}

Background Workers

#[pg_guard]
pub extern "C" fn my_worker_main(_arg: pg_sys::Datum) {
    // Runs in background
}

Server Programming Interface (SPI)

#[pg_extern]
fn count_rows(table_name: &str) -> i64 {
    Spi::connect(|client| {
        let query = format!("SELECT COUNT(*) FROM {}", table_name);
        client.select(&query, None, None)?
            .first()
            .get_one::<i64>()?
            .unwrap_or(0)
    })
}

Conclusion

Building PostgreSQL extensions with Rust and pgrx is genuinely enjoyable. You get:

• Memory safety: No more debugging segfaults at 3 AM

• Modern tooling: Cargo, crates.io, rust-analyzer

• Type safety: The compiler catches mistakes before they hit production

• Performance: Rust compiles to native code, matching C speed

The pg_mask extension we walked through is ~100 lines of Rust. A C equivalent would be 300+ lines, riddled with manual memory management and error handling boilerplate.

If you've been hesitant to write PostgreSQL extensions because of C's complexity, give pgrx a try. The learning curve is gentler than you might expect, especially if you already know Rust.

Source Code

Full source code for pg_mask: github.com/jatin510/postgres-ext-exp

Resources

• pgrx GitHub Repository

• pgrx Documentation

• PostgreSQL Extension Documentation

Ideas for Your First Extension

• Data validation functions: Email formats, phone numbers, URLs

• Text processing: Fuzzy matching, phonetic encoding, slugification

• Encryption wrappers: Field-level encryption with key management

• External API integrations: HTTP clients, queue publishers

• Custom index types: Specialized data structures for your domain

The PostgreSQL extension ecosystem needs more Rust. Go build something.

As someone who builds Rust-based systems every day — from high-throughput analytics pipelines to Web Crawler — I often find myself walking the fine line between speed and safety.

Rust gives me production-grade reliability, but I used to believe it wasn’t meant for rapid prototyping. Corrode challenged that idea — and they were right.

Reading Corrode’s Take on Prototyping

The post made a simple but powerful claim:

“You can prototype effectively in Rust — if you allow yourself to write imperfect code first.”

(Source: Corrode.dev)

It laid out a refreshing way to approach Rust in the early stages of an idea. Some of the lessons that stood out to me:

• Start messy, refine later. Don’t build abstractions too soon.

• Use simple types. Stick to owned values (String, Vec<T>) — references can wait.

• Lean on inference. Let the compiler infer types for you.

• Don’t fear unwrap(). It’s fine in a prototype; clarity is more important than polish.

• Keep it flat. One main.rs file is enough when you’re exploring.

• Avoid early optimization. Let the design breathe before tuning performance.

The tone of the article was liberating — it felt like permission to just build, not over-engineer.

Applying It to My Own Rust Projects

I decided to apply Corrode’s mindset to one of my ongoing practice projects — a Trie-based autocomplete system.

Normally, I would architect everything first: multiple modules, traits, and generic abstractions. This time, I didn’t.

Here’s what changed:

1. Flat structure first

   Instead of setting up a full module hierarchy, I began with a single file to keep my focus on the logic and flow. Once the core idea was validated, I refactored it into modules. That early simplicity made iteration much faster.

2. Concrete types everywhere

   Instead of juggling lifetimes and generics, I used owned types like String and Vec<Node>. When the structure became stable, I started refactoring.

3. Fearless use of unwrap() and todo!()

   They became markers of progress. Every unwrap() told me, “this part works for now — fix it later.”

4. Delayed optimization

   My first goal was correctness and clarity. Only later did I switch my lookups from a Vec to a HashMap, and profiling confirmed that’s all I needed.

Surprisingly, I had a working prototype in a single evening — clean, testable, and fast enough to ship internally.

Why I Love Rust

Corrode’s article didn’t just change how I prototype — it deepened my love for the language itself.

Here’s why Rust feels right for me:

• Compiler as a collaborator, not a barrier

  The compiler’s strictness isn’t punishment — it’s mentorship. When it complains, it’s teaching you something real about ownership, lifetimes, or concurrency.

• Zero-cost confidence

  Every cargo build gives me production-grade guarantees. I don’t need a separate rewrite phase; prototypes become products.

• Performance without paranoia

  I can write readable, safe code and still hit C-level performance. It’s freedom without fear.

• Ecosystem that grows with you

  Whether I’m using tokio, reqwest, or DataFusion, the crates ecosystem feels mature and practical — not bloated.

• It rewards thoughtfulness

  Rust doesn’t make things easy; it makes them clear. Once you internalize the model, it feels like your brain and the compiler are working on the same problem.

That’s why I love Rust — it gives me flow and discipline at the same time.

What Changed After That Experiment

After adopting Corrode’s philosophy, my workflow shifted:

• I no longer switch to Javascript or Python for quick experiments.

• My “prototypes” evolve seamlessly into production code.

• My iteration loop became faster — because the compiler and I now trust each other.

Rust stopped being the language I feared to prototype in — it became the one I rely on to think clearly.

Closing Thoughts

If you’ve ever felt Rust was too heavy for quick experimentation, you owe yourself 10 minutes to read Corrode’s Prototyping in Rust.

It’s not just about writing Rust differently — it’s about thinking differently.

Prototype fast. Iterate safely. Refactor confidently.

And when you do, you’ll realize — Rust doesn’t slow you down; it slows down your mistakes.

🚨 The Problem: Bottlenecks in Data Analytics

For years, analytical engines have faced a fundamental challenge: how to process massive datasets in memory efficiently.

Traditional formats like CSV or even JSON:

• Are row-oriented (bad for analytics)

• Require parsing + decoding before processing

• Don't support vectorized execution or SIMD

Even columnar formats like Parquet are designed for storage, not runtime execution.

What we needed was a standard, fast, language-agnostic format for in-memory columnar data.

🚀 Enter Apache Arrow

Apache Arrow was born to solve this. It's a language-independent specification and implementation for:

• In-memory columnar data layout

• Zero-copy reads and writes

• Interoperability between systems and languages

• Support for modern CPU hardware (SIMD, caches)

Arrow is not a database, and not a query engine—it's the foundation those tools build on.

🧠 Core Concepts in Arrow

🧱 1. Columnar Format

Arrow stores data by column, not by row. This means:

• Better cache locality

• Vectorized execution (e.g., compute on whole columns at once)

• Efficient compression

🪵 2. RecordBatch

A RecordBatch in Arrow is like a table in memory:

• It contains a schema (field names and types)

• And a set of column arrays
  Each column is an ArrowArray, backed by contiguous memory buffers.

🧩 3. Buffers

Each column has:

• A data buffer (the actual values)

• A null bitmap buffer (to track missing values)

• An optional offset buffer (for variable-width types like strings)

📚 4. Language Bindings

Arrow is implemented in:

• C++

• Rust

• Python (via PyArrow)

• Go, Java, and more

This means a dataset generated in Rust can be read directly in Python or Go without copying or converting.

🔥 Real-World Use Cases

📊 Pandas + PyArrow

PyArrow allows Arrow arrays to be passed to/from Pandas and NumPy without copying, speeding up IO and interoperability.

🦆 DuckDB

DuckDB uses Arrow to interface with Python, R, and even web clients. When you call .arrow() on a DuckDB result, you get a zero-copy view.

⚙️ Apache DataFusion

DataFusion is a Rust-based SQL engine that processes RecordBatch Arrow data. Its entire physical execution plan is Arrow-native.

🐻‍❄️ Polars

Polars uses Arrow arrays under the hood for lightning-fast, multi-threaded, Rust-native DataFrame processing.

TL;DR: Use Parquet to store data, use Arrow to process it.

🧪 Example: Arrow in Rust

use arrow::array::{Int32Array, Array};
use arrow::record_batch::RecordBatch;
use arrow::datatypes::{DataType, Field, Schema};
use std::sync::Arc;

fn main() {
    let data = Int32Array::from(vec![Some(1), None, Some(3)]);
    let field = Field::new("numbers", DataType::Int32, true);
    let schema = Arc::new(Schema::new(vec![field]));
    let batch = RecordBatch::try_new(schema, vec![Arc::new(data)]).unwrap();

    println!("Rows: {}", batch.num_rows());
    println!("Columns: {}", batch.num_columns());
}

🧵 Summary

Apache Arrow is the backbone of modern data systems:

• Columnar + cache-efficient layout

• Language-agnostic zero-copy interoperability

• Powering analytical engines from Polars to DuckDB

It's not just a data format—it's a standard for high-performance analytics.

🔗 References

• Apache Arrow Official Site

• Arrow Rust Docs

• Apache DataFusion

• https://duckdb.org/2021/12/03/duck-arrow.html

  ✨ About the Author

  I'm Jagdish Parihar, a backend engineer passionate about high-performance systems, distributed databases, and query engines.

  I've contributed to Apache DataFusion, focusing on SQL engine internals like custom aggregate functions and optimizer rule enhancements. I'm also exploring Apache Arrow in Rust as part of building scalable analytical systems.

  You can find me here:

  • 🌐 LinkedIn

  • 🧑‍💻 GitHub

  • 📬 jatin6972@gmail.com

If you’re building something cool with Arrow, DataFusion, or Rust — let’s connect!

Parquet has quickly become one of the most popular file formats for storing large-scale analytics data. Parquet is now a top choice due to its efficiency, compression, and seamless integration with big data frameworks. My experience contributing to Apache DataFusion, a query engine that extensively uses Parquet, has deepened my understanding and appreciation of this format.

What is Parquet?

Parquet is an open-source, columnar storage file format optimized for large-scale data processing and analysis. Unlike traditional row-oriented formats like CSV or JSON, Parquet stores data column-wise, offering significant performance improvements for analytical queries.

Why Columnar Storage Matters

In row-oriented formats, accessing a single column requires scanning entire rows, including unnecessary data. Columnar storage like Parquet solves this by:

• Efficient Querying: Columns can be read independently, dramatically speeding up analytical queries.

• Better Compression: Columnar data tends to have similar values, making compression techniques like RLE and dictionary encoding highly effective.

• Reduced I/O: Less disk access as queries often target specific columns.

While contributing to DataFusion, I realized how crucial predicate pushdown and efficient column pruning are, especially for performance-critical queries.

Parquet File Structure

A Parquet file consists of:

• Row Groups: Logical partitions of data within a file, each containing column chunks.

• Column Chunks: Segments within row groups storing individual columns.

• Page Headers and Pages: Within column chunks, data is divided into pages containing actual values.

• Metadata: Contains schema information and statistics like min/max values that help query optimization.

Understanding Parquet’s metadata handling significantly improved my contributions to DataFusion’s query optimizer, particularly in filtering and skipping irrelevant data.

Advantages of Parquet

Performance

• Faster query execution.

• Supports predicate pushdown (skips irrelevant data based on query predicates), a critical aspect I optimized while working on DataFusion.

Compression Efficiency

• Built-in compression codecs like Snappy, GZIP, LZO, Brotli, and Zstd.

Schema Evolution

• Flexible schema management allows you to add or remove columns over time without breaking compatibility.

Integration

• Seamlessly integrates with frameworks like Apache Spark, Hadoop, AWS Athena, Apache Drill, Apache Impala, and Apache DataFusion.

Use Cases

Parquet is ideal for:

• Big Data analytics

• Data warehousing

• Machine learning pipelines

• Ad-hoc querying and BI tools

Conclusion

Parquet's columnar structure, efficient compression, and strong ecosystem support make it indispensable for modern data engineering. Contributing to Apache DataFusion has shown me firsthand the value of efficient column pruning, predicate pushdown, and metadata utilization, making Parquet an exceptional format for scalable and performant data workflows.

Happy querying!

My journey into query engines, databases, and Rust began with curiosity and a passion for systems-level performance. Rust's promise of safety and efficiency drew me in, while databases provided the perfect playground to test its capabilities. In this post, I'll share my path, with a particular focus on my contributions to Apache DataFusion, and I hope to inspire others to explore this exciting intersection.

Discovering Rust & DataFusion

I first encountered Rust while exploring modern languages offering memory safety without sacrificing performance. The zero-cost abstractions, excellent concurrency model, and powerful tooling like Cargo hooked me instantly.

Soon after, I discovered Apache DataFusion—a powerful, query engine written in Rust. Its integration with Apache Arrow, Parquet, and extensibility through custom SQL extensions and optimizer rules made it a compelling choice for deepening my skills.

Diving into the Internals

Apache DataFusion offered me insights into the inner workings of query planners and optimizers. Its architecture revolves around parsing SQL into logical plans, then optimizing and executing queries efficiently with Rust's concurrency and safety guarantees.

Rust proved essential, enabling DataFusion to leverage Arrow's columnar data structures and ensuring safe, concurrent query execution without traditional overheads.

My Contributions

Contributing to DataFusion gave me practical experience and deepened my understanding of query engines. Some of my notable contributions include:

• SQL Engine Enhancements: Implementing custom aggregate functions and fixing optimizer rules, focusing on correctness and extensibility.

• Performance Tuning: Optimizing SQL expressions and query execution paths for improved performance and reliability.

• Community Collaboration: Actively reviewing PRs, collaborating on feature discussions, and enhancing documentation for clearer guidance.

You can explore all my contributions here: Apache DataFusion PRs.

Lessons Learned

Technical Growth

• Gained a deep understanding of query optimization strategies and SQL parsing mechanisms.

• Learned Rust best practices around ownership, lifetimes, concurrency, and efficient memory management.

Community and Open Source

• Experienced firsthand the power of community-driven development through PR reviews and collaborative discussions.

• Improved skills in communication, debugging, and writing maintainable code.

The Bigger Picture

Apache DataFusion represents a broader trend toward modular, embeddable query engines, shaping the future of analytics and data-driven systems. Its ecosystem, connecting Arrow, Parquet, Spark, and other systems like Ballista and Lance, is continuously growing, offering exciting opportunities for developers.

What's Next

Moving forward, I aim to continue contributing to Apache DataFusion, focusing on deeper query optimization, performance benchmarks, and expanding the extensibility of its SQL APIs. I also encourage anyone interested in databases or Rust to explore and contribute—there’s a vibrant community ready to help.

Conclusion

My journey with Rust and Apache DataFusion has been rewarding, teaching me not only technical skills but also the value of community collaboration. I'm excited for what's ahead and hope my story inspires your own exploration into query engines, databases, and Rust.

Happy coding!

Feel free to connect and follow my ongoing journey:

• GitHub

• LinkedIn

• X

Apache DataFusion is a Rust-native query engine with a powerful optimizer. One key component of its optimizer is expression simplification, often referred to as the simplify function. In this post, we'll walk through how we implemented a simplify rule for the starts_with string function in DataFusion, turning it into an equivalent LIKE pattern. We'll cover a high-level overview of DataFusion's simplify mechanism, why this change was needed, how we implemented it step by step (including basic regex handling and tests), and the benefits it brings to both the DataFusion project and Rust developers.

Note: Portions of this article were drafted with the help of an AI writing assistant, then heavily edited by me for clarity and accuracy.

What is DataFusion's simplify Function?

In DataFusion, simplify is an optimization mechanism that rewrites expressions into simpler or more efficient forms during query planning. This includes things like constant folding (e.g. replacing 1 + 2 with 3) and algebraic rewrites. Each built-in function or expression can provide its own simplify logic. Thanks to DataFusion's well-structured design, these simplification rules are modular – typically implemented as a method on the function's definition – making it easy to add new rules without touching the rest of the codebase. For example, the starts_with function in DataFusion has a fn simplify(...) method where we can plug in custom rewrite logic​. The optimizer automatically applies these rules, so if a function's arguments allow a simplification, DataFusion will use the simpler form in the query plan.

Why Simplify starts_with?

The starts_with(string, prefix) function returns true if string begins with the given prefix. Without simplification, starts_with would be treated as a black-box scalar function in the query plan. By introducing a simplification rule, we can rewrite starts_with(col, 'prefix') into a standard SQL pattern match col LIKE 'prefix%'.

This has big benefits:

SQL engines (and DataFusion) know how to optimize LIKE 'prefix%' patterns for string columns. In particular, converting to LIKE enables predicate pruning based on prefix filters​. In other words, DataFusion can use file metadata (like min/max column values) to skip reading data that can't match the prefix, significantly improving performance for queries that filter on string prefixes.

Thanks to the PR that implements predicate pruning: https://github.com/apache/datafusion/pull/12978

Another benefit is that the logical plan becomes more transparent. A LIKE 'prefix%' is a recognizable operation for developers and other optimizations, whereas a custom function call might be harder to leverage. Overall, simplifying starts_with makes the query plan more efficient and easier to reason about.

Implementing the Simplification for starts_with

Thanks to DataFusion's modular architecture, adding this optimization was straightforward. All the logic for starts_with is encapsulated in its own module, so we only needed to modify that and add tests. Here are the steps we took to implement the simplify rule for starts_with:

1. Introducing the simplify logic: We added a new simplify method in the starts_with function implementation. This method inspects the function's arguments. If the second argument (the prefix) is a literal string, we can simplify. DataFusion provides an ExprSimplifyResult type to indicate whether an expression was simplified or left unchanged. In our case, when we detect a literal prefix, we'll return a Simplified result with a new expression.

2. Converting to a LIKE expression (basic regex handling): If the prefix is literal, we construct a new pattern for a SQL LIKE. Essentially, we take the prefix and append a % wildcard to match any suffix. For example, starts_with(name, "Ja") becomes name LIKE "Ja%". We also had to handle any special characters in the prefix. In SQL LIKE patterns, the % and _ characters have special meaning (any sequence of characters and any single character, respectively). We made sure to escape these in the prefix before appending % so that, for instance, a literal prefix "ja%" is treated as "ja\%" in the pattern (meaning the string "ja%") and then becomes "ja\%%" after adding the wildcard​. This way, the semantics of starts_with (which treats the prefix as literal text) are preserved in the LIKE expression.

The core snippet of the implementation looked like this:

    if let Expr::Literal(ScalarValue::Utf8(Some(prefix))) = &args[1] {
        // Found constant prefix, convert to LIKE pattern
        let escaped = prefix.replace("%", "\\%").replace("_", "\\_");
        let pattern = format!("{}%", escaped);
        let like_expr = Expr::Literal(ScalarValue::Utf8(Some(pattern)));
        return Ok(ExprSimplifyResult::Simplified(Expr::Like(Like {
            negated: false,
            expr: Box::new(args[0].clone()),    // the string column
            pattern: Box::new(like_expr),       // the 'prefix%' pattern
            escape_char: None,
            case_insensitive: false,
        }))));
    }
    // If not a literal prefix, no change
    return Ok(ExprSimplifyResult::Original(args));

In practice, we wrote similar logic for all relevant string types (Utf8, LargeUtf8, and the newer Utf8View) since DataFusion supports multiple string array types. The idea is the same: detect literal prefix, build the 'prefix%' pattern, wrap it in a LIKE expression node, and mark the expression as simplified.

3. Updating tests: With the implementation in place, we updated and added tests to ensure everything works as expected. We wrote unit tests for starts_with to confirm that the function still returns correct boolean results for various cases (e.g. "alphabet", "alph" -> true, "alphabet", "bet" -> false, etc.)​. We paid special attention to edge cases like an empty prefix (which should always yield true as long as the string is not NULL, since every string starts with ""). We also added an integration test using DataFusion's SQL engine to verify the optimization kicks in. For example, using an EXPLAIN query on a filter with starts_with, we should see the plan has been rewritten to use LIKE. After our change, an EXPLAIN SELECT * FROM my_table WHERE starts_with(col, 'f') showed a filter of col LIKE 'f%' in the logical plan, and the physical plan included a pruning predicate based on the prefix​. This confirmed that predicate pushdown was working: DataFusion was able to determine, for instance, that only values between "f" and "g" (non-inclusive of "g") could match f%, and use that to skip irrelevant data. All new tests passed, demonstrating that the simplify rule behaves correctly.

Why DataFusion's Design Made This Easy

Implementing this feature highlighted how well-structured DataFusion is. The codebase cleanly separates concerns: the logic for each function (including execution and optimization rules) lives in one place. We didn't have to tinker with the core optimizer loop or planner; we just implemented the simplify method for starts_with and the existing optimization framework took care of invoking it. DataFusion already had patterns for similar rewrites (for instance, simplifying certain regex patterns to equality checks in the past), so we were able to follow an established approach. This modular design meant less risk of breaking unrelated parts of the system and made the code review process smoother. It’s a testament to DataFusion’s extensibility that a contributor could add such an optimization in a relatively small, focused PR.

Benefits and Impact

This enhancement brings several benefits:

• Performance Boost: Queries filtering on string prefixes can see significant speed-ups. By converting starts_with(col, "prefix") into col LIKE 'prefix%', DataFusion can apply predicate pruning at the data source level​. In practical terms, if you're querying a large dataset stored in Parquet files, DataFusion will only read the files (or row groups) that could possibly contain values with the given prefix. This reduces I/O and speeds up query execution.

• Optimized Plans: The logical and physical plans become more optimized and easier to understand. Using a native LIKE operator in plans means other optimizations (like combining with other conditions) work seamlessly. It also aligns with SQL standards, which can be helpful for developers debugging query plans. The plan shows a clear intention ("column starts with X") via a LIKE 'X%' clause, which is intuitive.

• Encouraging Extensibility: For Rust developers and contributors, this is a great example of how to extend DataFusion. If you have a custom scalar function or see an opportunity for a rewrite rule, DataFusion provides the hooks (like the simplify trait method) to implement it. You can add specialized knowledge (such as domain-specific optimizations) in the same way we did for starts_with. The fact that we accomplished this with a small change localized to one module underscores how approachable DataFusion's codebase is for newcomers and seasoned devs alike.

• Maintaining Correctness: Importantly, all these optimizations maintain the exact semantics of the original expression. By escaping special characters and carefully constructing the new expression, we ensure that starts_with and the equivalent LIKE behave identically for all inputs. The robust test suite gives confidence in the correctness of this rewrite.

Conclusion

Implementing the simplify rule for starts_with was a satisfying journey that improved DataFusion's optimizer and demonstrated the strength of its design. In just a few lines of code, we unlocked an optimization that can benefit many real-world queries. This change makes DataFusion smarter about string predicates, allowing it to skip unnecessary work and respond faster. For DataFusion users, it means you can write queries with starts_with and get performance similar to writing a LIKE by hand – the engine does it for you. For Rust developers, it's an example of how contributing to an open-source project like DataFusion can be straightforward thanks to clear abstraction boundaries. We hope this encourages more contributions and explorations into DataFusion's optimization capabilities. After all, as we've seen, a simple change (pun intended) can go a long way in making your queries run simply faster!

What is Promise.race() method.

The Promise.race() the method returns a promise that fulfills or rejects as soon as one of the promises in an iterable fulfills or rejects, with the value or reason from that promise.

Implementation

So, we can provide two inputs inside the race method, i.e:

1. API call promise.

2. Delay promise with our custom timeout logic.

    Promise.race([apiCall, timeoutFunc])

Here, The apiCall method will represent our API call promise.

    // your url 
    const apiCall = fetch("https://api.github.com/");

and the timeout function will contain the delay logic which will be responsible for throwing a timeout error.

    function delay(timeOutPeriod) {
    return new Promise((_resolve, reject) => {
        setTimeout(() => reject("Request timeout"), timeOutPeriod);
    });
    }

    const timePeriod =  5000;
    const timeoutFunc = delay(timePeriod);

So, when we will run the Promise.race method, whichever promise is completed first, will return the output.

    Promise.race([apiCall, timeoutFunc])
    .then((data) => data.json())
    .then((data) => console.log("Data : ", data))
    .catch((error) => console.log("Error : "error));

Here, in this particular example, we have provided the custom timeout logic of 5000 milliseconds. So, either the API will give a response with 5 seconds or our network request will get timed out after the 5 seconds.

To check out the code, visit my github link github.

The Queue is the data structure that works on FIFO Principle, which is the abbreviation for "First In First Out".

Queues can have various applications:

Let's suppose you are making a service that requires you to send the email, but you don't want your program to wait for mailing to finish. Then you can send that particular task to the mailing queue and it will take control of sending all the mails in FIFO manner.

Now let's implement some functionality of the queue

The queue has some operations such as:

1. enqueue which will add the elements at the end of the queue.

2. Dequeue removes the element from the start of the queue.

3. Front To find out the element at the beginning of the queue.

4. Size function will give you the size of the queue.

5. Empty function will check if the queue is empty or not, and return us the boolean value.

6. Print function will print the whole queue for us.

function Queue() {
  let items = [];


  this.enqueue = function (element) {
    items.push(element);
  };

  this.dequeue = function () {
    return items.shift();
  };

  this.front = function () {
    return items[0];
  };

   this.size = function () {
    return items.length;
  };

  this.isEmpty = function () {
    return items.length === 0;
  };

    this.print = function () {
    console.log(items);
  };

}

Result :

const q = new Queue();

q.enqueue(10); // [10]      : enqueue from end
q.enqueue(20); // [ 10, 20] : enqueue from end
q.dequeue();   // [ 20]     : dequeue from start
q.print();     // prints the whole queue
console.log(q.size()); // return the size : i.e 1