September 28, 2026
Day 7 — Kafka Was the Easy Part
Every day so far has been request/response — a client asks, something answers, synchronously, in the same call. Day 7 was the first genuinely different shape: a producer publishes an event and walks away, with no idea who's listening or when they'll read it. Redpanda (Kafka-API compatible, one Docker container instead of Kafka plus Zookeeper) made that easy to stand up. Actually running a producer and a consumer as two independent processes was where the day's real bugs turned out to live — and almost none of them were about Kafka.
Kafka is a class, not a client
const { Kafka } = require("kafkajs");
const producer = Kafka.producer();
Kafka.producer is not a function
Kafka is a constructor. producer/consumer live on its instances, not
on the import itself:
const kafka = new Kafka({ brokers: ["localhost:19092"] });
const producer = kafka.producer();
Once that clicked, the next omission was obvious in hindsight: no
brokers config existed anywhere, because there was no Kafka instance
to hold one.
A consumer needs to say which group it's in
KafkaJSError: Invalid consumer group configuration: groupId must be a valid string
kafka.consumer() with no arguments throws before it ever connects —
groupId is required, not optional. todo-consumer fixed it. This one
matters more than it looks: the group is also what determines whether a
restart replays history or resumes where it left off, which is exactly
what Part 4 of today's challenge turned out to hinge on.
producer.send() wants a value, not an object
await producer.send({
topic: TOPIC,
messages: [message.data], // { id, title, priority }
});
kafkajs's Message type is { value: Buffer | string | null, ... }.
Handing it the parsed todo directly gives it a message with no value at
all. The fix is one property, not a rewrite:
messages: [{ value: JSON.stringify(message.data) }],
Zod reused from Day 2's schema, same as every day since — priority
still needed the trim-before-min treatment from Day 6 to keep whitespace
titles from sneaking through.
Two processes, and the module system had opinions about both
The challenge is explicit that producer and consumer have to be separate terminals, not one script calling both functions in sequence — otherwise Part 4's decoupling test proves nothing. Getting each script to run at all on its own turned out to be its own small saga:
node producer.tswith no"type": "module"inpackage.json:SyntaxError: Cannot use import statement outside a module. Node defaults.tsto CommonJS, where top-levelimportisn't legal.- Node 22.2 (what I actually had installed) predates native TypeScript
support outright — that landed experimentally in 22.6, unflagged in
23.6+. So even fixing the module type wouldn't have been enough; there
was nothing to strip the type annotations.
tsxsidesteps the whole question — it works the same regardless of Node version. - Adding
await prod();/await cons();as top-level calls (so running the file actually does something) then hitTop-level await is currently not supported with the "cjs" output format—tsxwas still transpiling to CJS becausepackage.jsonhad no"type": "module". Adding that fixed it for real. - Along the way, installing
tsxfrom the wrong machine (a remote shell, not the Mac that would run it) pulled in@esbuild/aix-ppc64instead of@esbuild/darwin-arm64— esbuild ships a native binary per platform, andnode_modulesdoesn't travel between architectures.rm -rf node_modules package-lock.json && npm install, run on the machine that actually executes the code, fixed it.
None of this is a Kafka problem. All of it had to be solved before Kafka could even become the interesting part of the day.
Proving the decoupling — including the part I didn't plan
With both scripts finally running as separate processes, the core behavior showed up exactly as expected: start the consumer, run the producer, watch the message land in the other terminal.
The more interesting result was unplanned. After adding a second topic
(DEMO) to test isolation, and re-running the producer, the consumer —
still subscribed only to todos — logged the three todos messages
twice, and the DEMO message not at all:
{ topic: 'todos', partition: 0, value: { id: 1, title: 'Eat', priority: 'medium' } }
{ topic: 'todos', partition: 0, value: { id: 2, title: 'Sleep', priority: 'high' } }
{ topic: 'todos', partition: 0, value: { id: 3, title: 'repeat', priority: 'low' } }
{ topic: 'todos', partition: 0, value: { id: 1, title: 'Eat', priority: 'medium' } }
{ topic: 'todos', partition: 0, value: { id: 2, title: 'Sleep', priority: 'high' } }
{ topic: 'todos', partition: 0, value: { id: 3, title: 'repeat', priority: 'low' } }
The DEMO silence is topic isolation working correctly — a consumer only
ever sees the topic it subscribed to. The doubled todos messages are
fromBeginning: true doing exactly what it's documented to do, just not
what I expected: it only replays from the start when the consumer group
has no committed offset yet. My earlier consumer session had been
killed with Ctrl+C rather than shut down gracefully, so todo-consumer
had nothing durably committed — the next start replayed the entire
topic, including messages already delivered once. That's not a bug, it's
at-least-once delivery, live: a producer's send() resolving tells you
the broker has the message, and says nothing about whether, or how many
times, a consumer will actually process it.
What's next
Day 8 goes one level deeper into the mechanism that made today's replay
happen: consumer groups and partitioning — what actually happens when two
consumers share one group. Full code in the
repo — day-07-kafka-fundamentals/.