Unfinished article

Queue, like any other queue in a world
So, Kafka Topic is a queue… Kinda queue… More like distributed set of queues.
Distributed set of queues with routing and some consistency guarantees.

Partitioning

In Kafka terms, partition is separate queue. Then you have Topic, which is a group of partitions.

That predefined number of partitions is spread across nodes.

5 Kafka partitions across 3 nodes
5 partitions across 3 nodes
Real number of partitions is usually way higher, to allow even load balancing.

How do we know the partition for each specific message?

Queue, like any other queue in a world
Each message is sent to random (or specificly selected) partition.

Log Segments and Offsets

Each partition is stored as append-only immutable log file.
New messages are appended, replicated, and read by offsets. Old log files are deleted later.

Queue, like any other queue in a world
Offsets:

  1. Current position, where new messages are written;
  2. “HighWatermark” position that was replicated (if needed);
  3. Read position

Here your custom software received, process, and commits messages

  1. Commit position

old log file is deleted after retention window

Once again, Kafka creates log files for each partition separately, so

Queue, like any other queue in a world

Message Order

Understanding internals, one may ask how does Kafka keeps topic message ordering?

Queue, like any other queue in a world
It does not. Messages are ordered ONLY within a partition. If you need them ordered, make sure they appear in same partition. You can set it manually, but it’s easier to use message key.

partition = hash(key) % len(patitions)

Usually user_id, transaction_id, content_id or suchlike fields are set as message key.

Consistency

Usually you have to choose between duplicated messages (at least once) or lost messages (at most once)

There is no magic, you either loose message or get it duplicated
There is no magic, you either loose message or get it duplicated

Kafka claims to support “Exactly once” mode, via two phase commit, but that’s a separate topic of discussion.