So, Kafka Topic is a queue… Kinda queue… More like distributed set of queues.
Distributed set of queues with routing and some consistency guarantees.
In Kafka terms, partition is separate queue. Then you have Topic, which is a group of partitions.
That predefined number of partitions is spread across nodes. Real number of partitions is usually way higher, to allow even load balancing.
How do we know the partition for each specific message?
Each message is sent to random (or specificly selected) partition.
Log Segments and Offsets
Each partition is stored as append-only immutable log file.
New messages are appended, replicated, and read by offsets. Old log files are deleted later.
- Current position, where new messages are written;
- “HighWatermark” position that was replicated (if needed);
- Read position
Here your custom software received, process, and commits messages
- Commit position
old log file is deleted after retention window
Once again, Kafka creates log files for each partition separately, so
Understanding internals, one may ask how does Kafka keeps topic message ordering?
It does not. Messages are ordered ONLY within a partition. If you need them ordered, make sure they appear in same partition. You can set it manually, but it’s easier to use message key.
partition = hash(key) % len(patitions)
content_id or suchlike fields are set as message key.
Usually you have to choose between duplicated messages (at least once) or lost messages (at most once)
Kafka claims to support “Exactly once” mode, via two phase commit, but that’s a separate topic of discussion.