How Flink’s Kafka connector commits offsets to Kafka
Summary
A detailed technical dive into how Flink's Kafka connector commits offsets, exploring the unit of work (splits), checkpoints, and how transaction markers affect offset visibility. The post recounts a PR merge, multiple fixes to reconcile committed offsets with Kafka's view, and edge cases such as partitions with no records or transactions in progress, concluding with the claim that committed offsets are a best-effort side channel without breaking Flink's processing guarantees.