Kafka to iceberg ingestion using dynamic iceberg sink blog post

[swapna267]

This PR contains a technical blog post titled "From Stream to Lakehouse: Kafka Ingestion with the Flink Dynamic Iceberg Sink".

The article addresses a common pain point for data engineers: managing complex and brittle ingestion pipelines for thousands of evolving Kafka topics. It introduces the Flink Dynamic Iceberg Sink as a solution that enables a self-adapting, zero-downtime ingestion layer.

The post walks the reader through static pipelines using iceberg sink and then provides a detailed guide on building the same using dynamic iceberg sink. It focuses on a practical use case involving Kafka, Avro, and Confluent Schema Registry, and includes code examples to illustrate the key components.

Key Topics Covered:

• Building generic Kafka to iceberg ingestion pipeline and scaling to thousands of topics in single pipeline.
• The challenges of static Kafka-to-Iceberg pipelines (schema evolution, new topics).
• An architectural overview of the Dynamic Iceberg Sink pattern.
• A step-by-step implementation guide, including:
  • Preserving metadata with a KafkaRecord wrapper.
  • Using a DynamicRecordGenerator for late binding of schema and table information.
  • Assembling the final Flink job.
• Details on the feature's availability in Apache Iceberg and supported Flink versions.
• Credits to the original author of the proposal and key contributors.

Related Links

• This work is based on the original proposal: Flink Dynamic Iceberg Sink Proposal

[mbalassi]

Thanks. @mxm, PTAL a look and let me know if you would like to suggest any improvements.