Decoupling Storage and Compute: The Magic of Tiered Storage

Our migration to Kafka was more than an API change; it fundamentally altered how we stored and accessed stream data. Storing billions of events on high performance EBS (Elastic Block Store) volumes would be costly. This made Kafka Tiered Storage our key architectural feature. Tiered Storage splits the storage of Kafka topic data into two tiers: TierDescriptionDetailsLocal Tier (Tier 1)Utilizes local EBS volumes on the brokers to store “hot” data. Configured for the last 5 minutes.

Ensures real time consumers read directly from fast, low latency storage. Consumers reading beyond this window are not latency sensitive by definition. Remote Tier (Tier 2)Asynchronously offloads closed log segments to a highly durable, low cost object store (Amazon S3). Stores historical data with a 7 day retention configured for our target. Reading from S3 adds a few hundred milliseconds of latency, which is acceptable if the consumer is already over 5 minutes behind.

By moving to Tiered Storage, we reduced infrastructure costs significantly. We stopped over provisioning costly EBS volumes for worst case retention. Historical data reads now come entirely from S3, preventing heavy backfills from starving IOPS for our hot, real time consumers.