Synthèse
MSK scaling operations are not instantaneous, and they are not symmetric. Increasing broker storage gives the cluster more breathing room, but that disk allocation cannot be reduced later. Once storage has been expanded, the only practical way to return to a smaller disk footprint is to create a new cluster with the desired size and re route producers and consumers to it. Broker instance upgrades also require a rolling restart. In practice, this can take roughly 15 minutes per broker, so rolling
MSK scaling operations are not instantaneous, and they are not symmetric. Increasing broker storage gives the cluster more breathing room, but that disk allocation cannot be reduced later. Once storage has been expanded, the only practical way to return to a smaller disk footprint is to create a new cluster with the desired size and re route producers and consumers to it. Broker instance upgrades also require a rolling restart. In practice, this can take roughly 15 minutes per broker, so rolling instance upgrades across a large cluster can take hours, making them unsuitable as incident mitigation.
That is useful for planned scaling, but it is rarely fast enough to be the primary mitigation during an active incident. Scaling out by adding brokers has a similar limitation. New brokers increase theoretical cluster capacity, but they do not immediately help existing hot topics. Until partitions and leaders are rebalanced onto the new brokers, the overloaded brokers continue carrying the same traffic. Because rebalancing adds additional network, disk, and replication load, scaling out during an incident was not a realistic recovery option for our most severe failure modes.