Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The article doesn't mention which EBS volume type was used, but since Provisioned IOPS are mentioned, I assume it's gp3 or io2. One pattern that is especially often used in Time Series databases, but could work for Kafka too, is not tiering down to S3, but changing older volumes to a slower volume type, such as sc1 ($0.015/GiB-Mo). This can be done completely transparently to the application.

Another thing worth looking into is S3 Mountpoint with or without read caching, which offers a POSIX-like interface for S3 to applications that don't natively support S3.



This strategy will not work well for Apache Kafka because it is extremely IOPS hungry if you have more than a few partitions, and a replay of a large topic will require lots of IO bandwidth. It would work well e.g. a columnar database where a query targeting old data may only require reading a small fraction of the size of the volume, but Kafka is effectively a row-oriented storage system, so the IO pattern is different.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: