All the times I've used Kafka, it's felt like the microservices confusion. A technical solution that can be taken to the extreme, when really it's a solution to a political problem.
If we break our application into small enough chunks then we start running into internal consistency issues, so we will introduce Kafka to resolve this.
I'm dealing with it at the moment where my customer decided to break their crud application to CUD and R because "best practice". Now they have Kafka synchronising the two applications because it's a terrible design.
I understand that it might help some people, and I am referring to my clients desire as terrible rather than the whole concept. However I have yet to actually see it used without it increasing complexity without demonstrable improvements. I see that Martin Fowler agrees.
It falls into the "you are not Google" problem solving hole.
I'm dealing with a financial institution doing this. CQRS should be an evolution if you find it complex to model, use, and scale not the initial implementation. Now we have another location to lose data.
If we break our application into small enough chunks then we start running into internal consistency issues, so we will introduce Kafka to resolve this. I'm dealing with it at the moment where my customer decided to break their crud application to CUD and R because "best practice". Now they have Kafka synchronising the two applications because it's a terrible design.