Kafka : Generating unique IDs for strings across partitions
Kafka : Generating unique IDs for strings across partitions I'm trying to asses if Kafka could be used to scale-out our current solution. I can identify partitions easily. Currently, the requirement is there to be 1500 partitions, each having 1-2 events per second, but future might go as high as 10000 partitions. But there is one part of our solution which I don't know how would be solved in Kafka. The problem is that each message contains a string and I want to assign unique ID to each string across whole topic. So same strings have same ID while different strings have different IDs. The IDs don't need to be sequential, nor do they need to be always-growing. The IDs will then be used down-stream as unique keys to identify those strings. The strings can be hundreds of characters long, so I don't think they would make efficient keys. More advanced usage would be where messages might have different "kinds" of strings, so there would be multiple unique sequences ...