Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MessageConsumerKafkaImpl should allow for concurrent processing of messages from different topics (even if they have the same partition number) #49

Open
douggish opened this issue May 22, 2021 · 0 comments

Comments

@douggish
Copy link

Currently, it looks like MessageConsumerKafkaImpl passes a record's partition number as the "swimlane" to a SwimlaneBasedDispatcher:

EventuateKafkaConsumerMessageHandler kcHandler = (record, callback) -> swimlaneBasedDispatcher.dispatch(new RawKafkaMessage(record.value()),
record.partition(),
message -> handle(message, callback, handler));

I believe this is to allow for concurrent processing of messages from different partitions while preserving ordering within a partition. This is desirable, and I think it works well if you are subscribed to a single topic.

However, it is possible to subscribe to multiple topics/channels, in which case it will dispatch messages with the same partition number to the same swimlane, even if they are from different topics. I don't think there are any ordering guarantees for messages in different topics (even if they have the same partition number), so MessageConsumerKafkaImpl should probably allow you to process messages from different topics concurrently. Otherwise messages that may be slow to process from (Topic A, Partition 1) will unnecessarily delay processing of messages from (Topic B, Partition 1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant