AWS Lambda adds queue consumption mode for Apache Kafka

AWS announced on November 2 that Lambda can now consume compatible Apache Kafka topics in queue mode. The integration requires Kafka 4.2 or later with share groups enabled and uses the self-managed Kafka event source mapping, including compatible third-party managed offerings.

The new option allows multiple consumers to process records from the same partition. It provides no ordering guarantee, and duplicate processing can occur. Existing integrations keep stream mode by default. AWS says availability covers Regions that support Lambda event source mappings for self-managed Kafka.

JetherVerse perspective: the useful question is whether a business process needs events completed in sequence. Consider a hypothetical media platform creating a thumbnail for each uploaded photograph. Those jobs might be independent. A different workflow updating the same customer's account balance could need a much more careful approach to ordering. The right choice starts with the meaning of the work, rather than a headline about faster processing.

Before adopting a different processing model, ask the team to write down what a repeated task would do. Would it send a second message, charge twice, create another file, or simply replace an existing result? Give each business action a clear identifier and agree how the application should recognise work it has already completed. Test that behaviour with deliberate retries in a development environment.

A useful pilot should also include slow tasks, failed tasks and a temporary interruption in a downstream service. Measure the time from an event arriving to a usable business result, alongside operating cost and the effort needed to investigate failures. Higher activity is not automatically a better customer experience.

Finally, make the decision reversible. Keep a record of the previous configuration, assign someone to watch the pilot and define the conditions for stopping it. For business owners, the practical conversation is about reliable outcomes under pressure, not simply increasing the number of tasks running at once.

Image: AWS Lambda product illustration, courtesy of AWS.