Kafka consumer groups¶

The DSH supports the use of Kafka consumer groups, which allow Kafka consumers to work together and process messages from a Kafka topic in parallel:

Use shared consumer groups to share a group across multiple instances of the same service.
Use private consumer groups to assign the group to one single instance only.
The DSH generates the names of the consumer groups for you.
You can track the consumer group lag in the DSH Console, on the details page of your application or service.

Concepts¶

In order to configure Kafka consumer groups, you need to understand some of the basic concepts:

Consumer group: If a Kafka topic has multiple partitions, then consumers can process its messages in parallel. Using consumer groups, consumers can work together, and coordinate the partition assignment across the parallel processors. The DSH uses two types of consumer groups:
1. Private consumer group: This consumer group can be used by one single instance only.
2. Shared consumer group: This consumer group can be shared by multiple instances.
Consumer group name: Each consumer group has a unique name. All consumers that use the same consumer group name, are members of the same consumer group.
Consumer group offset commits: When a service or application consumes messages from Kafka as a member of a consumer group, it regularly records its progress. This is known as offset committing. When the workload is redistributed, other members of the consumer group can use the offset commit to know where to start processing messages from the partition.
Consumer group lag: This is the difference between the offset committed by the consumer group, and the offset of the latest published message in the topic partition. It indicates how far behind the consumer group is in processing the messages in that topic partition. The consumer group lag is an important metric of the health of your service.

Consumer group names¶

On the DSH, the name of a consumer groups typically follow the pattern <tenant>_<service-or-application>_<index>:

‘tenant-a_traffic-analyzer_1’: Consumer group ‘1’ for the service ‘Traffic Analyer’ of ‘Tenant-a’.
‘tenant-b_vehicle-tracker_4’: Consumer group ‘4’ for the service ‘Vehicle tracker’ of ‘Tenant-b’.

The DSH creates the names of consumer groups for you. You can find the list of consumer groups by running the get_signed_certificate.sh script in your service on the DSH. See Retrieve the Kafka configuration or Retrieving consumer group names for more information.

Tip

It’s recommended to use the DSH’s consumer group names only.
If you use ACL groups in the Kafka Proxy, then the format of the consumer group names changes. See Naming conventions for more information.

Retrieving consumer group names¶

The DSH creates the consumer group names for you. You can retrieve them by running the get_signed_certificate.sh script in your service:

See Retrieve the Kafka configuration to learn how you can run the get_signed_certificate.sh script.
This script generates the datastreams.json file.
In this file, the private_consumer_groups and shared_consumer_groups keys contain arrays with the names of your private consumer groups and shared consumer groups respectively.

Use only these consumer group names for your applications and services.

Consumer group lag¶

The consumer group lag is a metric of the health of your application or service. It’s the difference between the offset committed by the consumer group, and the offset of the latest published message in the topic partition. Take the following steps to inspect the consumer group lag of your service or application in the DSH Console:

Click “Services” > “Overview” in the menu bar of the DSH Console.
In the list of services and applications, click the one that you want to inspect.
The details page of the service displays its running tasks, and the aggregated consumer group lag for all of its tasks.
Click an individual task to navigate to its details page and inspect its consumer group lag.

See Managing services and applications for more information.