Skip to content

Kafka topic of type “scratch”

A Kafka topic of type “scratch” is a mechanism to store a sequence of messages:

  • If you create a Kafka topic on the DSH, then it’s a private topic: only your tenant’s applications or services can see and access it. On the DSH, private topics are also called scratch topics.
  • Under the hood, Kafka topics consist of files stored on disk.
  • If your application or service writes a message to a topic, then Kafka attaches it to the end of the topic. In other words, you can’t change the order of the messages in a topic.
  • If your application or service reads from a topic, then Kafka runs through the messages one by one, starting at an offset.
  • On the DSH, you can configure different aspects of Kafka topics of type “scratch”: cleanup policy, partitions, data replication, and rates for writing or reading messages.
  • See DSH streams if you want to share data with other tenants, or outside the DSH.
  • See Kafka Proxy if you want to share Kafka messages with external Kafka clients.

Note

Every DSH platform contains 1 Kafka cluster. The DSH stores the Kafka topics of all tenants on this Kafka cluster, and uses Access Control Lists (ACLs) to manage the access to these Kafka topics.

Kafka concepts

In order to create and configure a Kafka topic of type “scratch”, you need to understand some of the basic concepts in Kafka.

  • Message: The record of an event, such as an order, payment, activity, or measurement.
  • Topic: A log of events, in the form of messages.
  • Producer: A process that writes messages to a topic.
  • Consumer: A process that reads messages from a topic.
  • Kafka broker: Kafka stores topics on multiple servers. Such an individual server is called a Kafka broker, and brokers are organized into clusters. Every DSH platform has one Kafka cluster.
  • Partition: Kafka can divide a topic into multiple partitions, which then allow to scale easily and securely:
    • Kafka can store partitions on different brokers within one cluster, which distributes the workload.
    • If you use partitions on multiple brokers, then multiple instances of your application or service can read from the topic in parallel.
    • Kafka replicates a partition that exists on one single broker to other brokers in the same cluster. If the original broker fails, then replicated partitions are still available on other brokers.
  • Segment: Kafka doesn’t store partitions in one big file on disk, but it divides them into smaller files, called segments.
  • Message key: A unique identifier of a message that Kafka uses for log compaction.
  • Log compaction: A cleanup mechanism where Kafka removes any old messages from a partition if a newer message with the same key is present in the partition. On the DSH, log compaction takes place when a segment has reached its maximum size.

Limits

The DSH imposes the following limits on Kafka topics of type “scratch”:

  • Topic count: The maximum number of private topics in your tenant. This limit doesn’t count the topics that are part of a DSH stream.
  • Partitions: The maximum number of partitions across all your private topics. If you have 3 private topics with 3 partitions each, then your tenant contains 9 partitions.
  • Consumer rate: The maximum rate at which tenants can read data from Kafka, in mebibyte per second (MiB/s).
  • Producer rate: The maximum rate at which a tenant can write data to Kafka, in mebibyte per second (MiB/s).
  • Request rate: The maximum CPU time, in milliseconds (ms), across all of the I/O and network threads on the platform that a broker is allowed to use for handling requests:
    • If the request rate is 1 %, then one broker can use 10 ms of CPU time.
    • If the request rate is 10 %, then one broker can use 100 ms of CPU time.
    • If the broker exceeds this time, then the broker adds a delay before responding to bring the usage within the configured rate.
    • See Quotas in the Kafka documentation for more information.

Note

The consumer rate, producer rate and request rate also apply to DSH streams. See DSH stream for more information.

You can request new limits:

  • Click “Resources” > “Overview” in the menu bar of the DSH Console.
  • Under “Kafka”, click the “Request” button for the limit that you want to change.
  • Fill out the desired value and click “Request”. A platform administrator will process your request, which appears as a ticket in the Support Portal.

Managing Kafka topics

You can manage your Kafka topics of type “scratch” in the DSH Console.

Tip

You can also use endpoints in the Tenant Management API to manage Kafka topics of type “scratch”.

Adding a topic

Take the following steps to add a new private topic:

  1. Click “Resources” > “Topics” in the menu bar of the DSH Console.
  2. Click the “+ Topic” button to add a new topic. Fill out the following fields:
    • Topic name: Enter a name that is unique in your tenant.
    • Number of partitions: Enter the number of partitions that you want for your topic.
    • Replication factor: This indicates how many times Kafka replicates partitions across brokers. The DSH always uses a replication factor of “3”, which means that Kafka always replicates a partition on a specific broker to 2 other brokers in the same cluster.
    • Cleanup policy:
      • 1-day retention: Select this option if you want the DSH to delete old messages after one day. This works well for temporary event data, such as logging, where each message stands alone. This is the default value for the cleanup policy.
      • Log compaction: Select this option if you want to use a cleanup mechanism where Kafka removes any old messages from a partition if a newer message with the same key is present in the partition. The DSH executes the log compaction per partition, once a segment has reached its maximum size. This option guarantees that a partition always has the latest state for each unique message key.
    • Timestamp attribute in the message:
      • Producer timestamps: Select this option if the timestamp attribute of the message in the Kafka topic must contain the time when the producer creates the message. This is the default value.
      • Broker timestamps: Select this option if the timestamp attribute of the message in the Kafka topic must contain the time when the broker inserts the message into the topic.
    • Max message size: The maximum size of messages in the topic, in bytes. If you don’t enter a value, then the DSH uses the default value of 1048588 bytes, which corresponds to 1 mebibyte.
    • Segment size: The size of the segments in a partition. The segment size determines when Kafka executes log compaction as a cleanup policy. If you don’t enter a value, then the DSH uses the default value of 1073741824 bytes, which corresponds to 1 gibibyte.
  3. Click the “Create topic” button. The DSH will create your topic and it will appear in the list of Kafka topics.

Note

The DSH creates a namespace for your topic to make it unique inside your platform, and it uses the format scratch.<topic-name>.<tenant>. For example:

  • ‘tenant-a’ is the name of your tenant.
  • ‘temperature’ is the name of your topic.
  • ‘scratch.temperature.tenant-a’ is the namespace that the DSH creates for your Kafka topic of type “scratch”.

Inspecting a topic

You can inspect a private topic in two ways, via the DSH Console, or via the Kafdrop application.

Via the DSH Console

Take the following steps to inspect a topic’s properties in the DSH Console:

  1. Click “Resources” > “Topics” in the menu bar of the DSH Console.
  2. Click the “Properties & Metrics” button for the topic that you want to inspect.

  3. The DSH displays the values for the following properties:

    • Number of partitions
    • Replication factor
    • Cleanup policy
    • Message timestamps
    • Max message size
    • Segment size

    See Adding a topic to find out what these properties mean.

Via Kafdrop

In Kafdrop, you can inspect the messages in a topic. See App Catalog for more information.

Once you installed Kafdrop, you take the following steps to inspect a topic’s messages:

  1. Navigate to the vhost for your tenant’s Kafdrop.
  2. In the “Name” column of the table under “Topics”, click the topic that you want to inspect.
  3. Click “View messages” or “Search messages”, depending on what you want to do.

Note

You can only use Kafdrop to inspect your topic. It also displays buttons to create and delete topics, but these will return errors. Use the DSH Console to create and delete topics.

Deleting a topic

Take the following steps to delete a private topic:

  1. Click “Resources” > “Topics” in the menu bar of the DSH Console.
  2. Click the “Delete” button for the topic that you want to delete, and “Yes, delete” to confirm.