Retrieve Kafka configuration

If you add a custom service to your tenant in the DSH, it’s isolated inside your tenant:

• By default, it can connect to other services inside your tenant only.
• You can add vhosts to allow connections from outside your tenant, namely with other tenants or the public Internet.

However, every platform contains one Kafka cluster that is shared among all tenants on that platform:

• Your service needs to set up an SSL connection with the Kafka cluster if it wants to interact with it. The DSH contains a Public Key Infrastructure (PKI) that handles the certificates and keys for these SSL connections.
• Additionally, your service needs the address of the Kafka servers, and it’s recommended to assign a Kafka client ID to your service.

It’s no easy feat to set up the SSL connection correctly, and to retrieve the correct information to connect to the Kafka cluster. The DSH offers 2 ways to inspect the Kafka configuration:

• A bash script that you can run inside your container
• The get_signed_certificate.sh script in the Cmd Line app

Via bash script in your container

The DSH team created a bash script that does the following:

• It retrieves the environment variables, such as tenant name, DNS name, task ID of your service.
• It retrieves the server certificate for the Kafka cluster from the DSH’s PKI, and stores them.
• It generates a client certificate, public key and private key for your service, and requests that the DSH’s PKI verifies and signs the client certificate.
• It stores the following information as environment variables, so that your custom service can retrieve them:
  • The location of the certificates and your service’s keys
  • The tenant name and the DNS name
  • The Kafka configuration for your service, in JSON

See Script to configure SSL for Kafka for the bash script, and one of the following examples to see how to use it:

• Publish to Kafka topic of type “scratch”
• Consume from Kafka topic of type “scratch”
• Publish to public DSH stream
• Consume from public DSH stream

Via Cmd Line app

It may be helpful to inspect the Kafka configuration before you deploy a custom service on the DSH. You can do this via the Cmd Line app from the DSH’s App Catalog:

• First, deploy the Cmd Line app if you haven’t done so already. If you already deployed Cmd Line, then you can skip the following steps:
  • In the menu bar of the DSH Console, click “Services” > “App Catalog”.
  • Click the title of the “Cmd Line” app, and then the “Configure & Deploy” button.
  • Fill out the configuration form correctly, and click the “Deploy” button.
• Once the DSH deployed your Cmd Line app, you can open it:
  • Click “Services” > “Overview”, and then click the name that you chose for your Cmd Line app.
  • On the details page of your Cmd Line app, click the icon next to the link under “Services & resources” to navigate to your Cmd Line app.
  • Log in if necessary.
• Now, you can inspect the Kafka configuration:
  • Enter the command get_signed_certificate.sh. This script will create the necessary certificates.
  • Enter the command cat ~/datastreams.json | jq . to see the list of brokers, streams and consumer groups.
  • Enter the command printenv to see the list of environment variables of the Cmd Line app. This gives you an idea of the environment variables that are typically available on the DSH, even though they will be different for your custom services.
  • Check the documentation at the top of the Cmd Line app to discover other commands.