Step 3: Connect Your Kafka Cluster/s

This page describes how to connect Superstream to one or more Kafka clusters

Option 1: Using Superstream Console

Step 1: Login to SSM Console

If this is your first login, please use the link included in your welcome email.

Step 2: Add a new connection

In the upper-right corner, you will find this button:

Step 3: Fill in the details

Please do not close the window until a message is shown.

Option 2: During Superstream engine deployment

Every Engine is capable of establishing connections with one or several Kafka clusters simultaneously. To determine the optimal strategy for your setup, it is advisable to seek guidance from the Superstream team.

Kafka clusters should be defined in the config.yaml file:

connections:
  - name: <connection_name>
    type: <connection_type> # available types : confluent_kafka, confluent_cloud_kafka, MSK, apache_kafka
    hosts: # list of bootstrap servers
      - <bootstrap_server_1>
      - <bootstrap_server_2>
    authentication: # Specify one preferred method for connection
      method:
        noAuthentication:
          enabled: <true_or_false>
        ssl:
          enabled: <true_or_false>
          protocol: SSL
          ca: "<path_to_ca_cert>"
          cert: "<path_to_cert>"
          key: "<path_to_key>"
          insecureSkipVerify: <true_or_false>
        sasl:
          enabled: <true_or_false>
          mechanism: <sasl_mechanism> # available options: "PLAIN", "SCRAM-SHA-512"
          protocol: SASL_SSL
          username: "<username>"
          password: "<password>"
          tls:
            enabled: <true_or_false>
            ca: "<path_to_ca_cert>"
            cert: "<path_to_cert>"
            key: "<path_to_key>"
            insecureSkipVerify: <true_or_false>
  • name: A unique name for the connection to be displayed through Superstream GUI.

  • type: The type of Kafka. Options include confluent_kafka, confluent_cloud_kafka, MSK, and apache_kafka.

  • hosts: A list of bootstrap servers for the Kafka cluster.

Authentication:

  • noAuthentication: Set enabled to true if no authentication is required.

  • ssl: For SSL encryption without SASL authentication.

    • enabled: Set to true to enable SSL authentication.

    • protocol: Should always be SSL.

    • ca, cert, key: Paths to your CA certificate, client certificate, and client key files.

    • insecureSkipVerify: Set to true to bypass server certificate verification (not recommended for production environments).

  • sasl: For SASL authentication.

    • enabled: Set to true enable SASL authentication.

    • mechanism: The SASL mechanism to use. Options: PLAIN, SCRAM-SHA-512.

    • protocol: Should be SASL_SSL for encrypted connections.

    • username and password: Credentials for SASL authentication.

    • tls: Optional TLS configuration for SASL authentication.

If TLS is used with SASL, specify the following:

  • enabled: Set to true to enable TLS.

  • ca, cert, key: Paths to your CA certificate, client certificate, and client key files, if required.

  • insecureSkipVerify: Set to true to bypass server certificate verification (not recommended for production environments).

Example

Below is an example configuration for a SASL_SSL authenticated connection:

connections:
  - name: my_kafka_connection
    type: confluent_cloud_kafka
    hosts:
      - kafka.example.com:9092
    authentication:
      method:
        sasl:
          enabled: true
          mechanism: PLAIN
          protocol: SASL_SSL
          username: "myUsername"
          password: "myPassword"
          tls:
            enabled: false

Replace placeholders (e.g., <connection_name>, <bootstrap_server_1>) with your actual Kafka connection details. As shown in the examples, Follow the correct indentation and formatting.

For any questions or further assistance, please refer to the official Kafka documentation or reach out to your Kafka provider.

Add a vendor API key to gain deeper insights.

For eligible Kafka vendors such as Confluent, Aiven, Redpanda, and AWS, you can add an API key to have deeper insights like: the amount of potential savings in dollars.

Here's how to do it if you didn't set it up during the initial client connection establishment:

Step 1: Head over to the "Keys" section

Step 2: Fill in the below form with your key information

Step 3: Attach the key to one or more Kafka clusters by selecting them in the "Connections" select box

  1. Head over to your Confluent Console -> API Keys

  2. On the right-top corner, click on "+ Add API key"

    1. Choose "My account"

    2. The scope should be "Cloud resource management"

    3. Give the new key a name of your choice

  3. Paste the generated creds in the Superstream Console

Last updated

Logo

Copyright to Superstream.ai