Step 3: Connect Your Kafka Cluster/s
This page describes how to connect Superstream to one or more Kafka clusters
Last updated
This page describes how to connect Superstream to one or more Kafka clusters
Last updated
Product
Superstream ConsoleCopyright to Superstream.ai
If this is your first login, please use the link included in your welcome email.
In the upper-right corner, you will find this button:
Please do not close the window until a message is shown.
Every Engine is capable of establishing connections with one or several Kafka clusters simultaneously. To determine the optimal strategy for your setup, it is advisable to seek guidance from the Superstream team.
Kafka clusters should be defined in the config.yaml
file:
name
: A unique name for the connection to be displayed through Superstream GUI.
type
: The type of Kafka. Options include confluent_kafka
, confluent_cloud_kafka
, MSK
, and apache_kafka
.
hosts
: A list of bootstrap servers for the Kafka cluster.
Authentication:
noAuthentication
: Set enabled
to true
if no authentication is required.
ssl
: For SSL encryption without SASL authentication.
enabled
: Set to true
to enable SSL authentication.
protocol
: Should always be SSL
.
ca
, cert
, key
: Paths to your CA certificate, client certificate, and client key files.
insecureSkipVerify
: Set to true
to bypass server certificate verification (not recommended for production environments).
sasl
: For SASL authentication.
enabled
: Set to true
enable SASL authentication.
mechanism
: The SASL mechanism to use. Options: PLAIN
, SCRAM-SHA-512
.
protocol
: Should be SASL_SSL
for encrypted connections.
username
and password
: Credentials for SASL authentication.
tls
: Optional TLS configuration for SASL authentication.
If TLS is used with SASL, specify the following:
enabled
: Set to true
to enable TLS.
ca
, cert
, key
: Paths to your CA certificate, client certificate, and client key files, if required.
insecureSkipVerify
: Set to true
to bypass server certificate verification (not recommended for production environments).
Below is an example configuration for a SASL_SSL authenticated connection:
Replace placeholders (e.g., <connection_name>
, <bootstrap_server_1>
) with your actual Kafka connection details. As shown in the examples, Follow the correct indentation and formatting.
For any questions or further assistance, please refer to the official Kafka documentation or reach out to your Kafka provider.
You can add an API key to gain deeper insights for eligible Kafka vendors such as Confluent, Aiven, Redpanda, and AWS.
Here's how to do it if you didn't set it up during the initial client connection establishment:
Step 1: Head over to Settings -> "Keys"
Step 2: Fill in the form with your key information
Head over to your Confluent Console -> Right-side Menu -> API Keys
On the right-top corner, click on "+ Add API key"
Choose "My Account"
The scope should be "Cloud resource management"
Give the new key a name of your choice
Paste the generated credentials in the Superstream Console