Step 1: Prerequisites

Prerequisites before deploying a Superstream engine

1. A user for Superstream per Kafka cluster

The Superstream controller is compatible with all Kafka authentication methods, such as:

• SASL/SCRAM

• SASL/GSSAPI

• SASL/PLAIN

• ACL

• RBAC

For effective functioning, the user or token utilized should have permissions for:

• Read: All topics

• Describe

• DescribeConfigs

• Read, Create, and Write: single topic named superstream.metadata (A dedicated Superstream topic with infinite retention and a single partition).

• Admin API: Describe Topics, List Topics, Describe Configs, Describe Groups, List Consumer Groups, Describe Cluster

Apache Kafka

Example ACL statements:

ACL statement that grants read access to a user named Superstream for all topics in the Kafka cluster:

kafka-acls --bootstrap-server : -add --allow-principal User:Superstream --operation read --topic '' --group '' --command-config <PATH_TO_CRED_FILE>

ACL statement that grants describe access to a user named Superstream for all topics in the Kafka cluster:

kafka-acls --bootstrap-server : -add --allow-principal User:Superstream --operation Describe --topic '*' --command-config <PATH_TO_CRED_FILE>

ACL statement that grants DescribeConfigs access to a user named Superstream for all topics in the Kafka cluster:

kafka-acls --bootstrap-server <URL>:<PORT>  -add --allow-principal User:Superstream --operation DescribeConfigs --topic '*' --command-config <PATH_TO_CRED_FILE>

Confluent

Superstream requires a service account with the following configuration:

For each designated environment (Environment level):

• MetricsViewer

• DataDiscovery

• Operator

For each designated cluster (Cluster level):

• CloudClusterAdmin

The Superstream local engine is designed to analyze a limited subset of events and primarily metadata, sufficient to understand each topic and client's structure, size, and workload. This approach enables the controller to offer detailed monitoring, optimizations, and recommendations based on its analysis.

2. Network

Ports: The Superstream engine should be able to communicate with each designated Kafka cluster through the following ports:

• Port 9092 is used for data communication between the Superstream controller and the designated Kafka cluster.

• Port 9999 facilitates JMX and monitoring communication between the Superstream controller and the designated Kafka cluster.

• Port 6666 to enable secure communication for metadata transfer between the on-prem Superstream data plane to the external Superstream control plane

3. Engine deployment-related

1. Kubernetes Cluster: You need an up-and-running Kubernetes cluster. If you don't have one, you can create a cluster on platforms like Google Kubernetes Engine (GKE), Amazon EKS, Azure AKS, or Minikube for local development.

2. kubectl: The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. Install kubectl if you haven't already.

3. Helm: Helm is a package manager for Kubernetes that facilitates the deployment and management of applications. Install Helm if it's not already set up.

4. A defined default storage class. In case you can't define one, please use this appendix

5. Account ID and Activation Token. To be received by the Superstream team.

6. Fill in the "Environment readiness" checklist

K8S resources:

• CPU: A minimum of 4 CPUs.

• RAM: A minimum of 8 GB.

Storage Requirements:

Default Storage Class: A default storage class must be configured and available in the Kubernetes cluster to dynamically provision storage as required by the application.

4. Fill out the Environment Readiness Checklist

The Superstream Environment Readiness Checklist ensures that everything is set up for the successful deployment of the SSM engine with reliability and resilience.

You can find the sheet here. Please make a copy and share the link with your project manager.