Step 2: Engine deployment

This guide explaines how to deploy a Superstream on-prem engine on a Kubernetes platform.

If your environment is completely isolated from the public internet, please use this procedure.

Overview

The Superstream chart will deploy the following pods:

  • 2 Superstream engines

  • 3 NATS brokers

  • 1 Superstream syslog adapter

  • 1 Telegraf agent for monitoring

It is highly recommended to deploy one engine per environment (dev, staging, prod)

Getting started

1. Configure Environment Tokens

Create a custom_values.yaml file and edit the relevant values (An example can be found here)

custom_values.yaml
  The following values are required:
  engineName: "" #Engine name
  superstreamAccountId: ""
  superstreamActivationToken: ""

2. Optional. Preconfigure Kafka Connections

Every Engine is capable of establishing connections with one or several Kafka clusters simultaneously.

This type of connectivity is designed for IaaS-based environments.

To determine the optimal strategy for your setup, it is advisable to seek guidance from the Superstream team.

Connection details should be defined in the config.yaml file, example can be found here:

config.yaml
connections:
  - name: <connection_name>
    type: <connection_type> # available types : confluent_kafka, confluent_cloud_kafka, MSK, apache_kafka
    hosts: # list of bootstrap servers
      - <bootstrap_server_1>
      - <bootstrap_server_2>
    authentication: # Specify one preferred method for connection
      method:
        noAuthentication:
          enabled: <true_or_false>
        ssl:
          enabled: <true_or_false>
          protocol: SSL
          ca: "<path_to_ca_cert>"
          cert: "<path_to_cert>"
          key: "<path_to_key>"
          insecureSkipVerify: <true_or_false>
        sasl:
          enabled: <true_or_false>
          mechanism: <sasl_mechanism> # available options: "PLAIN", "SCRAM-SHA-512"
          protocol: SASL_SSL
          username: "<username>"
          password: "<password>"
          tls:
            enabled: <true_or_false>
            ca: "<path_to_ca_cert>"
            cert: "<path_to_cert>"
            key: "<path_to_key>"
            insecureSkipVerify: <true_or_false>
  • name: A unique name for the connection to be displayed through Superstream GUI.

  • type: The type of Kafka. Options include confluent_kafka, confluent_cloud_kafka, MSK, and apache_kafka.

  • hosts: A list of bootstrap servers for the Kafka cluster.

Authentication:

  • noAuthentication: Set enabled to true if no authentication is required.

  • ssl: For SSL encryption without SASL authentication.

    • enabled: Set to true to enable SSL authentication.

    • protocol: Should always be SSL.

    • ca, cert, key: Paths to your CA certificate, client certificate, and client key files.

    • insecureSkipVerify: Set to true to bypass server certificate verification (not recommended for production environments).

  • sasl: For SASL authentication.

    • enabled: Set to true enable SASL authentication.

    • mechanism: The SASL mechanism to use. Options: PLAIN, SCRAM-SHA-512.

    • protocol: Should be SASL_SSL for encrypted connections.

    • username and password: Credentials for SASL authentication.

    • tls: Optional TLS configuration for SASL authentication.

If TLS is used with SASL, specify the following:

  • enabled: Set to true to enable TLS.

  • ca, cert, key: Paths to your CA certificate, client certificate, and client key files, if required.

  • insecureSkipVerify: Set to true to bypass server certificate verification (not recommended for production environments).

Example

Below is an example configuration for a SASL_SSL authenticated connection:

connections:
  - name: my_kafka_connection
    type: confluent_cloud_kafka
    hosts:
      - kafka.example.com:9092
    authentication:
      method:
        sasl:
          enabled: true
          mechanism: PLAIN
          protocol: SASL_SSL
          username: "myUsername"
          password: "myPassword"
          tls:
            enabled: false

Replace placeholders (e.g., <connection_name>, <bootstrap_server_1>) with your actual Kafka connection details. Ensure to follow the correct indentation and formatting as shown in the examples.

For any questions or further assistance, please refer to the official Kafka documentation or reach out to your Kafka provider.

3. Deploy

  1. Go to the custom_values.yaml directory and run:

helm repo add superstream https://k8s.superstream.ai/ --force-update && helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

Deployment verification:

helm list

4. Expose (When Client connectivity is needed. Not a mandatory requirement)

For client connectivity from outside the Kubernetes environment being used, it is necessary to expose the Superstream engine on port 4222 outside of the Kubernetes cluster where Superstream is deployed.

Here is an example YAML file to illustrate the required service configuration:

apiVersion: v1
kind: Service
metadata:
  name: superstream-host-external
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-name: superstream-host-external
spec:
  ports:
  - name: superstream-host-external
    port: 4222
    protocol: TCP
    targetPort: 4222
  selector:
    app.kubernetes.io/component: nats
    app.kubernetes.io/instance: nats
    app.kubernetes.io/name: nats
  type: LoadBalancer

5. Enter Superstream Console

Superstream Console can be found here: https://app.superstream.ai

Appendixes

Appendix A - Non-HA Deployment

For testing purposes only, Superstream can be deployed without HA capabilities. Change to false the following parameter in the values.yaml file:

# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: false

Appendix B - Superstream Update

  1. Retrieve the Most Recent Version of the Superstream Helm Chart

 helm repo add superstream https://k8s.superstream.ai/ --force-update
  1. Make sure to use the same values:

helm get values superstream --namespace superstream
  1. Run the Upgrade command:

helm upgrade --install superstream superstream/superstream -f custom_values.yaml --namespace superstream --wait

Appendix C - Uninstall

Steps to Uninstall Superstream Engine.

  1. Delete Superstream Engine Helm Releases:

helm delete superstream -n <NAMESPACE>
  1. Remove Persistent Storage Bound to the Engine:

It's crucial to delete the stateful storage linked to the Engine. Ensure you carefully specify the namespace in the command below before executing it:

kubectl delete pvc -l app.kubernetes.io/instance=superstream -n <NAMESPACE>

Appendix D - Use Custom StorageClass

StorageClass definition

If there is no default storageClass configured for the Kubernetes cluster or there is a need to choose a custom storageClass, it can be done by specifying its name in the values.yaml file.

# NATS storageClass configuration. The default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""

Appendix E - Deploy Superstream Engine with internal authentication mode on

To enable secure client authentication for the Superstream Engine, edit thevalues.yaml file and set the skipLocalAuthentication parameter to false.

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                   # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: false

Appendix F - Deploy Superstream Engine using labels, tolerations, nodeSelector and etc'

  • To inject custom labels into all services deployed by Superstream, utilize the global.labels variable.

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                   # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  
  labels:
    tests: ok
  • To configure tolerations, nodeSelector, and affinity settings for each deployed service, the adjustments in the following example need to be done:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                   # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  
superstreamEngine:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"
syslog:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"
telegraf:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"
nats:
  podTemplate:
    merge:
      spec:
        tolerations:
          - effect: NoSchedule
            key: node-role.kubernetes.io/master
            operator: Exists

Best practices

Dev / Staging environments

Connecting your Development/Staging Kafka Clusters to Superstream is recommended. This can be done using either one or more dedicated Superstream engines (data planes) for each environment or the same engine connected to the production clusters.

Last updated

Logo

Copyright to Superstream.ai