All pages
Powered by GitBook
1 of 6

Engine deployment

Supertstream Engine Deployment using existing secrets

When the ACTIVATION_TOKEN cannot be exposed in the values.yaml file, it is possible to provide it to the Engine using a pre-created Kubernetes secret containing the relevant data. Follow these steps to create and configure the secret:

Create Kubernetes Secret

Use the following command to create a Kubernetes secret with the required data:

kubectl create secret generic superstream-creds --from-literal=ACTIVATION_TOKEN=<TOKEN> --from-literal=ENCRYPTION_SECRET_KEY=<RANDOM_STRING_OF_32_CHAR> -n superstream

Generate ENCRYPTION_SECRET_KEY

To create the ENCRYPTION_SECRET_KEY, run the following command:

openssl rand -hex 16

Specify Existing Secret in custom_values.yaml

Indicate that you are using an existing secret by adding the following lines to your custom_values.yaml file:

superstreamEngine:  
  secret:
    useExisting: true

Final Configuration

After configuring the secret, your overall configuration should look like this:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                    # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""          # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""    # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
# NATS storageClass configuration. Default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""
superstreamEngine:  
  secret:
    useExisting: true

Deploy

helm repo add superstream https://k8s.superstream.ai/ --force-update && helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

How to deploy and manage Superstream using ArgoCD

Deployment Process:

Create Values YAML Files

  • For easiness create custom_values.yaml file and edit relevant values, an example can be found here:

Edit global.image the section in case it's an air-gapped environment.

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                 # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""       # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: "" # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  
  image:
    # global image pull policy to use for all container images in the chart
    # can be overridden by individual image pullPolicy
    pullPolicy:
    # global list of secret names to use as image pull secrets for all pod specs in the chart
    # secrets must exist in the same namespace
    # https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    pullSecretNames: []
    # global registry to use for all container images in the chart
    # can be overridden by individual image registry
    registry:
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
      # NATS storageClass configuration. Default blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""

  • Official values.file with all abilities can be found here.

  • Push these values.yaml files to your ArgoCD repository.

Deploy Application YAMLs:

Follow the examples below to deploy the application YAML files. Pay close attention to the comments provided for each:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: superstream
  namespace: argocd
  labels:
    app.kubernetes.io/managed-by: Helm
spec:
  destination:
    server: https://kubernetes.default.svc # Destination cluster
    namespace: superstream # Adjust the destination namespace with the file environments/default.yaml
  ignoreDifferences:
  - group: apps
    jsonPointers:
    - /spec/replicas
    kind: Deployment
  project: default
  sources:
  - chart: superstream
    helm:
      valueFiles:
      # Path to the values files in your ArgoCD repository.
      - $values/kubernetes-values/superstream/custom-values.yaml
    repoURL: https://k8s.superstream.ai/
    targetRevision: 0.4.5 # Adjust chart version from helmfile
  - ref: values
    # Your ArgoCD repository  
    repoURL: git@github.com:superstreamlabs/argocd-yamls.git
    targetRevision: master

Sync and Monitor Deployment:

  • Once your values YAML files are pushed and the applications YAML are deployed, navigate to your ArgoCD dashboard.

  • Find the application you just deployed and click the 'Sync' button to initiate the deployment process.

  • Monitor the deployment status to ensure all components are successfully deployed and running.

By following these steps, you should be able to deploy and upgrade the Superstream Engine using ArgoCD successfully. If you have any questions or need further assistance, refer to the documentation or reach out to the support team.

Superstream Engine deployment for environments with a local container registry

How to deploy a Superstream Engine in a fully air-gapped environment with a private container registry.

This guide focuses on three critical applications managed by Superstream Helm Chart: NATS, Telegraf, and Superstream. We will cover each application's specific Docker images that are managed by the helm chart, ensuring you have the information to deploy Superstream in your environment.

Prerequisites

1. Container images

Please store the following images in your container registry:

NATS: An open-source messaging system renowned for its high performance and scalability.

  • Helm version: 1.2.4

  • Containers:

    • natsio/prometheus-nats-exporter:0.15.0

    • natsio/nats-server-config-reloader:0.15.1

    • Nats:2.10.20-alpine

Telegraf: As a versatile agent for collecting, processing, and writing metrics, Telegraf is pivotal in monitoring and observability.

Superstream: The engine itself.

  • Helm version: Releases

  • Helm Chart URL: https://k8s.superstream.ai/

  • Containers:

    • superstreamlabs/superstream-data-plane-be:latest

    • curlimages/curl:8.6.0

    • linuxserver/syslog-ng:4.5.0

To ensure that your private repositories use the correct Docker images, follow these steps to pull images from public repositories and tag them for your private repository. Below are command examples for the related Docker images you might use:

Prometheus NATS Exporter (natsio/prometheus-nats-exporter:0.15.0):

docker pull natsio/prometheus-nats-exporter:0.15.0 
docker tag natsio/prometheus-nats-exporter:0.15.0 YOURREPOSITORY/natsio/prometheus-nats-exporter:0.15.0 
docker push YOURREPOSITORY/natsio/prometheus-nats-exporter:0.15.0

NATS Server Config Reloader (natsio/nats-server-config-reloader:0.14.2):

docker pull natsio/nats-server-config-reloader:0.15.1
docker tag natsio/nats-server-config-reloader:0.15.1 YOURREPOSITORY/natsio/nats-server-config-reloader:0.15.1 
docker push YOURREPOSITORY/natsio/nats-server-config-reloader:0.14.2

NATS Server (nats:2.10.14-alpine):

docker pull nats:2.10.20-alpine
docker tag nats:2.10.20-alpine YOURREPOSITORY/nats:2.10.20-alpine
docker push YOURREPOSITORY/nats:2.10.16-alpine

Telegraf (docker.io/library/telegraf:1.30-alpine):

docker pull library/telegraf:1.30-alpine
docker tag library/telegraf:1.30-alpine YOURREPOSITORY/library/telegraf:1.30-alpine
docker push YOURREPOSITORY/library/telegraf:1.30-alpine

Curl (curlimages/curl:8.6.0):

docker pull curlimages/curl:8.6.0
docker tag curlimages/curl:8.6.0 YOURREPOSITORY/curlimages/curl:8.6.0
docker push YOURREPOSITORY/curlimages/curl:8.6.0

Superstream Engine (superstreamlabs/superstream-data-plane-be:latest):

docker pull superstreamlabs/superstream-data-plane-be:latest
docker tag superstreamlabs/superstream-data-plane-be:latest YOURREPOSITORY/superstreamlabs/superstream-data-plane-be:latest
docker push YOURREPOSITORY/superstreamlabs/superstream-data-plane-be:latest

Syslog (linuxserver/syslog-ng:4.5.0):

docker pull linuxserver/syslog-ng:4.5.0
docker tag linuxserver/syslog-ng:4.5.0 YOURREPOSITORY/linuxserver/syslog-ng:4.5.0 
docker push YOURREPOSITORY/linuxserver/syslog-ng:4.5.0

Getting started

1. Download Helm Chart

Download the Superstream Helm chart from the official source as described above.

2. Publish to Private Environments

Once downloaded, publish the chart to your private Helm chart repositories. This step ensures that you maintain control over the versions and configurations of the chart used in your deployments.

Docker Image Names: You must change the Docker image names within the Helmfile to reflect those stored in your private Docker registries. This customization is crucial for ensuring that your deployments reference the correct resources within your secure environment:

3. Configure All Services to Use a Private Docker Repository and Private PullSecret

For easiness create/use custom_values.yaml file and add global.image section values, an example can be found here:

```yaml
############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: "" # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: "" # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: "" # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  
  image:
    # global image pull policy to use for all container images in the chart
    # can be overridden by individual image pullPolicy
    pullPolicy:
    # global list of secret names to use as image pull secrets for all pod specs in the chart
    # secrets must exist in the same namespace
    # https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    pullSecretNames: []
    # global registry to use for all container images in the chart
    # can be overridden by individual image registry
    registry:
```

4. Deploy

To apply the Helmfile configurations and deploy your Kubernetes resources:

Apply Helmfile: Run the following command to apply the Helmfile configuration. This will sync your Helm releases to match the state declared in your helmfile.yaml:

helm repo add superstream <YOURREPOSITORY> --force-update
helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

Supertstream Engine Deployment using custom resource limits

Superstream Engine is deployed with default resource limits designed to ensure high performance. In some cases, these configured limits may not be sufficient. To address potential performance bottlenecks, you can adjust the resource limits using the procedure outlined below.

Specify desired resource limits in the custom_values.yaml .

To adjust the resource limits for the Superstream Engine Data Plane, add the following configuration:

superstreamEngine:  
  resources:
    limits:
      cpu: '8'
      memory: 8Gi

To modify the resource limits for NATS, add the following configuration:

nats: 
  container:
    merge:
      resources:
        limits:
          memory: 8Gi

Example: Final Configuration File

Below is an example of a complete configuration file (custom_values.yaml) after setting custom resource limits:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                    # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""          # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""    # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
# NATS storageClass configuration. Default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""
  container:
    merge:
      resources:
        limits:
          memory: 8Gi
superstreamEngine:  
  resources:
    limits:
      cpu: '8'
      memory: 8Gi

Deploy

Once you have updated the custom_values.yaml file with your desired resource limits, deploy the Superstream Engine using Helm:

helm repo add superstream https://k8s.superstream.ai/ --force-update && helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

Superstream Platform deployment for Air-Gapped environments

This guide provides instructions for deploying the Superstream Platform for air-gapped environments.

This manual requires a connection to the internet to pull container images and helm charts.

Step 1: Create Secrets with Randomly Generated Passwords for SSM

To create a secret for the Superstream with randomly generated passwords, run the following command:

kubectl create secret generic superstream-creds-control-plane \
  --from-literal=postgres-password=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9') \
  --from-literal=password=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9') \
  --from-literal=repmgr-password=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9') \
  --from-literal=admin-password=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9') \
  --from-literal=superstream-admin-password=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9') \
  --from-literal=control-plane-token=$(openssl rand -base64 48 | tr -dc 'a-zA-Z0-9' | head -c32) \
  --from-literal=encryption-secret-key=$(openssl rand -base64 48 | tr -dc 'a-zA-Z0-9' | head -c32) \
  --from-literal=jwt-secret-key=$(openssl rand -base64 48 | tr -dc 'a-zA-Z0-9' | head -c32) \
  --from-literal=jwt-api-secret-key=$(openssl rand -base64 48 | tr -dc 'a-zA-Z0-9' | head -c32) \
  -n superstream

Important: The secret name superstream-creds-control-plane cannot be changed in the current release. This will be fixed in an upcoming release.

The following keys should have a length of 32 characters:

  • encryption-secret-key

  • jwt-secret-key

  • jwt-api-secret-key

  • control-plane-token

Step 2: Configure Environment Tokens

For a more straightforward configuration, create a custom_values.yaml file and edit the following values:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                    # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""          # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""    # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  onPrem: true  
  ## If your environment uses a proxy server, uncomment the lines below and replace the URL with your proxy server's address.
  proxy:
    enabled: false
    proxyUrl: "https://your-proxy-server"

############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
# NATS storageClass configuration. The default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""
############################################################
# Telegraf config
############################################################
# Telegraf custom environment variables configuration.
# telegraf:
#   env:
#   - name: NO_PROXY
#     value: "10.0.0.0/8,8.8.8.8"

Proxy Configuration

If your environment requires a proxy server to connect to external services, set the global.proxy.enabled variable to true and provide the global.proxy.proxyUrl in the custom_values.yaml file. This configuration ensures that all critical services route traffic through the specified proxy. Additionally, make sure your proxy server permits connectivity to the following endpoints:

  • Prometheus: https://prometheus.mgmt.superstream.ai

  • Loki: https://loki.mgmt.superstream.ai

  • Stigg: https://api.stigg.io

Step 3: Deployment Instructions

To deploy the Superstream, run the following command:

helm repo add superstream-onprem https://k8s-onprem.superstream.ai/ --force-update && \
helm upgrade --install superstream superstream-onprem/superstream-onprem -f custom_values.yaml --create-namespace --namespace superstream --wait

Step 4: Configure valid FQDN records

To use the Superstream User Interface, the following two FQDN records should be exposed under the same domain.

  • Expose the Superstream Control Plane service. Using superstream-api at the beginning of the configured FQDN is a hard requirement. Example: "superstream-api.example.com"

  • Expose the Superstream Control Plane UI service. Example: superstream-app.example.com

  • Log in to the Superstream UI and connect your first Kafka cluster.

Follow these steps to successfully configure and deploy your Superstream Control Plane environment.