The problem

Engineering teams often lack the time, expertise, or tooling to optimize Kafka costs effectively. Cost reduction requires deep visibility into low-level internals, analyzing complex usage patterns over time, safely experimenting without risking data, accurately forecasting capacity needs, and building custom scripts — all of which demand a level of maturity and focus most teams can't afford amid competing priorities.

Introducing Superstream

Superstream AI Platform Helps Kafka Users Easily, Safely, and Automatically Improve Their Very Own Apache Kafka Cost Efficiency by up to 90%.

Common use cases:

• Reduce Kafka Usage Costs: Optimizing your cluster resources to align with your workload can cut costs by up to 80%.

• Increase Control: Enforce best practices and get valuable insights to ensure your Kafka runs at peak health.

• Avoid Migrating To A Different Kafka: Migration is often driven by cost or visibility—now, you can have both with your current vendor.

• Reduce Transfer Costs: By sampling your payload, Superstream can analyze and benchmark it to recommend ways to reduce the transfer footprint.

Start your journey by clicking on the centered button "Start here"

Start your journey by clicking on the centered button "Start here"

Superstream "Bring Your Own Cloud" is the perfect solution for customers who prefer or can't have an external connection from outside their cloud to their Kafka clusters.

Step 1: Check out our "Environment Readiness Checklist"

Step 2: Deploy

Superstream is using a Helm chart to deploy its agent.

You can deploy as many agents as needed and spread your clusters between them based on your environmental preferences.

The Superstream chart will deploy the following pods:

• 2 Superstream engines

• 2 Superstream auto-scaler instances

• 1 Telegraf agent for monitoring

1. Configure the custom_values.yaml file

You can always get your superstreamAccountId and superstreamActivationToken through the console or within the automatic email you received when you signed up.

############################################################
# GLOBAL configuration for Superstream Agent
############################################################
global:
  agentName: ""                       # Define the superstream agent name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""            # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""      # Enter the activation token required for services or resources that need an initial token for activation or authentication.

2. Deploy

Head over to the custom_values.yaml file location and run:

helm repo add superstream https://superstream-agent.k8s.superstream.ai/ --force-update && helm upgrade --install superstream superstream/superstream-agent -f custom_values.yaml --create-namespace --namespace superstream --wait

Deployment verification:

helm list

Appendixes

Appendix A - Superstream Update

1. Retrieve the Most Recent Version of the Superstream Helm Chart

 helm repo add superstream https://superstream-agent.k8s.superstream.ai/ --force-update

1. Make sure to use the same values:

helm get values superstream --namespace superstream

1. Run the Upgrade command:

helm upgrade --install superstream superstream/superstream-agent -f custom_values.yaml --namespace superstream --wait

Appendix B - Uninstall

Steps to Uninstall Superstream Engine.

1. Delete Superstream Engine Helm Releases:

helm delete superstream -n <NAMESPACE>

1. Remove Persistent Storage Bound to the Engine:

It's crucial to delete the stateful storage linked to the Engine. Ensure you carefully specify the namespace in the command below before executing it:

kubectl delete pvc -l app.kubernetes.io/instance=superstream -n <NAMESPACE>

Appendix C - Deploy Superstream Engine using labels, tolerations, nodeSelector and etc'

• To inject custom labels into all services deployed by Superstream, utilize the global.labels variable.

############################################################
# GLOBAL configuration for Superstream Agent
############################################################
global:
  agentName: ""                    # Define the superstream agent name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  
  labels:
    tests: ok

• To configure tolerations, nodeSelector, and affinity settings for each deployed service, the adjustments in the following example need to be done:

############################################################
# GLOBAL configuration for Superstream Agent
############################################################
global:
  agentName: ""                    # Define the superstream agent name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  
superstreamAgent:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"
autoScaler:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"
telegraf:
  tolerations:
  - key: "app"
    value: "connectors"
    effect: "NoExecute"

Appendix D - Deploy Superstream from local registry

• To deploy Superstream from local registry, override default values using global.image variable in custom.valuesfile:

############################################################
# GLOBAL configuration for Superstream Agent
############################################################
global:
  agentName: ""                    # Define the superstream agent name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""         # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""   # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  
  image:
    # Global image pull policy to use for all container images in the chart
    # can be overridden by individual image pullPolicy
    pullPolicy:
    # Global list of secret names to use as image pull secrets for all pod specs in the chart
    # secrets must exist in the same namespace
    # https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    pullSecretNames: []
    # Global registry to use for all container images in the chart
    # can be overridden by individual image registry
    registry: 

Best practices

Dev / Staging environments

Connecting your Development/Staging Kafka Clusters to Superstream is recommended. This can be done using either one or more dedicated Superstream engines (data planes) for each environment or the same engine connected to the production clusters.

Superstream Engine is deployed with default resource limits designed to ensure high performance. In some cases, these configured limits may not be sufficient. To address potential performance bottlenecks, you can adjust the resource limits using the procedure outlined below.

Specify desired resource limits in the custom_values.yaml .

To adjust the resource limits for the Superstream Engine Data Plane, add the following configuration:

superstreamEngine:  
  resources:
    limits:
      cpu: '8'
      memory: 8Gi

To modify the resource limits for NATS, add the following configuration:

nats: 
  container:
    merge:
      resources:
        limits:
          memory: 8Gi

Example: Final Configuration File

Below is an example of a complete configuration file (custom_values.yaml) after setting custom resource limits:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                    # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""          # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""    # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
# NATS storageClass configuration. Default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""
  container:
    merge:
      resources:
        limits:
          memory: 8Gi
superstreamEngine:  
  resources:
    limits:
      cpu: '8'
      memory: 8Gi

Deploy

Once you have updated the custom_values.yaml file with your desired resource limits, deploy the Superstream Engine using Helm:

helm repo add superstream https://k8s.superstream.ai/ --force-update && helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

When the ACTIVATION_TOKEN cannot be exposed in the values.yaml file, it is possible to provide it to the Engine using a pre-created Kubernetes secret containing the relevant data. Follow these steps to create and configure the secret:

Create Kubernetes Secret

Use the following command to create a Kubernetes secret with the required data:

kubectl create secret generic superstream-creds --from-literal=ACTIVATION_TOKEN=<TOKEN> --from-literal=ENCRYPTION_SECRET_KEY=<RANDOM_STRING_OF_32_CHAR> -n superstream

Generate ENCRYPTION_SECRET_KEY

To create the ENCRYPTION_SECRET_KEY, run the following command:

openssl rand -hex 16

Specify Existing Secret in custom_values.yaml

Indicate that you are using an existing secret by adding the following lines to your custom_values.yaml file:

superstreamEngine:  
  secret:
    useExisting: true

Final Configuration

After configuring the secret, your overall configuration should look like this:

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                    # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""          # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: ""    # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
# NATS storageClass configuration. Default is blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""
superstreamEngine:  
  secret:
    useExisting: true

Deploy

helm repo add superstream https://k8s.superstream.ai/ --force-update && helm install superstream superstream/superstream -f custom_values.yaml --create-namespace --namespace superstream --wait

Congratulations! Reaching this step means that all your clusters are well connected and continuously being analyzed and monitored by Superstream.

Here is what we are going to do next:

1. Configure Superstream notifications

2. Optimize Superstream automation for each cluster to align with its specific business logic.

3. Activate Superstream automation

Let's start:

1. Configure Superstream notifications

In Superstream Console -> Settings -> System Settings

Select the notifications, channels, and recipients that Superstream should notify.

2. Optimize Superstream automation

In Superstream Console -> Kafka Clusters -> choose a cluster to start with

Each row in the center represents a specific type of optimization that can be configured before activation.

Let's take "Inactive topics cleanup" as an example.

Click on the Gear Wheel ⚙️ icon

Adjust the values to align with your business logic, then click "Save."

3. Activate

Once we finish tuning the optimization logic, we can now click on "Activate."

Clicking "Activate" enables Superstream automation. This means jobs will be automatically triggered and executed after the specified delay time. If notifications are configured correctly, you will receive updates when a new job is created and when it is completed.

Congratulations! Reaching this step means that all your clusters are well connected and continuously being analyzed and monitored by Superstream.

Here is what we are going to do next:

1. Configure Superstream notifications

2. Optimize Superstream automation for each cluster to align with its specific business logic.

3. Activate Superstream automation

Let's start:

1. Configure Superstream notifications

In Superstream Console -> Settings -> System Settings

Select the notifications, channels, and recipients that Superstream should notify.

2. Optimize Superstream automation

In Superstream Console -> Kafka Clusters -> choose a cluster to start with

Each row in the center represents a specific type of optimization that can be configured before activation.

Let's take "Inactive topics cleanup" as an example.

Click on the Gear Wheel ⚙️ icon

Adjust the values to align with your business logic, then click "Save."

3. Activate

Once we finish tuning the optimization logic, we can now click on "Activate."

Clicking "Activate" enables Superstream automation. This means jobs will be automatically triggered and executed after the specified delay time. If notifications are configured correctly, you will receive updates when a new job is created and when it is completed.

Superstream requires a Kafka user with the below configuration to be able to communicate and analyze connected clusters.

By Kafka flavor/vendor:

AWS MSK

Apache Kafka (Self-hosted)

Step 1: Create a dedicated Kafka user for Superstream

For effective functioning, a user or token requires the following permissions:

• Cluster-level:

  • Describe all topics, List all topics, Describe configs, Describe cluster

• Topic-level:

  • Read: All topics

  • Alter: All topics

  • Delete: All topics

  • Describe: All topics

  • Alter: All topics

  • AlterConfigs: All topics

  • DescribeConfigs: All topics

• Consumer group-level:

  • Describe

  • List Consumer Groups

ACL statements examples:

# Cluster-level permissions
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Describe --cluster
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation DescribeConfigs --cluster
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Describe --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation List --topic '*'

# Topic-level permissions
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Read --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Alter --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Delete --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Describe --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation AlterConfigs --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation DescribeConfigs --topic '*'

# Consumer group-level permissions
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation Describe --group '*'
kafka-acls --authorizer-properties zookeeper.connect=<ZK_HOST:PORT> --add --allow-principal User:<USER> --operation List --group '*'

Step 2: Connection information per cluster

The following information will be required for each cluster:

• Bootstrap servers (Kafka URL)

• Authentication security protocol (No auth / SSL / SASL_SSL)

  • SSL with validation "on" would require a key.pem,cert.pem, and ca.pem

• JMX port and token

Superstream requires a Kafka user with the below configuration to be able to communicate and analyze connected clusters.

By Kafka flavor/vendor:

AWS MSK

Apache Kafka (Self-hosted)

Step 1: Create a dedicated Kafka user for Superstream

For effective functioning, a user or token requires the following permissions:

• Cluster-level:

  • Describe all topics, List all topics, Describe configs, Describe cluster

• Topic-level:

  • Read: All topics

  • Alter: All topics

  • Delete: All topics

  • Describe: All topics

  • Alter: All topics

  • AlterConfigs: All topics

  • DescribeConfigs: All topics

• Consumer group-level:

  • Describe

  • List Consumer Groups

ACL statements examples:

Step 2: Connection information per cluster

The following information will be required for each cluster:

• Bootstrap servers (Kafka URL)

• Authentication security protocol (No auth / SSL / SASL_SSL)

  • SSL with validation "on" would require a key.pem,cert.pem, and ca.pem

• JMX port and token

Deployment Process:

Create Values YAML Files

############################################################
# GLOBAL configuration for Superstream Engine
############################################################
global:
  engineName: ""                 # Define the superstream engine name within 32 characters, excluding '.', and using only lowercase letters, numbers, '-', and '_'.
  superstreamAccountId: ""       # Provide the account ID associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
  superstreamActivationToken: "" # Enter the activation token required for services or resources that need an initial token for activation or authentication.
  skipLocalAuthentication: true
  
  image:
    # global image pull policy to use for all container images in the chart
    # can be overridden by individual image pullPolicy
    pullPolicy:
    # global list of secret names to use as image pull secrets for all pod specs in the chart
    # secrets must exist in the same namespace
    # https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    pullSecretNames: []
    # global registry to use for all container images in the chart
    # can be overridden by individual image registry
    registry:
############################################################
# NATS config
############################################################
# NATS HA Deployment. Default "true"
nats:
  config:
    cluster:
      enabled: true
      # NATS storageClass configuration. Default blank "".
    jetstream:
      fileStore:
        pvc:
          storageClassName: ""

• Push these values.yaml files to your ArgoCD repository.

Deploy Application YAMLs:

Follow the examples below to deploy the application YAML files. Pay close attention to the comments provided for each:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: superstream
  namespace: argocd
  labels:
    app.kubernetes.io/managed-by: Helm
spec:
  destination:
    server: https://kubernetes.default.svc # Destination cluster
    namespace: superstream # Adjust the destination namespace with the file environments/default.yaml
  ignoreDifferences:
  - group: apps
    jsonPointers:
    - /spec/replicas
    kind: Deployment
  project: default
  sources:
  - chart: superstream
    helm:
      valueFiles:
      # Path to the values files in your ArgoCD repository.
      - $values/kubernetes-values/superstream/custom-values.yaml
    repoURL: https://k8s.superstream.ai/
    targetRevision: 0.4.5 # Adjust chart version from helmfile
  - ref: values
    # Your ArgoCD repository  
    repoURL: git@github.com:superstreamlabs/argocd-yamls.git
    targetRevision: master

Sync and Monitor Deployment:

• Once your values YAML files are pushed and the applications YAML are deployed, navigate to your ArgoCD dashboard.

• Find the application you just deployed and click the 'Sync' button to initiate the deployment process.

• Monitor the deployment status to ensure all components are successfully deployed and running.

By following these steps, you should be able to deploy and upgrade the Superstream Engine using ArgoCD successfully. If you have any questions or need further assistance, refer to the documentation or reach out to the support team.

This guide focuses on three critical applications managed by Superstream Helm Chart: NATS, Telegraf, and Superstream. We will cover each application's specific Docker images that are managed by the helm chart, ensuring you have the information to deploy Superstream in your environment.

Prerequisites

1. Container images

Please store the following images in your container registry:

NATS: An open-source messaging system renowned for its high performance and scalability.

• Helm version: 1.2.4

• Containers:

  • natsio/prometheus-nats-exporter:0.15.0

  • natsio/nats-server-config-reloader:0.15.1

  • Nats:2.10.20-alpine

Telegraf: As a versatile agent for collecting, processing, and writing metrics, Telegraf is pivotal in monitoring and observability.

• Helm version: 1.8.48

• Container:

Superstream: The engine itself.

• Helm Chart URL: https://k8s.superstream.ai/

• Containers:

  • superstreamlabs/superstream-data-plane-be:latest

  • curlimages/curl:8.6.0

  • linuxserver/syslog-ng:4.5.0

To ensure that your private repositories use the correct Docker images, follow these steps to pull images from public repositories and tag them for your private repository. Below are command examples for the related Docker images you might use:

Prometheus NATS Exporter (natsio/prometheus-nats-exporter:0.15.0):

NATS Server Config Reloader (natsio/nats-server-config-reloader:0.14.2):

NATS Server (nats:2.10.14-alpine):

Telegraf (docker.io/library/telegraf:1.30-alpine):

Curl (curlimages/curl:8.6.0):

Superstream Engine (superstreamlabs/superstream-data-plane-be:latest):

Syslog (linuxserver/syslog-ng:4.5.0):

Getting started

1. Download Helm Chart

Download the Superstream Helm chart from the official source as described above.

2. Publish to Private Environments

Once downloaded, publish the chart to your private Helm chart repositories. This step ensures that you maintain control over the versions and configurations of the chart used in your deployments.

Docker Image Names: You must change the Docker image names within the Helmfile to reflect those stored in your private Docker registries. This customization is crucial for ensuring that your deployments reference the correct resources within your secure environment:

3. Configure All Services to Use a Private Docker Repository and Private PullSecret

4. Deploy

To apply the Helmfile configurations and deploy your Kubernetes resources:

Apply Helmfile: Run the following command to apply the Helmfile configuration. This will sync your Helm releases to match the state declared in your helmfile.yaml:

Step-by-Step Upgrade Procedure

Step 1: Validate Services and Report Generation

Ensure all services are running and there is no report generation.

Check the logs for confirmation:

Step 2: Scale Down Deployments

Scale down the superstream-data-plane and superstream-syslog deployments.

Step 3: List and Remove Helm Releases

Step 4: Configure Environment Tokens

Step 5: Deploy New Superstream Helm Chart

Add the Superstream Helm repository and deploy using the custom values file.

Step 6: Scale Up Deployments

Scale up the superstream-data-plane and superstream-syslog deployments.

Step 7: Validate the Deployment

Ensure all services are running by listing the Helm releases and verifying their status.

Summary

This document outlines the steps to transition from a Helmfile-based deployment to the new Superstream Helm Chart. The process includes validating the current state, scaling down deployments, removing old Helm releases, configuring environment tokens, deploying the new Helm chart, scaling up the deployments, and validating the final state.

By following these steps, you can ensure a smooth transition and maintain the integrity and availability of the Superstream services.

Step 1: Create Secrets with Randomly Generated Passwords for SSM

To create a secret for the Superstream with randomly generated passwords, run the following command:

• encryption-secret-key

• jwt-secret-key

• jwt-api-secret-key

• control-plane-token

Step 2: Configure Environment Tokens

For a more straightforward configuration, create a custom_values.yaml file and edit the following values:

Step 3: Deployment Instructions

To deploy the Superstream, run the following command:

Step 4: Configure valid FQDN records

To use the Superstream User Interface, the following two FQDN records should be exposed under the same domain.

• Expose the Superstream Control Plane service. Using superstream-api at the beginning of the configured FQDN is a hard requirement. Example: "superstream-api.example.com"

• Expose the Superstream Control Plane UI service. Example: superstream-app.example.com

• Log in to the Superstream UI and connect your first Kafka cluster.

Follow these steps to successfully configure and deploy your Superstream Control Plane environment.

Clients play a vital role in your streaming environment, and Superstream enables you to control, govern, monitor, and optimize them for improved efficiency, throughput, and reliability.

Eligible for customers with "Client optimization" module.

Control your clients

With Superstream, you can instantly modify properties on any connected client without the hassle of coordinating.

Govern your clients

When scaling, it can be challenging to maintain the pace and control how your applications are configured to work with Kafka. Every property can greatly affect both your throughput and the stability of your system, whether it is a managed Kafka or not.

With Superstream, you can specify how each type of application should be configured, receive notifications if it is not, and, if necessary, overwrite the initial configuration with the desired one.

uperstream achieves cost efficiency, enhanced performance, and greater reliability through continuous optimization of both application and infrastructure layers.

• Application layer: Identify and eliminate inactive resources, reduce traffic, apply improved configurations, enforce standardization, identify storage reduction opportunities, and more.

• Infrastructure layer: Clusters are often sized for peak times, but Superstream dynamically identifies the exact resources needed at any given moment and automatically scales accordingly.

Reducing costs in AWS MSK

Your bill structure

AWS MSK Total Cost of Ownership (=TCO) includes:

• Compute (Number of instances X type)

• Egress (Read) in GB/hour

• Storage in GB/hour

Step 1: Optimizing the application layer (clusters/clients)

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions. Decreasing the number of partitions has a direct effect on Memory utilization and the number of needed brokers (instances)

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Step 2: Dynamically scale the cluster's resources

Superstream leverages advanced machine learning algorithms to anticipate resource demands based on historical data and current usage patterns. By continuously analyzing this information, Superstream adjusts cluster sizes in real-time, ensuring that resources are allocated precisely when and where they are needed. This dynamic scaling guarantees optimal performance and significantly reduces operational costs by avoiding the need to provision resources for peak load conditions that may only occur sporadically.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Getting started

Step 1: Access your MSK cluster through the Superstream Console

Step 2: Scroll down to "Autoscaler" and define its rules

Step 3: Activate

Superstream will now initiate and monitor the cluster, evaluating whether any defined rules are satisfied. If a rule is met, Superstream will execute the corresponding user-defined action.

Ref: Scale-in a cluster (reduce the number of brokers)

Ref: How to manually scale an MSK cluster in case Superstream is down

Overview

This manual provides step-by-step instructions for scaling Amazon Managed Streaming for Apache Kafka (MSK) clusters, including both horizontal and vertical scaling approaches.

Horizontal Scaling (Adding Brokers)

Planning Phase

1. Monitor current cluster metrics:

   • CPU utilization

   • Storage utilization

   • Network throughput

   • Partition distribution

2. Calculate required capacity:

   • Number of partitions per broker

   • Expected throughput per broker

Implementation Steps

Using AWS Console

1. Navigate to the Amazon MSK console

2. Select your cluster

3. Click "Actions" → "Edit cluster configuration"

4. Under "Brokers", modify the number of brokers per Availability Zone

5. Review and confirm changes

6. Monitor the scaling operation in the console

Using AWS CLI

aws kafka update-broker-count \
    --cluster-arn <your-cluster-arn> \
    --current-version <current-cluster-version> \
    --target-number-of-broker-nodes <new-broker-count>

Vertical Scaling (Broker Type Update)

Planning Phase

1. Identify target broker type based on:

   • CPU requirements

   • Memory needs

   • Network capacity requirements

   • Cost considerations

Implementation Steps

Using AWS Console

1. Navigate to the Amazon MSK console

2. Select your cluster

3. Click "Actions" → "Update broker type"

4. Select new broker type

5. Schedule the update

6. Review and confirm changes

Using AWS CLI

aws kafka update-broker-type \
    --cluster-arn <your-cluster-arn> \
    --current-version <current-cluster-version> \
    --target-instance-type <new-instance-type>

Best Practices

• Scale during low-traffic periods

• Maintain sufficient headroom (20-30%) for unexpected traffic spikes

• Monitor scaling operations closely

• Keep cluster configuration version updated

• Document all scaling operations

Troubleshooting

Common Issues

1. Insufficient Capacity Errors

   • Solution: Verify available capacity in target AZs

   • Contact AWS support if needed

2. Scaling Operation Timeout

   • Solution: Check AWS CloudWatch logs

   • Verify network connectivity

   • Review security group configurations

3. Uneven Partition Distribution

   • Solution: Run kafka-reassign-partitions tool

   • Review partition assignment strategy

Monitoring and Maintenance

Key Metrics to Monitor

• Broker CPU utilization

• Storage utilization

• Network throughput

• Producer/consumer latency

• Partition replication lag

Superstream provides an innovative way to streamline property modifications across all connected clients. By using Superstream, you can instantly update and synchronize changes without the hassle of manual coordination. This allows for real-time updates, ensuring that all clients receive the latest modifications promptly and efficiently, ultimately saving time and reducing the chances.

How to modify an individual client's property

Please go to: Menu -> Connected clusters -> <SOME CLUSTER> -> "Clients" tab

Choose the client you wish to modify. For this example, let's enable compression:

• "Current": The initial value that was given to the application

• "Overwrite": The overwrite value given by Superstream

Click "Save". No need to reboot the application.

Superstream is designed to deliver a seamless experience, using lightweight notifications to keep you and your team informed about actions across your clusters.

For Slack notifications

1. Log In to Slack

   • Navigate to Slack and log in to your workspace using your credentials.

2. Access Slack API Management

   • Visit the Slack API page.

   • Click on the "Your Apps" button in the top-right corner.

3. Create a New App

   • On the Your Apps page, click "Create an App".

   • Select "From Scratch".

   • Provide a name for your app and choose the workspace where you want to send messages.

   • Click "Create App".

4. Enable Incoming Webhooks

   • In the app settings, locate the "Features" section on the left-hand menu and click on "Incoming Webhooks".

   • Toggle the Activate Incoming Webhooks option to On.

5. Create a Webhook URL

   • Scroll down to the Webhook URLs for Your Workspace section.

   • Click "Add New Webhook to Workspace".

   • Choose the Slack channel where you want the messages to be sent.

   • Click "Allow" to grant the necessary permissions.

6. Copy the Webhook URL

   • Once created, you’ll see the webhook URL under the Webhook URLs for Your Workspace section.

   • Copy the URL to use it in your application or script.

7. In Superstream Console

   1. Toggle the REST channel

   2. Paste the URL

   3. Add the following header Content-type: application/json

Superstream is committed to maintaining the highest standards of data security and compliance.

Our platform is certified with leading industry standards, including ISO 27001 for information security management, GDPR for data protection, and SOC 2 (Type I+II) for managing customer data based on trust service principles. These certifications demonstrate our dedication to protecting client data and ensuring the integrity and confidentiality of your information.

1. SOC 2 Type 1 and 2: Superstream meets the stringent requirements of Service Organization Control (SOC) 2, both Type 1 and Type 2. This ensures the platform's security, availability, processing integrity, confidentiality, and privacy of customer data are in line with the American Institute of Certified Public Accountants (AICPA) standards.

2. ISO 27001: Superstream aligns with ISO 27001, a globally recognized standard for information security management systems (ISMS). This certification indicates a commitment to a systematic and ongoing approach to managing sensitive company and customer information.

3. GDPR: Compliance with the General Data Protection Regulation (GDPR) underscores Superstream's dedication to data privacy and protection in accordance with European Union regulations. This includes ensuring user data rights and implementing appropriate data handling and processing measures.

We offer multiple authentication methods to ensure secure and seamless access to our platform, catering to a variety of use cases. Whether you're integrating Superstream into an existing enterprise system or setting up individual access, our authentication options provide flexibility without compromising on security. This guide outlines the available authentication methods, including API key authentication, OAuth, and role-based access control (RBAC), to help you choose the best approach for your needs.

Manage permissions via Active Directory

A custom attribute named superstream_role must be created to define user permissions within Superstream. Currently, the attribute supports the following values:

• admin – Full access to manage and configure Superstream.

• read-only – Limited access with view-only permissions.

This attribute should be assigned to all users requiring access to Superstream.

This role system will be expanded to support tag-based roles in the future, enabling more granular permission control.

The data stays on the customer's premises, and only the metadata, which consists of calculated results, is transmitted according to the customer's choice to either:

• The Superstream control plane located off the premises.

• An on-premises control plane ensures that no data whatsoever leaves the premises.

A specific list of data to be fetched from each Kafka cluster:

Topic Metadata:

• List of Topics: Retrieve all available topics in the Kafka cluster.

• Topic Configuration: Access detailed configuration settings for each topic, such as retention policies, partition count, replication factor, activity events, connected CGs, and segment file settings.

• Partition Information: Details about each topic's partitions, including partition IDs and their current leaders.

• Replicas placements

• Data samples

Consumer Group Metadata:

• Consumer Group List: List of all consumer groups connected to the Kafka cluster.

• Consumer Group Offsets: Information about the offset position for each consumer group in each partition of a topic.

• Consumer Group State: Current state of consumer groups, like whether they are active or in a rebalance phase.

• Static membership

Broker Metadata:

• Broker IDs and Addresses: Information about each broker in the Kafka cluster, including their IDs and network addresses.

• Broker Configurations: Configuration details of each broker, like log file size limits, message size limits, and more.

• Broker Metrics and Health: Data about broker performance, such as CPU and memory usage, network I/O, and throughput metrics.

• Rack aware id

Cluster Metadata:

• Cluster ID: Unique identifier of the Kafka cluster.

• Controller Broker Info: Details about the current controller broker, which is responsible for maintaining the leader election for partitions.

Log Metadata:

• Log Size and Health: Information on the log size for each topic and partition, and details on log segment files.

Firewall rules

These terms of service (the "Terms") together with any other agreements or terms incorporated by reference, govern the use of the Services. These Terms constitute a binding and enforceable legal contract between Company and any party (the "Customer") entering into an order form incorporating these Terms by reference (an "Order") or otherwise using the Services.

1. Services; License

   1. Subject to the terms and conditions of these Terms, and any applicable Order, Company shall provide Customer with services detailed in the Order ("Services") through its proprietary technology as hosted on a third party cloud service (the "Platform"), in accordance with the service level agreement available at https://superstream.ai/legal/service-level-agreement.

   2. During the Term and subject to Customer’s compliance with the terms and conditions of this Agreement, Company grants Customer a non-exclusive, non-transferable, royalty-free, non-sublicensable, limited, revocable right (i) for Customer employees, agents, representatives and contractors who are permitted access to the Services by Customer ("Authorized Users") to access the Platform, for Customer’s internal business use, and up to such number of licenses set forth in the Order; provided by Company to enable the Services according to Company’s instructions and technical documentation provided by Company from time to time ("Documentation").

2. Customer’s Obligations. Customer hereby undertakes to:

   1. implement the software provided by Company according to the Documentation;

   2. provide Company with complete and accurate information necessary to implement the Service. Customer acknowledges and agrees that the accuracy of the Services depends on the quality, completeness and timeliness of the data provided to Company, and on Customer providing Company with prompt notice upon any changes in required information;

   3. use the Platform, the Services, and all related software and Documentation in compliance with all applicable laws and regulations, including but not limited to applicable data security and privacy laws;

   4. manage and secure all login credentials used by Authorized Users in connection with their use of the Platform and protect the same against unauthorized use or disclosure using reasonable standard in the industry. Customer shall be liable for all actions taken on the Platform through use of the login credentials of its Authorized Users.

3. Fees. In consideration for the Services, Customer shall pay Company the Fees, as described in the Order. Fees shall be payable within 30 days of issuance of the applicable invoice. All payments not made when due, shall be subject to a late charge of 1.5% per month compounded annually. Payment of Fees shall be made by wire transfer to the account designated by Company. The Fees are exclusive of VAT. In the event that Customer is required by law to deduct and/or withhold any amounts from any payments due hereunder, it shall gross-up and increase the amounts to be paid to Company so that the actual net amount to be paid to Company shall equal the fees that would have been due to Company without such deduction or withholding.

4. Intellectual Property Rights; Confidentiality

   1. All intellectual property rights in the Platform, Services, Documentation, all related software and any part thereof, including any and all derivatives, changes and improvements thereof lie exclusively with Company. Customer shall (i) not attempt to infiltrate, hack, reverse engineer, decompile, or disassemble the Platform, Services, any related software or any part thereof for any purpose or use it to build a competitive service or product; (ii) not represent that it possesses any proprietary interest in Platform, Service, Documentation, any related software or any part or derivative thereof; (iii) not directly or indirectly, take any action to contest Company's intellectual property rights or infringe them in any way; (iv) except as specifically permitted in writing by Company, not use the name, trademarks, trade-names, and logos of Company; (v) except as specifically permitted herein, not copy any part or content of the Platform, reports or documentation other than for Customer’s own internal business purposes; (vi) not copy any features, functions or graphics of the Platform or use it to build a competitive product or service; and (vii) not remove the copyright, trademark and other proprietary notices contained on or in Company's Platform, products, services or documentation. Customer shall take no action, directly or indirectly, to register Company trademarks (or their variation), domain names, or copyrights in its own name and shall provide commercially reasonable assistance to Company to prevent the occurrence of such activity by any third parties. All intellectual property rights, as well as any data, materials or Confidential Information, including any modifications, enhancements and derivatives thereof (collectively, “Customer IPR”) lie exclusively with Customer or its licensors.

   2. Customer hereby grants to Company: (a) a non-exclusive, non-sublicensable, revocable, non-transferable, royalty-free, worldwide license during the Term to use, reproduce, and prepare derivative works of all data provided to Company by Customer in connection with this Agreement solely to permit Company to perform the Services to Customer as set forth in this Agreement, and (b) a perpetual, non-exclusive, non-sublicensable, revocable, non-transferable, royalty-free, worldwide license to analyze anonymized data obtained through Services in order to provide Customer with analytics and reports, and improve the Services, all subject to Company’s compliance with applicable law and privacy regulations.

   3. The receiving party agrees (i) not to disclose the disclosing party’s Confidential Information to any third parties other than to its directors, employees, advisors, or consultants (collectively, its “Representatives”) on a “need to know” basis and provided that such Representatives are bound by confidentiality obligations not less restrictive than those contained herein; (ii) not to use or reproduce any of the disclosing party’s Confidential Information for any purposes except to carry out its rights and responsibilities under this Agreement; (iii) to keep the disclosing party’s Confidential Information confidential using at least the same degree of care it uses to protect its own confidential information, which shall in any event not be less than a reasonable degree of care. Notwithstanding the foregoing, if the receiving party is required by legal process or applicable law, rule, or regulation to disclose any of the disclosing party’s Confidential Information, then prior to such disclosure, if legally allowed, receiving party will give prompt notice to the disclosing party so that it may seek a protective order or other appropriate relief. The confidentiality obligations hereunder shall expire five years from the date of termination or expiration of this Agreement (and with respect to trade secrets- in perpetuity) and shall supersede any previous confidentiality undertakings between the parties.

   4. For the purposes hereof, "Confidential Information" means any proprietary or trade secret information disclosed by one party to the other which can be reasonably understood under the circumstances to be confidential, but excluding any information that: (i) is now or subsequently becomes generally available in the public domain through no fault or breach on the part of receiving party; (ii) the receiving party can demonstrate in its records to have had rightfully in its possession prior to disclosure of the Confidential Information by the disclosing party; (iii) the receiving party rightfully obtains from a third party who has the right to transfer or disclose it, without default or breach of this Agreement; (iv) the receiving party can demonstrate in its records to have independently developed, without breach of this Agreement and/or any use of or reference to the Confidential Information. For avoidance of doubt, Customer IPR shall be deemed Customer Confidential Information.

5. Warranties; Disclaimer; Limitation of Liability

   1. EXCEPT AS EXPLICITLY SET FORTH IN THIS AGREEMENT, COMPANY PROVIDES THE PLATFORM, SERVICES, ANY RELATED SOFTWARE AND DOCUMENTATION TO CUSTOMER ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR REPRESENTATION OF ANY KIND, AND COMPANY EXPRESSLY DISCLAIMS ALL WARRANTIES - STATUTORY, EXPRESS, IMPLIED OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHENTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR ACCURACY. COMPANY FURTHER DISCLAIMS ANY WARRANTY THAT THE OPERATION OF THE PLATFORM OR ANY RELATED SERVICES WILL BE UNINTERRUPTED OR ERROR-FREE.

   2. Customer acknowledges that the quality and accuracy of the Services are dependent on the accuracy and completeness of the information provided. CUSTOMER ACKNOWLEDGES THAT COMPANY SHALL NOT BEAR ANY LIABILITY OR RESPONSIBILITY FOR FAULTS, ERRORS OR ERRONEOUS OUTPUT PROVIDED ON THE BASIS OF UNTIMELY, INCOMPLETE, INACCURATE, FALSE OR MISLEADING INFORMATION PROVIDED BY CUSTOMER OR THE AUTHORIZED USERS.

   3. EXCEPT FOR WILLFUL MISCONDUCT, FRAUD OR BREACH OF CONFIDENTIALITY OBLIGATIONS HEREUNDER, AND TO THE MAXIMUM EXTENT PERMITTED BY LAW, EACH PARTY'S MAXIMUM AGGREGATE LIABILITY UNDER, ARISING OUT OF OR RELATING TO THIS AGREEMENT SHALL NOT EXCEED THE TOTAL AMOUNT OF FEES PAID BY CUSTOMER TO COMPANY DURING THE 12 MONTHS PRECEDING THE DATE THE LIABILITY FIRST ARISES. TO THE MAXIMUM EXTENT PERMITTED BY LAW, IN NO EVENT WILL EITHER PARTY BE LIABLE FOR LOST PROFITS, LOSS OF USE, LOSS OF DATA (INCLUDING END-USER INFORMATION), COST OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR FOR SPECIAL, PUNITIVE, INCIDENTAL, INDIRECT, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED, WHETHER FOR BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE AND STRICT LIABILITY), OR OTHERWISE, WHETHER OR NOT SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

   4. Company shall defend or settle, at its option and expense, any action brought by a third party against Customer, only to the extent such action arises from any third party claim brought against Customer alleging that the Platform, Services and/or Documentation infringes any patent, copyright, trademark, trade secret, or other intellectual property right of such third party, and shall pay all costs, liabilities, damages and reasonable legal fees finally awarded against Customer in, or paid in settlement of, such action. Indemnification hereunder shall be conditioned upon: (a) Customer notifying Company of the claim immediately upon becoming aware thereof, (b) Customer allowing Company to assume full control of the defense and settlement of such claim, and (c) Customer reasonably cooperating with Company in the defense and settlement of the claim.

6. Term; Termination

   1. This Agreement shall commence on the Effective Date and will continue for the Term set forth in the Order.

   2. Either party may terminate this Agreement immediately by giving written notice to the other party if: (i) the other party breaches a material provision of this Agreement and fails to cure the breach within seven days after being given written notice thereof; (ii) the other party is judged bankrupt or insolvent, makes a general assignment for the benefit of its creditors, a trustee or receiver is appointed for such party; or any petition by or on behalf of such party is filed under any bankruptcy or similar laws.

   3. Upon termination of this Agreement, Customer will immediately cease use of the Platform and any Services, each party shall return to the other party or destroy (at disclosing party’s option) all of the other party's Confidential Information in its possession and any outstanding undisputed Fees shall become due and payable. Sections ‎4, ‎5, ‎6.3, ‎7, and ‎9 shall survive any expiration or termination of this Agreement.

   4. Notices. All notices or other communications hereunder shall be in writing and given in person, by registered mail, by an overnight courier service which obtains a receipt to evidence delivery, or by facsimile or email transmission with written confirmation of receipt, addressed to the address set forth in the Order or to such other address as any party hereto may designate to the other in accordance with the aforesaid procedure. All notices and other communications delivered in person or by courier service shall be deemed to have been given upon delivery, those given by facsimile or email transmission shall be deemed given on the business day following transmission, and those sent by registered mail shall be deemed given three calendar days after posting.

7. Publicity. Company may issue publicity or general marketing communications concerning its involvement with the Customer, including a case study, and Customer agrees to reasonably cooperate with and contribute to such case study.

General. This Agreement constitutes the entire agreement between Company and Customer and supersedes any previous agreements or representations, either oral or written, with respect to the subject matter of this Agreement. All amendments will be made only in written agreement by both parties. Customer shall not transfer or assign its rights or obligations under this Agreement to any third party, except to an affiliate or in the case of merger or sale of all or substantially all of the assigning party's assets. Any purported assignment contrary to this section shall be void. If any part of this Agreement is declared invalid or unenforceable for any reason, such part shall be deemed modified to the extent necessary to make it valid and operative and in a manner most closely representing the intention of the parties, or if it cannot be so modified, then eliminated, and such elimination shall not affect the validity of any remaining portion, which shall remain in force and effect. Any failure by a party to insist upon or enforce performance by the other of any of the provisions of this Agreement or to exercise any rights or remedies under this Agreement or otherwise by law will not be construed as a waiver or relinquishment of any right to assert or rely upon the provision, right or remedy in that or any other instance. This Agreement is governed by the laws of the State of New York, without regards to its conflict of laws principles, and any dispute arising from this Agreement shall be brought exclusively before the courts of NYC, New York.

Superstream enhances the experience of Confluent Platform users in numerous ways, including significant cost reduction, improved performance, increased reliability, and robust governance.

By optimizing resource usage and streamlining operations, Superstream ensures that users not only save money but also achieve smoother, more efficient, and more secure data management. This comprehensive support helps Confluent Platform users maximize their investment and maintain a high standard of operational excellence.

Reducing costs

Your bill structure

The Confluent Platform is a self-hosted version of Kafka developed by Confluent. For our purposes, we will address costs by referring to this deployment as self-hosted Kafka, implying traditional self-hosted expenses.

The Total Cost of Ownership (=TCO) include:

• Compute (Number of instances X type)

• Egress (Read) in GB/hour

• Storage in GB/hour

How Superstream Can Help Reduce Costs

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions. Decreasing the number of partitions has a direct effect on the Memory utilization and the number of needed brokers (instances)

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Improve performance

Where are the bottlenecks in Confluent's Kafka?

• Untuned filesystem and operating system

• Data Transfer: Limited network bandwidth can restrict the amount of data that can be transferred, especially in high-throughput scenarios.

• Producer Latency: High producer latency can occur due to batching, request sizes, or insufficient resources.

• Consumer Lag: Consumer lag occurs when consumers cannot keep up with the rate of messages produced.

• Large Number of Partitions: Managing a large number of partitions can strain broker resources and increase metadata overhead.

• Uneven Partition Distribution: Uneven partition distribution can lead to hotspots, where some brokers handle more load than others.

• Replication Lag: Replication lag occurs when follower replicas cannot keep up with the leader, leading to data inconsistencies.

• Acknowledgment Settings: Configuring acks=all can increase latency as the producer waits for acknowledgments from all replicas.

Here is how Superstream mitigates these issues

1. Optimize kernel-level parameters and the filesystem (both XFS and EXT4) to best suit Kafka workloads.

2. Ensure compression is enabled and enforce otherwise to reduce data size and optimize network usage.

3. Optimize producer configurations, such as batch size and linger.ms. Ensure producers have adequate CPU and memory resources.

4. Optimize consumer configurations, such as fetch size and max.poll.records. Scale consumer groups and partitions to distribute the load more effectively.

5. Optimize the number of partitions per topic based on throughput requirements.

6. Suggests improved key and partition strategies.

7. Ensure sufficient resources for follower replicas. Optimize replication settings, such as min.insync.replicas and replication.factor.

8. Adjust the ack parameter as needed.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactive Measures: Enforce strict governance over clients to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactive Measures: Acknowledge that "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. High-availability in case of a failure.

Here is how Superstream can help

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user is still able to commit a violation, Superstream will identify it in its continuous scanning and mitigate it automatically or manually.

3. High-availability: While you can use MirrorMaker 2.0 or Cluster linking in the case of Confluent, the solution is not hermetic. Here is why:

   1. You would still need to fix offset translations

   2. Clients will need to reroute to the DR cluster

   3. You would need to flush the clients' cache or build an internal mechanism to sync the leftover offsets directly in the DR cluster

   4. You can't failback

   It is true that the need to failover is not common, but what happens if it does happen? Can your organization or product sustain that? The potential RPO and RTO is high. Can your reputation sustain it? Can you bear the data loss? We can do better. With SuperShield, your RPO and RTO would be as minimal as possible, and the entire process would take place completely automatically. So you would be able to sleep better, your customers would trust you more, and your business would continue to work as usual.

This Privacy Policy was last revised on 31 Dec, 2024.

1. Purpose of this Privacy Policy.

Strech, Inc. (dba Superstream) is committed to protecting your privacy. We have prepared this Privacy Policy to describe to you our practices regarding the Personal Data (as defined below) we collect from users of our website located at superstream.ai and in connection with our Superstream products and services (the "Products"). In addition, this Privacy Policy tells you about your privacy rights and how the law protects you.

It is important that you read this Privacy Policy together with any other privacy notice or fair processing notice we may provide on specific occasions when we are collecting or processing Personal Data about you so that you are fully aware of how and why we are using your data. This Privacy Policy supplements the other notices and is not intended to override them.

2. Controller and Contact Details.

Strech Inc. (collectively referred to as “Superstream,” “we,” “us” or “our” in this Privacy Policy) is the controller of Personal Data submitted in accordance with this Privacy Policy and is responsible for that Personal Data. We have appointed a data protection officer (DPO) who is responsible for overseeing questions in relation to this Privacy Policy. If you have any questions about this Privacy Policy, including any requests to exercise your legal rights, please contact our DPO at support@superstream.ai.

3. Types Of Data We Collect.

We do not collect Personal Data (besides full name and company for login purposes). We do collect anonymous data from you when you visit our site, when you send us information or communications, when you engage with us through online chat applications, when you download and use our Products, and when you register for white papers, web seminars, and other events hosted by us. "Personal Data" means data that identifies, relates to, describes, can be used to contact, or could reasonably be linked directly or indirectly to you, including, for example, identifiers such as your real name, alias, postal address, unique personal identifier, online identifier, Internet Protocol (IP) address, email address, account name, or other similar identifiers; commercial information, including records of products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies; Internet or other electronic network activity information. "Anonymous Data" means data that is not associated with or linked to your Personal Data; Anonymous Data does not permit the identification of individual persons. We do not collect any Special Categories of Personal Data about you (this includes details about your race or ethnicity, religious or philosophical beliefs, sex life, sexual orientation, political opinions, trade union membership, information about your health and genetic and biometric data).

3.1 Personal Data You Provide to Us.

We collect Personal Data from you, such as your first and last name, e-mail and mailing addresses, professional title, and company name when you download and install the Products, create an account to log in to our network, engage with us through online chat applications, or sign-up for our newsletter or other marketing material (internet/electronic activity).

3.2 Personal Data Collected Via Cookies.

We also use Cookies (as defined below) and navigational data like Uniform Resource Locators (URL) to gather information regarding the date and time of your visit and the solutions and information for which you searched and which you viewed (Internet/electronic activity). Like most technology companies, we automatically gather this Personal Data and store it in log files each time you visit our website or access your account on our network. "Cookies" are small pieces of information that a website sends to your computer’s hard drive while you are viewing a web site. We may use both session Cookies (which expire once you close your web browser) and persistent Cookies (which stay on your computer until you delete them) to provide you with a more personal and interactive experience on our website. Persistent Cookies can be removed by following Internet browser help file directions. You may choose to refuse or disable Cookies via the settings on your browser, however, by doing so, some areas of our website may not work properly.

3.5 Personal Data That We Collect From You About Others.

If you decide to create an account for and invite a third party to join our network, we will collect your and the third party's names and e-mail addresses (identifiers) in order to send an e-mail and follow up with the third party. You or the third party may contact us at support@superstream.ai to request the removal of this information from our database.

4. Use Of Your Data.

4.1 General Use.

Any Data you submit to us is only used to respond to your requests or to aid us in serving you better.

4.2 Creation of Anonymous Data.

We may create Anonymous Data records from collected data by excluding information (such as your name and IP address) that makes the data personally identifiable to you. We use this Anonymous Data to analyze request and usage patterns so that we may enhance the content of our Products and improve site navigation, and for marketing and analytics.

4.3 Feedback.

If you provide feedback on any of our Products or our website, we may use such feedback for any purpose, provided we will not associate such feedback with your Personal Data. We will collect any information contained in such communication and will treat the Personal Data in such communication in accordance with this Privacy Policy.

5. Disclosure Of Your Personal Data.

5.1 Disclosure to Affiliates.

We will never share your Personal Data with other companies for advertisements, ads, or affiliation. If another company acquires us or our assets, that company will possess the Personal Data collected by it and us and will assume the rights and obligations regarding your Personal Data as described in this Privacy Policy. We may also disclose your Personal Data to third parties in the event that we sell or buy any business or assets, in which case we may disclose your Personal Data to the prospective seller or buyer of such business or assets.

5.2 Disclosure to Third Party Service Providers.

Superstream does not and will not sell any data or metadata collected. Except as otherwise stated in this policy, we do not share the Personal Data that we collect with other entities. However, we may share your Personal Data -- including each category of Personal Data described above -- with third party service providers to: (a) provide you with the Products that we offer you through our website; (b) process payments; (c) conduct quality assurance testing; (d) facilitate creation and maintenance of accounts; (e) collect and analyze data; (f) provide technical support; or (g) provide specific business services, such as synchronization with other software applications and marketing services. These third party service providers are required by written agreement not to retain, use, or disclose your Personal Data other than to provide the services requested by us.

5.3 Disclosure to Other Third Party Companies.

We will not disclose your Personal Data to other Third Party Companies except as otherwise stated in this policy.

5.4 Other Disclosures.

Regardless of any choices you make regarding your Personal Data (as described below), we may disclose Personal Data if we believe in good faith that such disclosure is necessary to (a) comply with relevant laws or to respond to subpoenas or warrants served on us; (b) protect or defend our rights or property or the rights or property of users of the Products; or (c) protect against fraud and reduce credit risk.

6. Your Choices Regarding Your Personal Data.

We offer you choices regarding the collection, use, and sharing of your Personal Data. We will periodically send you free newsletters and e-mails that directly promote the use of our site or the purchase of our Products. When you receive newsletters or promotional communications from us, you may indicate a preference to stop receiving further communications from us and you will have the opportunity to "opt-out" by following the unsubscribe instructions provided in the e-mail you receive or by contacting us directly (please see contact information above). Despite your indicated e-mail preferences, we may send you notices of any updates to our Privacy Policy.

7. Your Legal Rights regarding your Personal Data.

Under certain circumstances, you may have rights under applicable data protection laws in relation to your Personal Data.

Where applicable, you may have the right to:

Request information about how we collect, process, use and share your Personal Data (commonly known as a “right to know request”).

Request access to your Personal Data (commonly known as a “data subject access request”). This enables you to receive a copy of the Personal Data we hold about you and to check that we are lawfully processing it.

Request correction of the Personal Data that we hold about you. This enables you to have any incomplete or inaccurate data we hold about you corrected, though we may need to verify the accuracy of the new data you provide to us.

Request erasure of your Personal Data (commonly known as a “request to be forgotten”). This enables you to ask us to delete or remove Personal Data. You also have the right to ask us to delete or remove your Personal Data where you have successfully exercised your right to object to processing (see below), where we may have processed your information unlawfully or where we are required to erase your Personal Data to comply with local law. Note, however, that we may not always be able to comply in full with your request of erasure for specific legal reasons which will be notified to you, if applicable, at the time of your request.

Object to processing of your Personal Data where we are relying on a legitimate interest (or those of a third party) and there is something about your particular situation which makes you want to object to processing on this ground as you feel it impacts on your fundamental rights and freedoms. You also have the right to object where we are processing your Personal Data for direct marketing purposes. In some cases, we may demonstrate that we have compelling legitimate grounds to process your information which override your rights and freedoms.

Request restriction of processing of your Personal Data. This enables you to ask us to suspend the processing of your Personal Data in the following scenarios: (a) if you want us to establish the data’s accuracy; (b) where our use of the data is unlawful but you do not want us to erase it; (c) where you need us to hold the data even if we no longer require it as you need it to establish, exercise or defend legal claims; or (d) you have objected to our use of your data but we need to verify whether we have overriding legitimate grounds to use it.

Request the transfer of your Personal Data to you or to a third party. We will provide to you, or a third party you have chosen, your Personal Data in a structured, commonly used, machine-readable format. Note that this right only applies to automated information which you initially provided consent for us to use or where we used the information to perform a contract with you.

Withdraw consent at any time where we are relying on consent to process your Personal Data. However, this will not affect the lawfulness of any processing carried out before you withdraw your consent. If you withdraw your consent, we may not be able to provide certain products or services to you. We will advise you if this is the case at the time you withdraw your consent.

If you wish to exercise any of the above rights please contact support@superstream.ai. You will not have to pay a fee to access your Personal Data (or to exercise any of the other rights), and Superstream does not discriminate based on whether you choose to exercise your choice and rights. We will not, based on your exercise of rights, deny our Products to you, charge you different rates, provide a different level or quality of Products to you, or suggest that you may receive such different treatment. However, we may charge a reasonable fee if your request is clearly unfounded, repetitive or excessive.

Alternatively, we may refuse to comply with your request in these circumstances. We may need to request specific information from you to help us confirm your identity and ensure your right to access your Personal Data (or to exercise any of your other rights).

This is a security measure to ensure that Personal Data is not disclosed to any person who has no right to receive it. As part of the verification process, we match the information submitted as part of your request against information stored by Superstream. In some instances, we will require additional information in order to verify your request. If an authorized third party makes a data subject request on your behalf, we will require sufficient written proof that you have designated them as your authorized agent. We try to respond to all legitimate requests within one month. Occasionally it may take us longer than a month if your request is particularly complex or you have made a number of requests. In this case, we will notify you and keep you updated.

9. Data Retention.

We will only retain your Personal Data for as long as necessary to fulfill the purposes we collected it for, including for the purposes of satisfying any legal, accounting, or reporting requirements.

To determine the appropriate retention period for Personal Data, we consider the amount, nature, and sensitivity of the Personal Data, the potential risk of harm from unauthorised use or disclosure of your Personal Data, the purposes for which we process your Personal Data and whether we can achieve those purposes through other means, and the applicable legal requirements.

10. Dispute Resolution.

If you believe that we have not adhered to this Privacy Policy, please contact us by e-mail at support@superstream.ai. We will do our best to address your concerns. If you feel that your complaint has been addressed incompletely, we invite you to let us know for further investigation. If we are unable to reach a resolution to the dispute, we will settle the dispute exclusively under the rules of the American Arbitration Association.

11. Changes To This Privacy Policy.

This Privacy Policy is subject to occasional revision, and if we make any substantial changes in the way we use your Personal Data, we will notify you by sending you an e-mail to the last e-mail address you provided to us or by prominently posting notice of the changes on our website. Any material changes to this Privacy Policy will be effective upon the earlier of thirty (30) calendar days following our dispatch of an e-mail notice to you or thirty (30) calendar days following our posting of notice of the changes on our site. These changes will be effective immediately for new users of our website and Products. Please note that at all times you are responsible for updating your Personal Data to provide us with your most current e-mail address. In the event that the last e-mail address that you have provided us is not valid, or for any reason is not capable of delivering to you the notice described above, our dispatch of the e-mail containing such notice will nonetheless constitute effective notice of the changes described in the notice. In any event, changes to this Privacy Policy may affect our use of Personal Data that you provided us prior to our notification to you of the changes. If you do not wish to permit changes in our use of your Personal Data, you must notify us prior to the effective date of the changes that you wish to deactivate your account with us. Continued use of our website or Products, following notice of such changes shall indicate your acknowledgement of such changes and agreement to be bound by the terms and conditions of such changes.

12. Transfers of Personal Data outside the EEA.

Your personal information may be transferred and stored in countries outside the EEA that are subject to different standards of data protection. Superstream takes appropriate steps to ensure that transfers of personal information are in accordance with applicable law and carefully managed to protect your privacy rights and interests, including through the use of standard contractual clauses and our certifications to the EU-US Data Privacy Framework (“DPF”), the UK Extension to the EU-US DPF, and the Swiss-US DPF. Additionally, Superstream uses a limited number of third-party service providers to assist us in providing our services to customers. These third parties may access, process, or store Personal Data in the course of providing their services. Superstream obtains contractual commitments from them to protect your Personal Data.

13. Accessibility.

We are committed to making our products and services accessible to everyone. If you need help with your accessibility-related requests and other servicing needs please contact us at support@superstream.ai.

Superstream achieves cost efficiency, enhanced performance, and greater reliability through continuous optimization of both application and infrastructure layers.

• Application layer: Identify and eliminate inactive resources, reduce traffic, apply improved configurations, enforce standardization, identify storage reduction opportunities, and more.

• Infrastructure layer: Clusters are often sized for peak times, but Superstream dynamically identifies the exact resources needed at any given moment and automatically scales accordingly.

Reducing costs

Your bill structure

Kafka's Total Cost of Ownership (=TCO) includes:

• Compute (Number of instances X type)

• Egress (Read) in GB/hour

• Storage in GB/hour

Step 1: Optimizing the application layer (clusters/clients)

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions. Decreasing the number of partitions has a direct effect on Memory utilization and the number of needed brokers (instances)

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Step 2: Dynamically scale the cluster's resources

Superstream leverages advanced machine learning algorithms to anticipate resource demands based on historical data and current usage patterns. By continuously analyzing this information, Superstream adjusts cluster sizes in real-time, ensuring that resources are allocated precisely when and where they are needed. This dynamic scaling guarantees optimal performance and significantly reduces operational costs by avoiding the need to provision resources for peak load conditions that may only occur sporadically.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Improve performance

Where are the bottlenecks in Apache Kafka?

• Untuned filesystem and operating system

• Data Transfer: Limited network bandwidth can restrict the amount of data that can be transferred, especially in high-throughput scenarios.

• Producer Latency: High producer latency can occur due to batching, request sizes, or insufficient resources.

• Consumer Lag: Consumer lag occurs when consumers cannot keep up with the rate of messages produced.

• Large Number of Partitions: Managing a large number of partitions can strain broker resources and increase metadata overhead.

• Uneven Partition Distribution: Uneven partition distribution can lead to hotspots, where some brokers handle more load than others.

• Replication Lag: Replication lag occurs when follower replicas cannot keep up with the leader, leading to data inconsistencies.

• Acknowledgment Settings: Configuring acks=all can increase latency as the producer waits for acknowledgments from all replicas.

Here is how Superstream mitigates these issues

1. Optimize kernel-level parameters and the filesystem (both XFS and EXT4) to best suit Kafka workloads.

2. Ensure compression is enabled and enforce otherwise to reduce data size and optimize network usage.

3. Optimize producer configurations, such as batch size and linger.ms. Ensure producers have adequate CPU and memory resources.

4. Optimize consumer configurations, such as fetch size and max.poll.records. Scale consumer groups and partitions to distribute the load more effectively.

5. Optimize the number of partitions per topic based on throughput requirements.

6. Suggests improved key and partition strategies.

7. Ensure sufficient resources for follower replicas. Optimize replication settings, such as min.insync.replicas and replication.factor.

8. Adjust the ack parameter as needed.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactive Measures: Enforce strict governance over clients to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactive Measures: Acknowledge that "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. High-availability in case of a failure.

Here is how Superstream can help

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user is still able to commit a violation, Superstream will identify it in its continuous scanning and mitigate it automatically or manually.

3. High-availability: While you can use MirrorMaker 2.0 or Cluster linking in the case of Confluent, the solution is not hermetic. Here is why:

   1. You would still need to fix offset translations

   2. Clients will need to reroute to the DR cluster

   3. You would need to flush the clients' cache or build an internal mechanism to sync the leftover offsets directly in the DR cluster

   4. You can't failback

   It is true that the need to failover is not common, but what happens if it does happen? Can your organization or product sustain that? The potential RPO and RTO is high. Can your reputation sustain it? Can you bear the data loss? We can do better. With SuperShield, your RPO and RTO would be as minimal as possible, and the entire process would take place completely automatically. So you would be able to sleep better, your customers would trust you more, and your business would continue to work as usual.

Superstream achieves cost efficiency, enhanced performance, and greater reliability through continuous optimization of both application and infrastructure layers.

• Application layer: Identify and eliminate inactive resources, reduce traffic, apply improved configurations, enforce standardization, identify storage reduction opportunities, and more.

• Infrastructure layer: Clusters are often sized for peak times, but Superstream dynamically identifies the exact resources needed at any given moment and automatically scales accordingly.

Reducing costs in AWS MSK

Your bill structure

AWS MSK Total Cost of Ownership (=TCO) includes:

• Compute (Number of instances X type)

• Egress (Read) in GB/hour

• Storage in GB/hour

Step 1: Optimizing the application layer (clusters/clients)

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions. Decreasing the number of partitions has a direct effect on Memory utilization and the number of needed brokers (instances)

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Step 2: Dynamically scale the cluster's resources

Superstream leverages advanced machine learning algorithms to anticipate resource demands based on historical data and current usage patterns. By continuously analyzing this information, Superstream adjusts cluster sizes in real-time, ensuring that resources are allocated precisely when and where they are needed. This dynamic scaling guarantees optimal performance and significantly reduces operational costs by avoiding the need to provision resources for peak load conditions that may only occur sporadically.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Improve performance

Where are the bottlenecks in Apache Kafka?

• Untuned filesystem and operating system

• Data Transfer: Limited network bandwidth can restrict the amount of data that can be transferred, especially in high-throughput scenarios.

• Producer Latency: High producer latency can occur due to batching, request sizes, or insufficient resources.

• Consumer Lag: Consumer lag occurs when consumers cannot keep up with the rate of messages produced.

• Large Number of Partitions: Managing a large number of partitions can strain broker resources and increase metadata overhead.

• Uneven Partition Distribution: Uneven partition distribution can lead to hotspots, where some brokers handle more load than others.

• Replication Lag: Replication lag occurs when follower replicas cannot keep up with the leader, leading to data inconsistencies.

• Acknowledgment Settings: Configuring acks=all can increase latency as the producer waits for acknowledgments from all replicas.

Here is how Superstream mitigates these issues

1. Optimize kernel-level parameters and the filesystem (both XFS and EXT4) to best suit Kafka workloads.

2. Ensure compression is enabled and enforce otherwise to reduce data size and optimize network usage.

3. Optimize producer configurations, such as batch size and linger.ms. Ensure producers have adequate CPU and memory resources.

4. Optimize consumer configurations, such as fetch size and max.poll.records. Scale consumer groups and partitions to distribute the load more effectively.

5. Optimize the number of partitions per topic based on throughput requirements.

6. Suggests improved key and partition strategies.

7. Ensure sufficient resources for follower replicas. Optimize replication settings, such as min.insync.replicas and replication.factor.

8. Adjust the ack parameter as needed.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactive Measures: Enforce strict governance over clients to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactive Measures: Acknowledge that "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. High-availability in case of a failure.

Here is how Superstream can help

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user is still able to commit a violation, Superstream will identify it in its continuous scanning and mitigate it automatically or manually.

3. High-availability: While you can use MirrorMaker 2.0 or Cluster linking in the case of Confluent, the solution is not hermetic. Here is why:

   1. You would still need to fix offset translations

   2. Clients will need to reroute to the DR cluster

   3. You would need to flush the clients' cache or build an internal mechanism to sync the leftover offsets directly in the DR cluster

   4. You can't failback

   It is true that the need to failover is not common, but what happens if it does happen? Can your organization or product sustain that? The potential RPO and RTO is high. Can your reputation sustain it? Can you bear the data loss? We can do better. With SuperShield, your RPO and RTO would be as minimal as possible, and the entire process would take place completely automatically. So you would be able to sleep better, your customers would trust you more, and your business would continue to work as usual.

Superstream enhances the experience of Confluent Cloud users in numerous ways, including significant cost reduction and improved reliability.

By optimizing resource usage and streamlining operations, Superstream ensures that users save money and achieve smoother, more efficient, and more secure streaming infrastructure. This comprehensive support helps Confluent Cloud users maximize their investment and maintain a high standard of operational excellence.

Reducing costs

Your bill structure

• E-CKU: Represents the maximum number of partitions and connections an account can reach.

• Ingress (Write) in GB/hour

• Egress (Read) in GB/hour

• Storage in GB/hour

Whether you are using a dedicated or an enterprise cluster, cost reduction is not measured by decreasing your current spending only but also by making sure you are not increasing it.

How Superstream Can Help Reduce Costs

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions.

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive client connections, optimizing connection usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Reduce unused schemas

5. Transfer reduction: Superstream will automatically identify the most optimal compression and batch.size per each type of workload/topic

6. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The identification and remediation process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

A Faster Catch and Respond to Issues

1. Anomaly Detection: Know Before it Breaks Superstream’s AI-powered anomaly detection continuously monitors your Kafka clusters, automatically identifying irregular patterns in message rates, consumer lag, partition imbalance, and system bottlenecks. Instead of relying on static thresholds, our system adapts to your workload, highlighting potential failures before they cause downtime.

2. Troubleshoot with Insights, Not Charts Traditional Kafka monitoring tools flood you with dashboards and metrics—but when something goes wrong, you need answers, not just data. Superstream translates complex Kafka behavior into clear, actionable insights. Instead of digging through charts and logs, you get direct recommendations on what's wrong, why it happened, and how to fix it. This means faster resolution times and fewer headaches for your team.

3. Kafka360: A Complete Picture of Your Streaming Infra When troubleshooting, context is key. Kafka360 provides a holistic view of your Kafka ecosystem, bringing together producers, consumers, brokers, and performance metrics into a single, intuitive interface. This lets you trace issues across the entire data flow, pinpoint root causes, and optimize performance—all in real time.

Superstream is an autonomous platform designed to optimize and improve Kafka of all types to reduce costs, reduce operations, and increase reliability.

Superstream enhances the experience of Redpanda Cloud users in numerous ways, including significant cost reduction and increased visibility.

By optimizing resource usage, Superstream ensures that Redpanda users not only save money but also achieve a more reliable streaming environment with deep observability and client-level insights.

Reducing costs

Billed components

• Uptime

• Ingress (Write)

• Egress (Read)

• Storage in GB/hour

• Partitions per hour

How Superstream Can Help Reduce Costs

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions.

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Utilize Producer-level Compression: Superstream finds producers that are not using compression, determines their capability to enable it, its outcome, and surface who they are

5. Identify your heavy requester: Quickly identify how many requests each client performs and ways to reduce that.

6. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactiveness: Enforce strict configuration governance over clients and created resources to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactiveness: Acknowledge that configuration "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. Anomaly detection: Identify issues and their potential root cause before they reach your customers and applications

Superstream can help establish all three:

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user can still commit a violation, Superstream will identify it in its continuous scanning and quickly notify the operators.

3. Anomaly detection: Superstream will continuously learn your workloads and patterns, and alert when there is significant change in traditional patterns across clients and clusters.

Superstream enhances the experience of AWS MSK Serverless users in numerous ways, including significant cost reduction, improved performance, increased reliability, and robust governance.

By optimizing resource usage and streamlining operations, Superstream ensures that users save money and achieve smoother, more efficient, and more secure data management. This comprehensive support helps AWS MSK Serverless users maximize their investment and maintain a high standard of operational excellence.

Reducing costs

Your bill structure

• Cluster-hours

• Partition-hours

• Storage (Per GB-month)

• Data in

• Data out

How Superstream Can Help Reduce Costs

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions.

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Improve performance

Where are the bottlenecks in AWS MSK Kafka?

• Data Transfer: Limited network bandwidth can restrict the amount of data that can be transferred, especially in high-throughput scenarios.

• Producer Latency: High producer latency can occur due to batching, request sizes, or insufficient resources.

• Consumer Lag: Consumer lag occurs when consumers cannot keep up with the rate of messages produced.

• Large Number of Partitions: Managing a large number of partitions can strain broker resources and increase metadata overhead.

• Uneven Partition Distribution: Uneven partition distribution can lead to hotspots, where some brokers handle more load than others.

• Replication Lag: Replication lag occurs when follower replicas cannot keep up with the leader, leading to data inconsistencies.

• Acknowledgment Settings: Configuring acks=all can increase latency as the producer waits for acknowledgments from all replicas.

Here is how Superstream mitigates these issues

1. Ensure compression is enabled and enforce otherwise to reduce data size and optimize network usage.

2. Optimize producer configurations, such as batch size and linger.ms. Ensure producers have adequate CPU and memory resources.

3. Optimize consumer configurations, such as fetch size and max.poll.records. Scale consumer groups and partitions to distribute the load more effectively.

4. Optimize the number of partitions per topic based on throughput requirements.

5. Suggests improved key and partition strategies.

6. Ensure sufficient resources for follower replicas. Optimize replication settings, such as min.insync.replicas and replication.factor.

7. Adjust the ack parameter as needed.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactive Measures: Enforce strict governance over clients to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactive Measures: Acknowledge that "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. High-availability in case of a failure.

Here is how Superstream can help

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user is still able to commit a violation, Superstream will identify it in its continuous scanning and mitigate it automatically or manually.

3. High-availability: While you can use MirrorMaker 2.0, the solution is not hermetic. Here is why:

   1. You would still need to fix offset translations

   2. Clients will need to reroute to the DR cluster

   3. You would need to flush the clients' cache or build an internal mechanism to sync the leftover offsets directly in the DR cluster

   With the current version of MirrorMaker 2.0, Superstream complements it by addressing the challenges of failover situations with an extreme focus on clients.

V1.1.100

We've been busy making things better, wiser, and faster.

Latest updates:

• Autoscaler improvements: -force flag to force MSK clusters to scale down even when partition limitation is blocking the operation

• Cluster auto-discovery improvements

• New cluster cost report on each cluster's page

• Cluster summary usage update

• Warning indication for connected clusters in case some information is missing

• Users management

Recent fixes:

• Autoscaler issues

• UI fixes

• Cluster information

Thanks for using Superstream!

V1.0.600

We've been busy making things better, smarter, and faster.

Latest updates:

• A new auto scaler for Aiven and AWS MSK!

• An ability to remediate optimizations automatically!

• Client properties modifications at runtime

Recent fixes:

• Setting different enforced retention policies

• UI fixes

• Missing consumer params

Thanks for using Superstream!

Superstream enhances the experience of Aiven Kafka users in numerous ways, including significant cost reduction, improved performance, increased reliability, and robust governance.

By optimizing resource usage and streamlining operations, Superstream ensures that users save money and achieve smoother, more efficient, and more secure data management. This comprehensive support helps AWS MSK Serverless users maximize their investment and maintain a high standard of operational excellence.

Reducing costs

Your bill structure

• A selected plan is defined by the number of VMs (brokers), CPU, Memory, and Storage. The bigger you go, the more you pay.

To reduce costs, we need to -> reduce the cluster utilization -> decrease the selected plan.

Here is how Superstream Can Help to decrease the cluster resources utilization

1. Identify and Reduce Inactive Topics and Partitions: Superstream employs proprietary methodologies to continuously detect and minimize inactive topics and partitions.

2. Identify and Reduce Inactive Connections: Superstream constantly monitors and reduces inactive connections, optimizing resource usage.

3. Identify and Reduce Inactive Consumer Groups: Superstream continuously tracks and diminishes inactive consumer groups to ensure efficient system performance.

4. Optimize Producers without Compression: Superstream assesses producers not using compression, determines their capability to enable it, and seamlessly activates the most suitable compression algorithm.

5. Optimize Non-Binary Payloads: Superstream identifies producers writing non-binary payloads and activates payload reduction techniques, ensuring no disruption to the existing workflow.

Each optimization listed greatly impacts your existing CPU/Memory/Storage utilization, which may force you to select a larger plan than necessary.

The entire identification process is fully configurable to accommodate various business logic while continuously self-improving through reinforcement learning.

Although certain environments may not support all actions, implementing even a small portion can significantly reduce the monthly bill.

Improve performance

Where are the bottlenecks in (any) Kafka?

• Data Transfer: Limited network bandwidth can restrict the amount of data that can be transferred, especially in high-throughput scenarios.

• Producer Latency: High producer latency can occur due to batching, request sizes, or insufficient resources.

• Consumer Lag: Consumer lag occurs when consumers cannot keep up with the rate of messages produced.

• Large Number of Partitions: Managing a large number of partitions can strain broker resources and increase metadata overhead.

• Uneven Partition Distribution: Uneven partition distribution can lead to hotspots, where some brokers handle more load than others.

• Replication Lag: Replication lag occurs when follower replicas cannot keep up with the leader, leading to data inconsistencies.

• Acknowledgment Settings: Configuring acks=all can increase latency as the producer waits for acknowledgments from all replicas.

Here is how Superstream mitigates these issues

1. Ensure compression is enabled and enforce otherwise to reduce data size and optimize network usage.

2. Optimize producer configurations, such as batch size and linger.ms. Ensure producers have adequate CPU and memory resources.

3. Optimize consumer configurations, such as fetch size and max.poll.records. Scale consumer groups and partitions to distribute the load more effectively.

4. Optimize the number of partitions per topic based on throughput requirements.

5. Suggests improved key and partition strategies.

6. Ensure sufficient resources for follower replicas. Optimize replication settings, such as min.insync.replicas and replication.factor.

7. Adjust the ack parameter as needed.

increase reliability

Achieving a high standard of reliability in Kafka involves three key components:

1. Proactive Measures: Enforce strict governance over clients to prevent abusive usage and apply necessary limitations to ensure optimal performance.

2. Reactive Measures: Acknowledge that "drifts" will occur and implement an additional layer of protection to address issues as they arise.

3. High-availability in case of a failure.

Here is how Superstream can help

1. Proactive Measures: Enforce client and cluster policies based on business groups, tags, or teams that determine what can and cannot be done across the different layers in terms of both configuration and client parameters.

2. Reactive Measures: If a client or user is still able to commit a violation, Superstream will identify it in its continuous scanning and mitigate it automatically or manually.

3. High-availability: While you can use MirrorMaker 2.0, the solution is not hermetic. Here is why:

   1. You would still need to fix offset translations

   2. Clients will need to reroute to the DR cluster

   3. You would need to flush the clients' cache or build an internal mechanism to sync the leftover offsets directly in the DR cluster

   With the current version of MirrorMaker 2.0, Superstream complements it by addressing the challenges of failover situations with an extreme focus on clients.

V1.0.700

We've been busy making things better, smarter, and faster.

Latest updates:

• Brand new UI!

• Superstream will now automatically discover clusters from a vendor API key

• Charts and other impact-related information such as savings, have been refactored and calculations were tuned

Recent fixes:

• Autoscaler issues

• UI fixes

Thanks for using Superstream!

V1.0.500

We've been busy making things better, smarter, and faster.

Latest updates:

• Topic protection's user experience got improved

• The clients' tab was improved and enriched

• Enhancements to the notifications center

• Support in retrying all the jobs

• New Java client library: 3.5.116

Recent fixes:

• Empty owner in different audit logs

• UI fixes

• Issue when trying to add an ARN to an already connected MSK cluster

• Enabled compression log

Thanks for using Superstream!

V1.0.800

We've been busy making things better, wiser, and faster.

Latest updates:

• Autoscaler improvements

• Cluster auto-discovery improvements

• Algorithms and calculations have been tuned

• The Todo list has been removed. Tasks cannot be fixed manually or individually but rather through automation only

• New cluster page

• Users management - backend only.

Recent fixes:

• Autoscaler issues

• UI fixes

Thanks for using Superstream!

V1.0.400

We've been busy making things better, smarter, and faster.

Latest updates:

• UI performance improvements

• Redpanda support is now generally available

• Auditing page improvements

• All client properties per client are collected and displayed in the UI

• Ability to perform bulk fixes

• System notifications center

• Python library is now generally available: 2.4.0

• New Java client library: 3.5.114

Recent fixes:

• Analysis algorithm improvements

Thanks for using Superstream!

August 22, 2024 —

V1.0.300

We've been busy making things better, smarter, and faster.

Latest updates:

• All client properties are now being collected and can be viewed. This feature will be useful for the next release, as it allows users to modify them manually or via Superstream suggestions. Note: A client library upgrade is required.

• Dashboard performance improvements and additional metrics

• New optimization: Check for uncompliant retention policies

• Auditing improvements

• New Java client library: 3.5.113

Recent fixes:

• Minor engine issues

• Error while switching compression algorithm

Thanks for using Superstream!

V1.0.200

We've been busy making things better, wiser, and faster.

Latest updates:

• UI improvements: the ability to pin clusters, enhanced visibility at the all clusters page

• Manual / Automatic mode: Ability to set Superstream automation to manual for validation before automatic execution

• Notifications improvements

Recent fixes:

• Autoscaler issues

• UI fixes

Thanks for using Superstream!