Step 4: Connect Your Clients

This page describes how to connect your Kafka clients to Superstream

Superstream client is a supplement for your existing Kafka library.

Key features

  • Identical to your existing Kafka SDKs

  • Easy to replace

  • You control your Kafka Clients: batch, linger, compression, data format

  • Enforce governance

  • Runtime client configuration updates

  • Transparent client migration between topics and clusters

  • Zero additional computing resources are needed

Getting started

Step 1: Set environment variable

Please set the following environment variables for each client

### SUPERSTREAM_HOST - REQUIRED!
Specify the host URL of the SuperStream service to connect to the appropriate Superstream environment.

### SUPERSTREAM_LEARNING_FACTOR - Default 20
This variable adjusts the learning algorithms used within the SDK to optimize data streaming performance.

### SUPERSTREAM_REDUCTION_ENABLED - Default false
Set this variable to enable data reduction features.

### SUPERSTREAM_TAGS - Default is an empty string
Set this variable to tag the client, this is a string - comma-separated list of tags.

Step 2: Library replacement

Within each lib repo made by Superstream, under the "Releases" section, you will find the equivalent version to the existing "stable" library according to the major release.

For example, if you are currently using Apache Kafka 3.1.0, then you would use the latest release of 3.x.x by Superstream.

Appendixes

Appendix A - Securing Client Connections

To enable secured client connections to the Superstream Engine, ensure that skipLocalAuthentication is turned off. Detailed instructions can be found here.

For clients with enabled authentication, set the following environment variable for each client in addition:

### SUPERSTREAM_TOKEN - REQUIRED!
This authentication token is required to access the Superstream services securely.

Appendix B - Setting Up a Maven Repository and Managing Artifacts with JFrog Artifactory

Prerequisites

  1. Access to a JFrog Artifactory instance.

  2. GitHub account and access to a repository with releases.

Create a New Maven Repository

  1. Log in to your JFrog Artifactory.

  2. Navigate to Admin > Repositories > Repositories.

  3. Click on 'Add Repositories' and select 'Local Repository'.

  4. Choose 'Maven' as the package type.

  5. Enter the repository key and other configuration details.

  6. Save the repository settings.

Download the artifact from GitHub:

https://github.com/superstreamlabs/apache-kafka-java-sdk/releases

Find your equivalent release

  1. Navigate to the 'Releases' section of the desired GitHub repository.

  2. Select the release that contains the needed kafka-clients-yourversion.tar.gz file.

Download the artifact

  1. Right-click on the kafka-clients-yourversion.tar.gz link and copy the URL.

  2. Use curl or wget to download the file from the terminal.

Deploy via the JFrog UI

  1. Navigate to the Artifactory repository where you want to deploy the artifact.

  2. Click on 'Deploy' and select the kafka-clients-yourversion.tar.gz file from your local system.

  3. Specify the target path within the repository if required.

  4. Click 'Deploy' to upload the artifact.

Knowledge Base

Q: I am using Kafka Connect. How can I ensure Superstream identifies it when declaring a topic inactive?

A: To enable Superstream to detect a source or sink that is activated for a specific period and then removed, the connector must be configured with a group.id. This configuration will "leave a trace" in the form of a zombie consumer group. You can find an example of this from Confluent here.

Last updated

Logo

Copyright to Superstream.ai