Improve Your Producers Efficiency With This Powerful Calculator
Go
LogoLogo
WebsiteStart FreeSuperstream Console
  • Superstream
  • Getting started
    • Option 1: BYOC
      • Step 1: Agent Deployment
      • Step 2: Create a Kafka user
      • Step 3: Connect your Kafka cluster/s
      • Step 4: Activate
      • Additional resources
        • Engine deployment
          • Supertstream Engine Deployment using existing secrets
          • How to deploy and manage Superstream using ArgoCD
          • Superstream Engine deployment for environments with a local container registry
          • Supertstream Engine Deployment using custom resource limits
          • Superstream Platform deployment for Air-Gapped environments
        • Engine upgrade
          • Upgrading From Helmfile based Deployment to Superstream Helm Chart
    • Option 2: Fully managed
      • Step 1: Create a Kafka user
      • Step 2: Connect your Kafka cluster/s
      • Step 3: Activate
  • Optimizations
    • Improve Network Efficiency
    • Resize AWS MSK Size
  • Settings
    • Notifications
  • Security & Legal
    • Processed data
    • Compliance
    • Firewall rules
    • Authentication
    • Legal
      • Terms and Conditions
      • Privacy Policy
  • Solution briefs
    • Superstream for Confluent Platform
    • Superstream for Confluent Cloud
    • Superstream for Redpanda
    • Superstream for Apache Kafka
    • Superstream for AWS MSK
    • Superstream for AWS MSK Serverless
    • Superstream for Aiven
  • Release notes
    • Changelog
      • Feb 2, 2025
      • Jan 11, 2025
      • Dec 5, 2024
      • Nov 1, 2024
      • Oct 14, 2024
      • Sep 24, 2024
      • Sep 10, 2024
      • August 26, 2024
Powered by GitBook
LogoLogo

Product

  • Superstream Console

Copyright to Superstream Labs Inc. 2025

On this page
  • Superstream Client For Java
  • Overview
  • Supported Libraries
  • Features
  • Installation
  • Prerequisites
  • License

Was this helpful?

Edit on GitHub
Export as PDF
  1. Optimizations

Improve Network Efficiency

Apache Kafka is a high-throughput distributed messaging system that powers many real-time applications. However, Kafka's performance can often be constrained by inefficient network usage—especially in high-throughput or multi-region deployments. Improving Kafka’s network efficiency means optimizing how data flows between clients and brokers, reducing bandwidth usage, minimizing latency, and ultimately ensuring cost-effective and reliable data pipelines.

At Superstream, we can make it easier to manage and optimize Kafka networking, particularly through our open-source superstream-clients-java library. This guide walks through how to use the library to boost network efficiency when interacting with Kafka.


Superstream Client For Java

A Java library for automatically optimizing Kafka producer configurations based on topic-specific recommendations.

Overview

Superstream Clients works as a Java agent that intercepts Kafka producer creation and applies optimized configurations without requiring any code changes in your application. It dynamically retrieves optimization recommendations from Superstream and applies them based on impact analysis.

Supported Libraries

Works with any Java library that depends on kafka-clients, including:

  • Apache Kafka Clients

  • Spring Kafka

  • Alpakka Kafka (Akka Kafka)

  • Kafka Streams

  • Kafka Connect

  • Any custom wrapper around the Kafka Java client

Features

  • Zero-code integration: No code changes required in your application

  • Dynamic configuration: Applies optimized settings based on topic-specific recommendations

  • Intelligent optimization: Identifies the most impactful topics to optimize

  • Graceful fallback: Falls back to default settings if optimization fails

Installation

Step 1: Add Superstream package

Maven

<dependency>
    <groupId>ai.superstream</groupId>
    <artifactId>superstream-clients</artifactId>
    <version>1.0.0</version>
</dependency>

Gradle

implementation group: 'ai.superstream', name: 'superstream-clients', version: '1.0.0-beta'

Step 2: Add Environment Variables

Required Environment Variables

  • SUPERSTREAM_TOPICS_LIST: Comma-separated list of topics your application produces to

Optional Environment Variables

  • SUPERSTREAM_LATENCY_SENSITIVE: Set to "true" to prevent any modification to linger.ms values

  • SUPERSTREAM_DISABLED: Set to "true" to disable optimization

Example:

export SUPERSTREAM_TOPICS_LIST=orders,payments,user-events
export SUPERSTREAM_LATENCY_SENSITIVE=true

Step 3: Run

Add the Java agent to your application's startup command:

java -javaagent:/path/to/superstream-clients-1.0.0.jar -jar your-application.jar

Common example:

java -javaagent:$MAVEN_REPOSITORY$/ai/superstream/superstream-clients/1.0.0/superstream-clients-java-1.0.0.jar -jar your-application.jar

Docker Integration

When using Superstream Clients with containerized applications, include the agent in your Dockerfile:

FROM openjdk:11-jre

# Copy your application
COPY target/your-application.jar /app/your-application.jar

# Copy the Superstream agent
COPY path/to/superstream-clients-1.0.0.jar /app/lib/superstream-clients-1.0.0.jar

# Set environment variables
ENV SUPERSTREAM_TOPICS_LIST=your-topics

# Run with the Java agent
ENTRYPOINT ["java", "-javaagent:/app/lib/superstream-clients-1.0.0.jar", "-jar", "/app/your-application.jar"]

Alternatively, you can use a multi-stage build to download the agent from Maven Central:

# Build stage
FROM maven:3.8-openjdk-11 AS build

# Get the Superstream agent
RUN mvn org.apache.maven.plugins:maven-dependency-plugin:3.2.0:get \
-DgroupId=ai.superstream \
-DartifactId=superstream-clients \
-Dversion=1.0.0

RUN mvn org.apache.maven.plugins:maven-dependency-plugin:3.2.0:copy \
-Dartifact=ai.superstream:superstream-clients:1.0.0 \
-DoutputDirectory=/tmp

# Final stage
FROM openjdk:11-jre

# Copy your application
COPY target/your-application.jar /app/your-application.jar

# Copy the agent from the build stage
COPY --from=build /tmp/superstream-clients-1.0.0.jar /app/lib/superstream-clients-1.0.0.jar

SUPERSTREAM_LATENCY_SENSITIVE Explained

The linger.ms parameter follows these rules:

  1. If SUPERSTREAM_LATENCY_SENSITIVE is set to true:

    • Linger value will never be modified, regardless of other settings

  2. If SUPERSTREAM_LATENCY_SENSITIVE is set to false or not set:

    • If no explicit linger exists in original configuration: Use Superstream's optimized value

    • If explicit linger exists: Use the maximum of original value and Superstream's optimized value

Prerequisites

  • Java 11 or higher

  • Kafka cluster that is connected to the Superstream's console

  • Read and write permissions to the superstream.* topics

License

This project is licensed under the Apache License 2.0.

PreviousStep 3: ActivateNextResize AWS MSK Size

Last updated 21 days ago

Was this helpful?

Always use the

Always use the

latest version
latest version