Getting started - Python

Apache Kafka is a high-throughput distributed messaging system that powers many real-time applications. However, Kafka's performance can often be constrained by inefficient network usage—especially in high-throughput or multi-region deployments. Improving Kafka’s network efficiency means optimizing how data flows between clients and brokers, reducing bandwidth usage, minimizing latency, and ultimately ensuring cost-effective and reliable data pipelines.

At Superstream, we can make it easier to manage and optimize Kafka networking, particularly through our open-source superstream-clients library. This guide walks through how to use the library to boost network efficiency when interacting with Kafka.

---

Superstream Client For Python

A Python library for automatically optimizing Kafka producer configurations based on topic-specific recommendations.

Overview

Superstream Clients works as a Python import hook that intercepts Kafka producer creation and applies optimized configurations without requiring any code changes in your application. It dynamically retrieves optimization recommendations from Superstream and applies them based on impact analysis.

Supported Libraries

Works with any Java library that depends on kafka-clients, including:

• kafka-python

• aiokafka

• confluent-kafka

• Faust

• FastAPI event publishers

• Celery Kafka backends

• Any custom wrapper around these Kafka clients

Features

• Zero-code integration: No code changes required in your application

• Dynamic configuration: Applies optimized settings based on topic-specific recommendations

• Intelligent optimization: Identifies the most impactful topics to optimize

• Graceful fallback: Falls back to default settings if optimization fails

---

Installation

Superstream package: https://pypi.org/project/superstream-clients

Step 0: Add permissions

Any app that runs Superstream lib should be able to READ/WRITE/DESCRIBE from all topics with the prefix superstream.*

Step 1: Install the Superstream lib

pip install superstream-clients && python -m superclient install_pth

Step 2: Add Environment Variables

ENV	Required?	Description	Example
SUPERSTREAM_TOPICS_LIST	Yes	Comma-separated list of topics your application produces to	SUPERSTREAM_TOPICS_LIST=orders,payments,user-events
SUPERSTREAM_LATENCY_SENSITIVE=false	No	Set to true to prevent any modification to linger.ms values	SUPERSTREAM_LATENCY_SENSITIVE=true
SUPERSTREAM_DISABLED=false	No	Set to true to disable optimization	SUPERSTREAM_DISABLED=true

That's it! Superclient will now automatically load and optimize all Kafka producers in your Python environment.

After installation, SuperClient works automatically. Just use your Kafka clients as usual.

Optional. Docker Integration

When using Superstream Clients with containerized applications, include the package in your Dockerfile:

FROM python:3.8-slim

# Install superclient
RUN pip install superstream-clients
RUN python -m superclient install_pth

# Your application code
COPY . /app
WORKDIR /app

# Run your application
CMD ["python", "your_app.py"]

Prerequisites

• Python 3.8 or higher

• Kafka cluster that is connected to the Superstream's console

• Read and write permissions to the superstream.* topics