What are the shard limits for Amazon Kinesis Data Streams?

Each shard can handle a maximum of 1,000 records per second or 1MB per second. Understanding these limits is essential for designing partition keys and determining how many shards your stream needs.

How does the smart partition key function work?

The function hashes a user ID (if present in the data) modulo 1000 to generate an even distribution across shards, or falls back to hashing the entire data record if no user ID exists. This ensures balanced shard utilization and prevents hot partitions.

What is the difference between Lambda and KCL consumers in this skill?

Lambda consumers decode base64 payloads and process records independently, routing failures to a dead-letter queue. KCL consumers implement `RecordProcessorBase`, batch records per shard, track sequence numbers, and checkpoint on a time interval rather than per record to avoid reprocessing on errors.

When should you use Kinesis Data Firehose instead of building a custom consumer?

Use Kinesis Data Firehose for simple batch ETL workflows where you don't need custom consumer code. This skill is better suited for scenarios requiring high-throughput producers, low-latency consumption, or complex event processing logic.

What monitoring indicators does this skill recommend for detecting stream lag?

Iterator age is the primary lag indicator; CloudWatch alarms should monitor both high incoming record volume and high iterator age to detect when consumers are falling behind the producer.

Skill

Process Real-Time Data Streams with Kinesis

Name: Kinesis Stream Processor
Availability: OnlineOnly
Author: VibeBaza

A Kinesis stream processing expert that builds scalable producers, Lambda/KCL consumers, and CloudFormation-configured stream monitoring.

Get skill

Works with aws kinesisboto3cloudwatchgithubaws lambda

VibeBaza

Own this? Claim it

Spark score

out of 100

Updated 11 days ago

Version 1.0.0

Models

claude

Add to Favorites

Why it matters

Build scalable, fault-tolerant real-time data processing applications using Amazon Kinesis. This asset specializes in stream processing fundamentals, performance optimization, and robust error handling for Kinesis Data Streams.

Outcomes

What it gets done

Implement partition key strategies for even data distribution.

Configure Kinesis streams and CloudWatch alarms using CloudFormation.

Develop high-performance Kinesis producers and consumers (Lambda/KCL).

Optimize record processing for throughput and cost efficiency.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/vb-kinesis-stream-processor | bash

Overview

Kinesis Stream Processor

A Kinesis stream processing expert that builds high-throughput producers, Lambda and KCL consumers with checkpointing and DLQ handling, and CloudFormation-provisioned stream monitoring. Use it to build or harden a Kinesis streaming pipeline - producer, consumer, and monitoring - rather than reaching for Kinesis on workloads that fit simpler managed ETL.

What it does

Builds scalable, fault-tolerant real-time processing applications on Amazon Kinesis Data Streams, Kinesis Analytics, and the Kinesis Client Library (KCL). Fundamentals cover partition-key design for even shard distribution, working within shard limits of 1,000 records/sec or 1MB/sec per shard, checkpoint management for exactly-once or at-least-once processing, retry mechanisms with a dead-letter queue for failed records, and CloudWatch metrics alongside custom business metrics. It implements a high-throughput producer (buffered batching - 100ms buffer time, 500-record batches, 24 max connections - plus a hash-based partition-key function that distributes by user_id or falls back to a random key), two consumer patterns - a Lambda handler that decodes base64 Kinesis payloads, validates required fields, batches successes downstream, and routes failures to a DLQ; and a KCL RecordProcessorBase implementation that batches records, tracks the largest sequence number processed, and checkpoints on a time interval (default every 60 seconds) rather than per record - and a CloudFormation template for a production stream: a 10-shard stream with 7-day retention and KMS encryption, plus CloudWatch alarms on incoming-record volume and iterator age. An enhanced fan-out helper registers or looks up a stream consumer for sub-200ms-latency delivery.

When to use - and when NOT to

Use it to build or harden a Kinesis-based streaming pipeline - a high-throughput producer, a Lambda or KCL consumer with proper checkpointing and error handling, or the CloudFormation and monitoring setup around a production stream - rather than reaching for Kinesis on workloads that fit a simpler managed ETL path like Kinesis Data Firehose.

Inputs and outputs

Input is the event stream to produce into or consume from, plus its throughput and latency requirements. Output is a Kinesis producer with a partition-key strategy, a Lambda or KCL consumer with checkpointing and DLQ handling, a CloudFormation stack for the stream and its CloudWatch alarms, and, where sub-200ms delivery is needed, an enhanced fan-out consumer registration.

Integrations

Built on boto3 and the Kinesis Producer Library for producers, AWS Lambda or the Kinesis Client Library (amazon_kclpy) for consumers, CloudFormation for provisioning (AWS::Kinesis::Stream, AWS::CloudWatch::Alarm), and CloudWatch for IncomingRecords/IteratorAgeMilliseconds monitoring - with Kinesis Data Firehose and Kinesis Analytics named as alternatives for simpler ETL or SQL-based processing.

Who it's for

For teams building production Kinesis pipelines who need reliability and cost discipline built in, not bolted on after an incident. It covers cost optimization (using Firehose instead of a custom consumer for simple ETL, record aggregation to cut per-record cost, reserving enhanced fan-out for genuinely latency-sensitive paths, shard-utilization-based auto-scaling) and reliability practices (exponential backoff, a DLQ for permanent failures, iterator-age monitoring to catch processing lag, circuit breakers on downstream dependencies, and cross-region replication for disaster recovery).

  IteratorAgeHigh:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${StreamName}-IteratorAge-High"
      MetricName: GetRecords.IteratorAgeMilliseconds
      Namespace: AWS/Kinesis
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 2
      Threshold: 300000  # 5 minutes
      ComparisonOperator: GreaterThanThreshold

FAQ

Common questions

Discussion