Configure ELK Stack for Production

This skill provides expertise in configuring the ELK Stack (Elasticsearch, Logstash, and Kibana), specializing in building reliable, scalable, and secure log management solutions, with deep knowledge of cluster configuration, data pipelines, index management, security, and performance optimization. Core configuration principles cover Elasticsearch configuration (production cluster settings, heap sizing at roughly 50% of available RAM up to a 32GB max, discovery settings for multi-node clusters, proper index lifecycle management, dedicated master nodes for 3+ node clusters), Logstash pipeline design (clear input-filter-output flow, conditional logic for multiple log types, proper error handling and dead-letter queues, appropriately sized batches and workers), and Kibana security/access (security features always enabled in production, proper authentication and authorization, SSL/TLS encryption for all communication, role-based access control).

Elasticsearch configuration is demonstrated through a production elasticsearch.yml setting cluster name and node roles (master/data/ingest), network and discovery settings for a 3-node cluster with cluster.initial_master_nodes, memory settings (bootstrap.memory_lock, index buffer size), X-Pack security with SSL enabled for transport and HTTP, and monitoring collection. An index template for logs defines shard/replica counts, an ILM rollover alias, and field mappings for timestamp/level/message/host/service.

Logstash pipeline patterns are demonstrated through a multi-input pipeline configuration accepting Beats, TCP syslog, and HTTP webhook inputs, filtering with grok patterns for syslog, conditional field additions, JSON parsing for structured log messages, GeoIP enrichment for client IPs, and field cleanup, outputting to a date-based Elasticsearch index with debug stdout output for debug-level logs. A pipelines.yml configuration defines separate named pipelines (main and security-logs) with distinct worker counts, batch sizes, and batch delays.

Index lifecycle management is demonstrated through an ILM policy defining hot phase rollover (10GB max size or 1 day max age), warm phase at 7 days (reduced priority, zero replicas), cold phase at 30 days (lowest priority), and deletion at 90 days. Kibana configuration is demonstrated through a production kibana.yml setting server host/port, Elasticsearch connection with the kibana_system service account (password supplied via secret management, not hardcoded), X-Pack security with an encryption key and SSL certificate/key paths, and reduced log verbosity.

Security and monitoring best practices cover enabling X-Pack security on all components, strong password rotation, network segmentation and firewall rules, audit logging for compliance, and SSL/TLS certificates from a trusted CA. Performance optimization covers monitoring cluster health via dedicated monitoring indices, alerting on high CPU/memory/disk usage, hot-warm-cold architecture for cost optimization, appropriate index refresh intervals, and proper shard sizing (20-50GB per shard). Operational guidance covers testing configurations in staging first, using configuration management tools (Ansible, Puppet) for consistency, backup strategies for critical indices, monitoring log ingestion rate to tune pipeline worker counts, centralized logging for the ELK stack's own components, and using index aliases for zero-downtime index management.

Use this skill when configuring or hardening an ELK Stack deployment - Elasticsearch cluster settings, Logstash pipeline design, index lifecycle policies, or Kibana security. It is well suited to production log management deployments needing reliability, security, and cost-optimized retention. It is not meant for small single-node development setups with no clustering, security, or lifecycle management needs.

Input: the log sources, expected ingestion volume, and retention/security requirements for the ELK deployment.

Output: Elasticsearch cluster configuration, Logstash pipeline definitions, ILM policies, and Kibana security configuration. Example ILM hot-phase rollover policy:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "10GB", "max_age": "1d" }
        }
      }
    }
  }
}

Configures Elasticsearch, Logstash (with Beats, TCP, and HTTP inputs), and Kibana, integrating with X-Pack security and configuration management tools like Ansible or Puppet.

Infrastructure and platform engineers configuring production ELK Stack log management, and teams that need secure, scalable, cost-optimized log retention rather than a bare-bones single-node setup.