When should I use Debug Buttercup?

Use it when pods are in CrashLoopBackOff/OOMKilled states, multiple services restart simultaneously, Redis is unresponsive, queues are growing without task progress, nodes show disk/memory/PID pressure, build-bot can't reach Docker, the scheduler is stuck, or health probes fail unexpectedly. Do not use it for deploying/upgrading Buttercup, issues outside the `crs` namespace, or performance tuning without actual failure symptoms.

What does Debug Buttercup do first when investigating an issue?

It starts every investigation with three commands: checking pod status (`kubectl get pods -n crs -o wide`), viewing sorted events, and filtering to warning-only events. This provides a baseline before narrowing down to specific pod failures.

How does Debug Buttercup detect cascade failures?

When many pods restart together, it checks for matching errors across multiple `--previous` logs. Redis going down typically produces `ConnectionError`/`ConnectionRefusedError` across many services simultaneously, indicating you should debug Redis rather than each individual service.

Skill

Debug Kubernetes Pod Failures

Name: Debug Buttercup
Availability: OnlineOnly
Author: Antigravity

Kubernetes triage playbook for Buttercup's crs namespace - crash loops, Redis cascades, queue backlogs, and resource pressure.

Get skill

Works with kubernetes redis dockerhelm

Antigravity

Own this? Claim it

Spark score

out of 100

Updated last month

Version 13.4.0

Add to Favorites

Why it matters

Diagnose and resolve issues with Kubernetes pods in the 'crs' namespace, including CrashLoopBackOff, OOMKilled, and cascading service restarts.

Outcomes

What it gets done

Analyze pod statuses, events, and logs to identify root causes of failures.

Inspect resource utilization and node conditions for pressure indicators.

Debug critical services like Redis and investigate queue backlogs.

Verify deployment configurations against actual pod states.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/ag-debug-buttercup | bash

Overview

Debug Buttercup

Kubernetes triage playbook for Buttercup's crs namespace - crash loops, OOM kills, cascade failures, Redis health, queue backlogs, resource pressure, and health-check staleness - with a fixed diagnostic workflow and an automated diagnose.sh snapshot script. Use it when crs pods are crashing, restarting in cascade, Redis is unresponsive, queues aren't progressing, nodes show resource pressure, or deployed Helm values don't match actual pod config. Not for deploying Buttercup itself or issues outside the crs namespace.

What it does

Debug Buttercup is a Kubernetes triage playbook for Buttercup's crs namespace, covering its infra (redis, dind, litellm, registry-cache), orchestration (scheduler, task-server, task-downloader), fuzzing (build-bot, fuzzer-bot, coverage-bot, tracer-bot, merger-bot), analysis (patcher, seed-gen, program-model, pov-reproducer), and interface (competition-api, ui) services. It starts every investigation with the same three commands - pod status, sorted events, and warning-only events - then narrows down to a specific pod's last-state reason, actual resource limits, and previous/current logs.

When to use - and when NOT to

Use it when pods in crs are CrashLoopBackOff/OOMKilled/restarting, multiple services restart simultaneously (a cascade failure), Redis is unresponsive or showing AOF warnings, queues are growing but tasks aren't progressing, nodes show disk/memory/PID pressure, build-bot can't reach the Docker daemon, the scheduler is stuck, health probes fail unexpectedly, or deployed Helm values don't match actual pod config. Not for deploying/upgrading Buttercup itself (use Helm/deployment guides), issues outside the crs namespace, or performance tuning with no actual failure symptom. Cascade detection is a specific pattern to check first when many pods restart together: Redis going down produces ConnectionError/ConnectionRefusedError across many services simultaneously, so matching errors across multiple --previous logs means debug Redis, not each individual service.

Inputs and outputs

Key diagnostic commands: kubectl get pods -n crs -o wide, kubectl get events -n crs --sort-by='.lastTimestamp' (and filtered to type=Warning), kubectl describe pod ... | grep -A8 'Last State:', and kubectl logs -n crs <pod> --previous --tail=200 for the crashed container specifically. --since=300s and --timestamps help confirm an issue is actively ongoing rather than an old accumulated restart count. Redis diagnostics go through redis-cli (INFO memory/persistence/clients/stats, CONFIG GET appendonly/appendfsync, CLIENT LIST, DBSIZE) plus checking whether /data is backed by disk or tmpfs. Queue inspection covers 13 named Redis-stream queues (build, crash, patches, index, POV requests/responses, etc.) via XLEN, XINFO GROUPS, and XPENDING against known consumer groups (build_bot_consumers, orchestrator_group, patcher_group, index_group, tracer_bot_group). Health checks write a timestamp to /tmp/health_check_alive, checked for staleness when a pod restart-loops (usually meaning the main process is blocked on Redis or I/O). Resource pressure is checked via kubectl top pods/nodes, node conditions, and in-pod disk usage (/corpus, /scratch). OpenTelemetry/Signoz tracing helps pinpoint bottlenecks across the scheduler -> build-bot -> fuzzer-bot chain when Signoz is deployed.

Integrations

An automated scripts/diagnose.sh (with a --full flag to dump recent logs from all pods) collects pod status, events, resource usage, Redis health, and queue depths in one pass; deployment/collect-logs.sh gathers all logs to disk. Deeper per-service failure patterns (DinD, build-bot, fuzzer-bot, patcher, scheduler) live in references/failure-patterns.md.

Who it's for

Operators and SREs running Buttercup's fuzzing/CRS infrastructure on Kubernetes who need a systematic triage path for crash loops, Redis-driven cascade failures, stuck queues, or resource pressure, instead of investigating each failing pod from scratch.

FAQ

Common questions

Discussion