Debug system performance with Brendan Gregg USE+TSA

This skill runs a fixed, evidence-first procedure for system performance debugging, root-cause analysis (RCA), and incident reporting, distilled from Brendan Gregg's published methodologies. Instead of running whichever commands are familiar, it poses questions first and then finds metrics to answer them: the USE Method (Utilization, Saturation, Errors) sweeps every resource, the TSA Method (Thread State Analysis) decomposes thread time into Executing, Runnable, Anonymous Paging, Sleeping, Lock, and Idle states, and off-CPU analysis plus flame graphs drill into what the sweeps find. It adapts material from the community repository thecsdoctor/brendangregg-use-tsa-skill, which holds the full checklists, reference library, and report templates. Every investigation ends in a structured triage note, RCA report, or postmortem where each claim traces back to a command and its output.

Use this skill when a server, VM, or container is "slow" and the cause is unknown; when latency or throughput regressed after a deploy, config change, or load shift; when CPU, memory, disk, or network metrics look abnormal and need interpretation; when an application hangs or threads pile up; or when an incident needs an RCA report or a blameless postmortem with an evidence trail. It does not replace environment-specific validation, testing, or expert review, and some of its deepest instrumentation (eBPF/bcc, perf, sar) needs Linux 4.8+, root, or specific tooling that may be absent; missing instruments are reported as known-unknowns rather than silently skipped.

The workflow starts with a Step 0 problem statement (what makes you think there's a problem, what changed recently, can it be expressed as latency or run time, who else is affected, what's the environment). It then runs a 60-second Linux triage sweep:

uptime                 # load trend (includes uninterruptible I/O on Linux)
dmesg | tail           # kernel errors: oom-killer, SYN flooding, hardware
vmstat 1               # r > CPU count = CPU saturation; si/so = swapping; wa = disk
mpstat -P ALL 1        # per-CPU imbalance (single hot CPU = single-threaded app)
pidstat 1              # per-process CPU over time
iostat -xz 1           # await (app-suffered latency), avgqu-sz, %util
free -m                # memory; buffers/cache near zero hurts
sar -n DEV 1           # NIC throughput vs link limit
sar -n TCP,ETCP 1      # active/passive connections, retransmits
top                    # spot variable load

From there it runs a resource-oriented USE sweep (checking errors and saturation before utilization across CPUs, memory, network, storage, controllers, software resources, and imposed limits), a thread-oriented TSA sweep, and a drill-down step that follows the biggest contributor (CPU profile and flame graphs for Executing time, off-CPU stacks for Sleeping/Lock time, the RED method for microservices). Output is a causal-chain root-cause statement, a fix applied in cheapest-first order, a same-instrument before/after verification, and a final triage note, RCA report, or full postmortem with an evidence table.

The skill relies on standard local Linux observability tools rather than any external service: vmstat, iostat, mpstat, pidstat, sar, dmesg, top, /proc/PID/schedstat, perf sched latency, bcc tools (offcputime, cpudist), /proc/lock_stat, valgrind --tool=drd, and flamegraph.pl. There are no network fetches or credential handling involved.

SREs, on-call engineers, and anyone doing performance triage, root-cause analysis, or postmortem writing on Linux servers, VMs, or containers who wants a fixed, falsifiable methodology instead of ad hoc troubleshooting.