Debug system performance with Brendan Gregg USE+TSA

This skill is a fixed, evidence-first procedure for system performance debugging, root-cause analysis (RCA), and incident reporting, distilled from Brendan Gregg's published methodologies. Instead of running whichever commands happen to be familiar, the agent poses questions first and then finds metrics to answer them. The USE Method (Utilization, Saturation, Errors) sweeps every resource, the TSA Method (Thread State Analysis) decomposes thread time, and off-CPU analysis plus flame graphs drill into what the sweeps find. Every investigation ends in a structured triage note, RCA report, or postmortem in which each claim traces to a command and its output. The full checklists, reference library, and report templates live in the community repository it adapts.

The procedure has eight steps:

0. Problem statement. Define the problem before measuring: what makes you think there is a problem, has it ever performed well, what changed recently in software, hardware, or load, can it be expressed as latency or run time, who else is affected, and what is the environment, including container and VM limits.
1. A 60-second Linux triage with a ten-command sweep, checking errors and saturation first because they are easiest to interpret, then utilization, and recording every exonerated resource.
2. A USE sweep. For every resource, check utilization, saturation, and errors, covering CPUs, memory, network interfaces, storage, controllers, and interconnects, plus software resources such as locks, thread pools, and file descriptors and imposed limits such as cgroup quotas. Check errors before utilization. A clean sweep is a result because it narrows the search space.
3. A TSA sweep. For each thread of interest, split time into executing, runnable, anonymous paging, sleeping, lock, and idle. If more than about 10 percent of time is runnable or paging, fix those first.
4. Drill down on the biggest contributor with CPU profiles and flame graphs, off-CPU stacks, time-division decomposition, or the RED method (rate, errors, duration) for microservices, preferring eBPF in-kernel aggregation and sub-second traces in production.
5. Confirm the root cause with a causal chain of trigger, mechanism, and symptom, every link backed by evidence, keeping falsifiable hypotheses on record, asking why up to five times, and checking that removing the cause would prevent recurrence.
6. Fix and verify with the cheapest effective fix, in the order: do not do it, cache it, do it less, do it later, do it off-peak, do it concurrently, make it cheaper. Re-measure with the same instruments as the evidence; deployed is not verified.
7. Report as a triage note, RCA report, or full postmortem with summary, impact, root cause, detection, investigation log, evidence table, resolution, and prevention actions, using absolute dates and marking unknowns as known-unknowns.

Use it when a server, VM, or container is slow and the cause is unknown, when latency or throughput regressed after a deploy, config change, or load shift, when CPU, memory, disk, or network metrics look abnormal, when an application hangs or threads pile up, or when an incident needs an RCA report or a blameless postmortem with an evidence trail.

The deepest checklists target Linux, and other operating systems follow the same resource by metric matrix with different instruments. It does not replace environment-specific validation or expert review.

The input is a symptom description and access to the affected host. The triage sweep is:

uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

The output is a report where each conclusion cites evidence. In the worked examples, a slow server shows out-of-memory kills, swapping, and almost no page cache, so memory capacity saturation is the cause while CPU, disk, and network are exonerated with numbers. A post-deploy latency regression shows a clean host sweep but runnable-dominant threads, leading to cgroup CPU throttling after a replica increase, fixed by raising the limit and verified over 72 hours.

Standard local observability tools: vmstat, iostat, sar, perf, bcc tools such as offcputime, and /proc reads. Some need privileges or packages: eBPF and bcc need Linux 4.8 or newer and usually root, perf needs perf_events access, and sar needs sysstat. Missing instruments are reported as known-unknowns, not silently skipped. Related skills are devops-troubleshooter, incident-responder, and application-performance-performance-optimization.

Safety: diagnostics are read-only first, and any remediation such as config edits, restarts, or limit changes needs explicit user confirmation. Production tracing has overhead, since scheduler events can reach millions per second. Use it only on systems you are authorized to operate.

Best practices: quantify everything, record what was ruled out with evidence, check errors and saturation before utilization, and re-measure after the fix. Avoid random tuning, trusting low average utilization to rule out saturation, treating a green dashboard as working, and blaming another team's component without data.

It is for SREs, DevOps and platform engineers, and backend developers who troubleshoot slow systems and need a disciplined, reproducible method with an audit trail.