Gcp Cost Review

gcp-cost-review

A Claude Code skill for reading, explaining and reducing Google Cloud spend from the
BigQuery billing export - and for running a repeatable monthly cost review.

Most cloud-cost advice is about ideas: turn idle things off, right-size, delete orphans.
Those are cheap and mostly public. What actually goes wrong is measurement: the
billing export quietly supports a wrong conclusion, and you ship a change, declare a
saving, and move on while the number came from somewhere else.

So this skill front-loads a measurement contract and only then looks for money. It comes
out of a real multi-week engagement, including the mistakes - several findings in it
exist because a confident estimate turned out to be wrong in a way that was expensive to
discover.

What's inside

SKILL.md	the workflow - bootstrap, baseline, attribute, price, ship, verify
references/measurement-contract.md	how to make your numbers match the console, and the settle gate
references/traps.md	the catalogue of expensive mistakes, each with its tell
references/where-the-money-hides.md	where GCP spend actually accumulates, by service
references/monthly-review.md	the recurring review loop and its report template
schedule/	run it on a schedule - a runner, a launchd plist, cron and systemd lines
references/no-export-yet.md	no export yet? enable it, and what you can measure meanwhile
scripts/discover.py	find the billing export and describe its shape
scripts/bq.py	run a parameterised query and print a readable table
assets/queries/	the canonical SQL, account-agnostic

Install

Prerequisites, neither of which the clone gives you - measured in a clean Debian container,
where pip, pip3, python3 -m pip and ensurepip were all absent, and so was gcloud:

• Python 3.9+ with pip and venv. On slim Debian/Ubuntu images ensurepip cannot fix this;
  install the distro packages as root first.
• The Google Cloud CLI (gcloud) if you authenticate that way - it is a separate install.
  (ADC does not universally require gcloud; an attached service identity or workload identity
  federation works too. gcloud is simply the usual route on a laptop.)

git clone https://github.com/BayramAnnakov/gcp-cost-review.git
ln -s "$PWD/gcp-cost-review" ~/.claude/skills/gcp-cost-review
cd gcp-cost-review

# only if pip/venv are missing (slim Linux images), as root:
apt-get update && apt-get install -y python3-pip python3-venv

python3 -m venv .venv && . .venv/bin/activate
python -m pip install -r requirements.txt

gcloud auth application-default login   # credentials for the BigQuery queries
gcloud auth login                       # ALSO needed for the gcloud resource sweeps —
                                        # ADC alone does not authenticate those

What access you need

The gcloud login above grants none of this. All of it is read-only:

to…	role	on
read the bill / invoices	roles/billing.viewer	the billing account
query the export	roles/bigquery.dataViewer (export dataset) + roles/bigquery.jobUser (the project running queries - often a different one)	BigQuery
sweep resources	roles/viewer + roles/recommender.viewer	each project
enable the export, if absent	roles/billing.admin	the billing account

Those first three are a modest ask and usually land same-day. billing.admin is the one that
stalls - it typically sits with finance or a founder, so if the export does not exist yet, ask
for it on day one and work through references/no-export-yet.md meanwhile.

Check what you actually have with:

python scripts/discover.py                    # tries your ADC quota + default projects
python scripts/discover.py my-billing-project # or name it explicitly

Querying the export is billed on bytes scanned, so scripts/bq.py dry-runs every
query, prints the estimate, and refuses anything over --max-gb (default 20).

Then just ask: "why did our GCP bill go up last month?" or "run the monthly cloud cost
review".

To make it recurring, see schedule/ - it pre-pulls the data on the 5th (the
month that just closed) and the 20th (a mid-month watch for new SKUs and reverted savings), so
the review starts with numbers. Run it by hand once before you schedule it; the README there
explains why that is not optional.

Keep your outputs out of git. The export table name contains your billing account id
and the reports contain real spend. .gitignore already excludes cost-reviews/ and
calibration.local.md; keep the calibration note outside the repo if you can.

The five ways the billing export will mislead you

These are the reason the skill exists. Each has produced a confident, wrong, expensive
answer:

1. An unsettled day reads low - and it reads low across every service at once, which
   looks exactly like a successful optimization. (And the gate against it is necessary,
   not sufficient - services export on their own schedules.)
2. Credits come in several shapes - free-tier pots, proportional discounts, promotions
   that expire and commitments that floor your bill all share one array. Where a discount
   already takes a SKU to $0.00 net there is no opportunity however large gross looks; where
   a commitment covers it, cutting usage may save nothing at all.
3. Tiered allowances reset monthly - a drop on the 1st is the calendar, not your work.
4. The credit draw is front-loaded - so a late-month net run rate overstates the month.
5. The biggest mover may not be your change - take credit for it and you will mislead
   the owner and stop looking for the real win.

The defence against all five is the same: establish the measurement contract before you
look for money, find the step date rather than splitting a delta, and keep MEASURED
and INFERRED apart in your own words.

Two techniques worth stealing even if you don't use the skill

The settle gate. Pick a SKU that bills an identical amount every day - a fixed-capacity
managed instance, a cluster fee, a reserved address. A day is complete when that SKU reads
its known value, and not before. This beats "wait N days" because lag varies, and beats a
completeness percentage because that is derived from the incomplete data you're trying to
judge.

Zero versus a fraction. When you claim a change worked, show the controls. A partial
export yields a fraction of something; a change to zero yields zero. "It went to zero"
is weak; "it went to zero while ten unrelated workloads all sat at 83-84% of yesterday" is
strong. Check the row count too - zero dollars across a normal number of rows is a real
zero, while zero dollars across zero rows is missing data wearing the same costume.

What this skill does not do

• It does not enable the billing export for you. The export collects forward from when it is
  switched on, with one exception: a first export into a US/EU multi-region dataset backfills
  from the start of the previous month. So "why did last quarter change" may be unanswerable
  • the skill says so rather than substituting a worse instrument. No export at all? Start
    at references/no-export-yet.md - the console does attribute by service, SKU, project and
    label, so there is a real degraded path.
• It does not make changes to your infrastructure. It reads, prices, and tells you what
  to verify.
• It is GCP-specific. The method generalises; the SQL does not.

references/measurement-contract.md

The measurement contract

Establish this once per billing account, write the answers down, and do not re-derive
them. Re-deriving invites a different answer, and the whole point is that every later
number rests on the same footing.

1. The export is the only real instrument

The Cloud Console billing reports do more than a total - they break down by service, SKU,
project and label, and export CSV. What they cannot give you is SQL over arbitrary windows,
per-row credit detail, and reproducibility: a figure someone else can re-run and check. That
is what the BigQuery billing export adds, and the account owner enables it in
Billing → Billing export.

Two things to know before promising an answer:

• The export essentially only collects from the moment it is enabled. One exception
  worth knowing: a first export - standard or detailed - into a US or EU
  multi-region dataset backfills from the start of the previous month. That is the only
  backfill on offer, so if it was turned on last week, "why did last quarter change" is still
  unanswerable from it. Say so rather than substituting a worse instrument.
• There are two exports. Standard gives service + SKU + project + labels, and is
  enough for almost all cost work. Detailed adds per-resource rows, which you need
  only when one SKU is shared by many resources and you must know which one. Detailed
  costs more to store.
• Querying the export costs money, in proportion to bytes scanned. A cost review that
  runs up a BigQuery bill is a bad joke, so scripts/bq.py estimates with a dry run and
  refuses anything over a cap. Keep windows narrow.

2. Make your query match the console

Three conventions, and all three matter:

Convention	Why
Group by day in US Pacific (America/Los_Angeles)	That is what Cloud Billing reports use, with daylight saving. Not UTC, and not the account owner's local zone - either shifts spend across day and month boundaries.
Use net = cost + sum of the credits array	The console shows net. cost alone is gross and will read high.
Filter cost_type = 'regular'	Otherwise tax, adjustment and rounding_error rows mix into service totals.

SELECT DATE(usage_start_time, 'America/Los_Angeles') AS day,
       service.description AS service,
       SUM(cost) AS gross,
       SUM(cost + IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)) AS net
FROM `<BILLING_EXPORT_TABLE>`
WHERE cost_type = 'regular'
  AND DATE(usage_start_time, 'America/Los_Angeles') BETWEEN '<START>' AND '<END>'
GROUP BY day, service

Those three conventions are for analysis - "what did we consume, and when". They are
the wrong key for "what were we charged": an invoice is keyed on invoice.month, and
it includes tax and adjustments that cost_type='regular' drops. Reconcile a bill with
assets/queries/invoice-reconcile.sql, and expect the two totals to differ slightly.

Say which console view you are matching. "Charge period" excludes taxes and
adjustments; "Billing period" includes them. Comparing your cost_type='regular' sum
to the wrong one produces a mismatch you will waste an afternoon on.

Validate against the console before trusting anything downstream. If a complete
month does not reconcile, the error is in your conventions, and it will silently
propagate into every finding.

3. The settle gate

Export rows arrive in batches. A partial day is not flagged; it simply reads low, and it
reads low across every service at once, which is exactly what a successful
optimization also looks like.

So gate on a flat-rate control: a SKU that bills the same amount every day
regardless of traffic. Good candidates, in rough order of reliability:

• a managed cache or database instance with fixed capacity
• a per-cluster or per-instance management fee
• a reserved/static IP address charge
• a flat per-day storage or licence line

(Deliberately not a committed-use or subscription line: those carry a
FEE_UTILIZATION_OFFSET or similar by construction, which is exactly what rules a control
out - see the warning below.)

⚠ A control must carry no credit. Check it with credit-inventory.sql first. The
obvious candidate - a per-cluster or per-instance management fee - is often exactly the
SKU a capped free-tier pot is applied to: measured on a real account, a cluster-fee SKU
had a perfectly flat gross and a net that swung from 0× to 1× of it across the month
as the pot drained. Gate on gross, and prefer a control with no credit at all.

Record its exact daily gross value. Then:

Until the control reads its known value, the day is certainly incomplete.

Note the direction: this tells you when to stop trusting a day, not when to start. A
historical p95 is a distribution with a tail, and Google publishes no delivery-time guarantee,
so past punctuality never certifies today's missing rows.

This is better than "wait N days" because lag varies, and better than a completeness
percentage because that is itself derived from the incomplete data.

Measure your own account's lag once, with export_time. The export carries an
export_time column recording when each row was appended. Per service, compare
export_time to the usage day it describes:

SELECT service.description AS svc,
       APPROX_QUANTILES(TIMESTAMP_DIFF(export_time, usage_end_time, HOUR),
                        100)[OFFSET(95)] AS p95_hours_after_usage
FROM `<BILLING_EXPORT_TABLE>`
WHERE cost_type='regular'
  AND DATE(usage_start_time,'America/Los_Angeles') BETWEEN '<START>' AND '<END>'
GROUP BY svc ORDER BY p95_hours_after_usage DESC

⚠️ Compare export_time against usage_end_time - the clean "how long after the usage did
this row arrive" question. The tempting alternative,
TIMESTAMP(DATE(usage_start_time,'America/Los_Angeles')), is a bug: DATE() returns a
Pacific calendar date and TIMESTAMP() then reads it as UTC midnight, inflating every figure
by the UTC offset. Measured: exactly +7 h on all 15 services. If you do want "age since the usage
day began", pass the zone explicitly -
TIMESTAMP(DATE(...,'America/Los_Angeles'), 'America/Los_Angeles') - and label it as the
different measurement it is.

The spread is large enough to matter. Measured over a complete month on one account, p95
arrival after usage_end_time ran from low tens of hours for most services to roughly ten
days for the slowest. So a gate built on a fast-arriving control passes while a slow
service's rows for that same day are still days out - quote that service and you are quoting a
partial. Measure your own; these are an order of magnitude, not a constant.

Weight the alarm by money: on that account every service had fully landed within five days
except the slowest, and the slowest billed $0.00. A service that arrives late and costs
nothing does not threaten a conclusion. Sort your lag table next to the spend table before
deciding which days you can use.

That tells you which services are slow enough to distrust, in your account rather than in
general. Do it once in Step 0 and record it. It does not make the gate sufficient - nothing
does - but it stops you quoting a service whose rows are known to arrive days late.

Keep two or three controls if you can. One flat SKU can change for its own reasons - a
resize, a price change - and you want to notice that rather than mistake it for lag.

4. Normalise to a 30.44-day month

Months are 28-31 days. Comparing a 31-day month to a 30-day month shows a 3% "saving"
that is the calendar. Comparing a complete month to a 9-day sample is worse.

Normalise everything: sum(window) / days(window) × 30.44. State that you did.

5. Gross or net - decide per SKU, and say which

Neither is universally right.

• Capped-pot credit (a fixed monthly free allowance, e.g. a per-account cluster-fee
  allowance): the pot is a constant. Judging a change on net hides the change behind the
  pot. Judge on gross.
• Proportional discount (a credit that is a fixed percentage of the line, including
  100%): net is what you pay. A 100% discount means net is $0.00 and there is nothing
  to save, however large gross looks.
• Tiered free allowance (Cloud Monitoring metrics, Cloud Logging: first N units free
  each month): gross is already net of the allowance. Comparing partial months is
  meaningless because the allowance resets. Use complete months, or better, compare
  usage units - pair usage.amount with usage.unit, or usage.amount_in_pricing_units with usage.pricing_unit. Those are the only two correct pairings; crossing them compares raw units to priced ones and can be wrong by orders of magnitude.
  ⚠ Not every observability SKU works this way - Managed Service for Prometheus is priced
  per sample with no free allotment. Check the SKU's pricing unit rather than assuming
  a free tier exists.

Read credits.type rather than guessing from shape. It is documented as a string with a
listed set of values, not a closed schema enum - so handle an unexpected value rather than
assuming the list is exhaustive. The documented values are
FREE_TIER, PROMOTION, DISCOUNT, COMMITTED_USAGE_DISCOUNT,
COMMITTED_USAGE_DISCOUNT_DOLLAR_BASE, SUSTAINED_USAGE_DISCOUNT,
FEE_UTILIZATION_OFFSET, RESELLER_MARGIN and SUBSCRIPTION_BENEFIT. What each means
for your arithmetic:

type	shape	what to do
FREE_TIER	an allowance - sometimes dollars, often units (free instance-hours)	judge on gross; subtract once, never below eligible spend. It does not always drain early: a unit allowance consumed by something always-on spreads across the month. ⚠ Note a free tier does not always carry this type - one measured account books its GKE pot as plain DISCOUNT - so classify by behaviour too
PROMOTION	trial, milestone or marketing credits; a balance that runs out	a strong candidate when a bill jumps for no structural reason - check the remaining balance and expiry before forecasting
DISCOUNT	contractual, e.g. spend-threshold based	the daily shape tells you how it behaved, not what the contract says - find the contract before relying on it
SUSTAINED_USAGE_DISCOUNT	proportional, up to ~30%; booked unevenly across rows	read it against the SKU's full gross over a whole month. Against credited rows alone the ratio exceeds 1 and is meaningless
COMMITTED_USAGE_DISCOUNT*, FEE_UTILIZATION_OFFSET	tied to a commitment	the commitment is payable either way, so a cut to covered usage can save ~nothing - never price one without checking commitments
RESELLER_MARGIN, SUBSCRIPTION_BENEFIT	you are billed through a reseller, or hold a support/subscription plan	your effective price is not list price. Do not price any change off public rates without checking the reseller agreement

⚠ Take the ratio against the SKU's FULL gross, never against the credited rows alone - and
be careful what you conclude if you got it wrong. An earlier draft of this file declared the
ratio heuristic unreliable and cited a sustained-use discount at −5.2 and a "pot" at −0.50,
−0.86 and −0.93. Every one of those numbers came out of the broken inner-join query that
trap A10 is about: it compared each credit against only the rows carrying it. Re-measured with
the corrected query, the same sustained-use discount reads a clean 30% of full SKU gross -
exactly its documented maximum - and the −0.50/−0.86/−0.93 figures turn out to belong to
unrelated allocation-time and introductory discounts, not to a pot at all.

So the heuristic is usable once the join is right. The lesson is narrower and more
uncomfortable: guidance derived from a buggy query inherits the bug, and it reads as a
finding about the world rather than about your SQL. Still confirm the shape against the daily
series (credit-shape.sql, per credit name) and against credits.type before acting - a ratio
tells you what happened inside your window, not what the contract says. Classify from the daily series (credit-shape.sql
per credit name: a pot is flat then abruptly zero) and from credits.type, and use the
ratio only as a hint.

Inspect the credit rows rather than assuming:

SELECT c.name, c.type, ROUND(SUM(c.amount),2) AS total
FROM `<BILLING_EXPORT_TABLE>`, UNNEST(credits) c
WHERE DATE(usage_start_time, 'America/Los_Angeles') BETWEEN '<START>' AND '<END>'
GROUP BY 1,2 ORDER BY total

If a credit's total magnitude exactly tracks its SKU's gross, it is proportional. If it
plateaus at a round number and stops, it is a pot.

6. Credit shape, for projection

Run assets/queries/credit-shape.sql. A fixed pot is usually drawn down in the first
days of the month, so the daily credit is large early and small later.

Consequence: a run rate computed on net from a late-month window overstates the month,
because those days carry little credit. Project on gross, then subtract the pot as a
monthly constant.

7. Tax

Tax is typically stamped at the month boundary and exported later, so a just-closed
month can show tax incomplete for days. Record the historical tax as a percentage of net
and use that to estimate an invoice; do not assume the tax rows you can see are final.

8. Write it down - and keep it out of a public repo

Use a deterministic path so a later session, or a student's agent, knows where to look.
calibration.local.md at the repo root (already in .gitignore), or
~/.config/gcp-cost-review/calibration.md if the repo is public:

# Cost review calibration — <ACCOUNT NAME>
export_table:   <project>.<dataset>.gcp_billing_export_v1_XXXXXX
export_began:   YYYY-MM-DD        # nothing before this is answerable
timezone:       America/Los_Angeles   # fixed by Cloud Billing, not a choice
console_view:   Billing period    # or "Charge period" (excludes tax/adjustments)
control_sku:    "<sku LIKE pattern>"  -> <N.NNN>/day GROSS, carries no credit
second_control: "<sku LIKE pattern>"  -> <N.NNN>/day GROSS
export_lag_p95: <svc>: <N>h, <svc>: <N>h   # measured once, with export_time
credits:
  - name: "<credit name>"  type: FREE_TIER   shape: pot, ~$X/mo, drains by day ~N
  - name: "<credit name>"  type: PROMOTION   balance: $X remaining, expires YYYY-MM-DD
tax_rate:       ~N.N% of net
commitments:    none | <describe - a cut to covered usage may save ~nothing>

A calibration note with: export table id, the control SKU(s) and their exact daily gross
values, the credit inventory with each one's shape and expiry, the tax rate, and the
date the export began. Every future session reads this first.

⚠ This note is sensitive, and so is every report you generate from it. The export
table name embeds your billing account id; the reports contain your spend. If the
repo is public - or is a course repo that will become public - put both somewhere that
cannot be committed by accident:

# cost review outputs - contain billing account ids and real spend
cost-reviews/
calibration.local.md
*.costreview.md

Better: keep the calibration note outside the repo entirely (~/.config/), and have
scheduled jobs write reports to a private directory. When you want to publish or teach
from a review, publish a redacted or synthetic copy - ratios and shapes carry the
lesson; absolute spend and resource names do not.

references/traps.md

Traps

Each of these produced a confident wrong answer in a real engagement. They are ordered
by how often they bite, and each has a tell you can check cheaply.

---

A. Measurement traps

A1. Reading an unsettled day

Looks like: yesterday's spend is down 15%. Something worked.
Actually: the export is partial. It is down 15% across every service, including
ones you did not touch.
Tell: your flat-rate control is below its known daily value.
Rule: gate every day on the control. Never quote a number from an ungated day.

A2. Two credit mechanisms in one array

Looks like: a SKU shows meaningful gross spend, so it is an opportunity.
Actually: it carries a matching 100% discount credit and nets to $0.00. Turning it
off saves nothing and may break something.
Tell: sum the credits array for that SKU. If it equals -gross, the opportunity is
zero.
Paid for: a service was written up as a recurring saving, scheduled for removal, and
was $0.00 net the whole time. It was caught only because an external source
contradicted the claim. Afterwards, sweeping every SKU above a threshold for matching
credits found a second one in the same state.
Rule: check the credit shape before pricing anything. The habit of "judge on gross"
is correct for a capped pot and actively wrong for a proportional discount.

A3. Attributing an allowance reset to your change

Looks like: metrics or logging spend collapsed on the 1st. Our cleanup worked.
Actually: the monthly free allowance reset.
Tell: the drop lands exactly on a month boundary; usage units did not move.
Rule: for tiered services compare complete months, or compare usage units (MiB, GiB)
rather than dollars.

A4. The inverse: a $0.00 that is about to start billing

Looks like: logging now costs nothing.
Actually: the allowance has not been crossed yet this month. It will cross around
day 20-25 and start billing.
Tell: cumulative usage is tracking toward the allowance, not below it.
Rule: project the full month's usage before declaring a tiered SKU free.

A5. A late-month run rate, taken on net

Looks like: the last settled week implies a monthly cost of X.
Actually: the fixed credit pot was consumed in the first days of the month, so those
late days carry almost no credit and X is too high.
Tell: daily credit totals step down sharply after the first week.
Rule: project on gross and subtract the pot as a monthly constant.

A6. Quoting an invoice from a usage-date sum

Looks like: summing cost_type = 'regular' by usage date gives the month's bill.
Actually: it gives neither. An invoice is keyed on invoice.month, and a usage row
can land on a different invoice than its usage date implies, because late-arriving usage
is billed on the next one. And cost_type = 'regular' silently drops tax and
adjustments, which are on the invoice.
Tell: your figure does not match what finance sees, usually by a small but
embarrassing amount.
Rule: two different questions, two different keys. "What did we consume, and when"
→ usage date, cost_type='regular'. "What were we charged" → invoice.month, no
cost_type filter. Use assets/queries/invoice-reconcile.sql for the second, and never
quote a bill from the first.

A7. Crediting your own work for someone else's change

Looks like: the biggest line in the month-over-month diff fell dramatically right
when you were working.
Actually: a different team, a different product in the same billing account, or a
demand shift.
Tell: find the step date and try to name the change. Split by project. If you
cannot name it, you do not own it.
Paid for: in one engagement the largest single line was another product's config
flip on an unrelated day. The honest attributable figure was roughly a third of the
headline.

A8. Treating a lumpy new line as a run rate

Looks like: a new SKU appeared and a 7-day extrapolation says it is material.
Actually: the daily values span two orders of magnitude and there is no baseline.
Tell: the series is zero for weeks, then erratic.
Rule: report "this was reliably zero and is now not, starting on date D" - which is
the actionable fact - and refuse to annualise it until it stabilises.

A9. A command that fails through a pipe and still exits 0

Looks like: <cmd> | wc -l returns a plausible count.
Actually: the command errored and you counted the lines of the error message. Expired
credentials and disabled APIs both do this.
Tell: run the command bare before piping it. Check that the output looks like data.
Rule: prefer application-default credentials over short-lived printed tokens for
anything scripted, and never let a pipeline be the first place a command runs.

A10. An UNNEST join silently drops the rows you need

Looks like: FROM export, UNNEST(credits) c then comparing SUM(c.amount) to
SUM(cost) tells you what share of a SKU is discounted.
Actually: that is an inner join. Rows with an empty credits array disappear, so
you compared the credit against only the credited rows' cost. A SKU with one credited
$5 row and one uncredited $95 row reads as 100% discounted while costing $95 net.
Tell: a SKU you know you pay for shows ~100% offset.
Paid for: a cluster-fee SKU read as almost fully covered by a free tier. Corrected, only
about a third of it was offset and the rest was a real bill.
Rule: aggregate the full total first, and pull the nested values per row with a
scalar subquery ((SELECT SUM(c.amount) FROM UNNEST(credits) c)), which keeps rows whose
array is empty. The same trap applies to labels and system_labels.

---

B. Pricing traps

B1. Confusing current spend with realisable saving

Current spend is what the item costs today. The saving depends on what you do:

• removed outright → full cost
• time-sliced (non-prod on demand) → only the idle fraction, and usually less than hoped
• migrated → zero until the old side is deleted
• resized → the delta between tiers, not the whole line

Report the two numbers separately and label them. A list of "current spend" summed and
presented as "savings available" is the most common way cost work loses credibility.

B2. Forgetting that a migration pays twice

During a dual-run you pay for both sides. The saving banks on deletion, which is the
step people defer because it is the scary one. A migration left dual-running
indefinitely is a cost increase.

Rule: make the deletion an explicit, scheduled, owned step with a rollback window,
or do not start.

B3. Right-sizing requests on a bursting platform

Looks like: pods request far more CPU than they use; cutting requests saves money.
Actually: on GKE Autopilot you are billed for requests, but you also burst into the
sum of requests on the node. Cutting requests shrinks the pool you burst into, so the
workload can get slower while saving less than modelled.
Tell: actual usage is spiky, and the headroom is doing real work.
Rule: treat request right-sizing as a performance change that happens to affect cost.
Prove it with a load test, not a spreadsheet.

B4. Mistaking credit arithmetic for the economics of removal

Looks like: this SKU's credits cancel its cost, so there is nothing to save; or, this
SKU has no credit, so removing it saves the full amount.
Actually: both can be wrong, because a credit is a billing artifact and removal is an
economic question.

• a 100% discount may be a time-limited promotion. "Net is $0" can mean "not yet".
• a committed-use discount can make the saving evaporate: the commitment is payable
  regardless, so cutting covered usage drops the usage charge and the offset together and the
  total barely moves. The fee line's net goes up, which looks alarming and is not itself a
  cost increase.
• allowances shared across projects mean a saving in one project is absorbed by another.
  Tell: the credit's name mentions a promotion, trial or commitment; or the account
  has committed-use discounts at all.
  Rule: read the credit's name and expiry, not just its magnitude, and sanity-check any
  material claim as an account-level before/after rather than a per-SKU subtraction.

B5. On GKE Standard, stopping pods does not stop the bill

Looks like: scaling staging workloads to zero saves their share of the cluster cost.
Actually: that is true on Autopilot, which bills pod requests. Standard bills
nodes. Pods going away changes nothing until the autoscaler removes a VM - and it will
not if the pool has a non-zero minimum, or DaemonSets, system pods or another workload
keep the node occupied.
Tell: pods are at zero and the node count has not moved.
Rule: on Standard, price "can a node be removed", and verify with the node count after
the change, not the pod count.

B6. Summing items that are not additive

Two savings can overlap (deleting a cluster removes the nodes and the fee and some
network charges counted separately), or be mutually exclusive, or be ordered (A is only
available after B). Say which, and give the ordering constraint.

---

C. Execution traps

C1. Turning off something that was turned on for a reason

Rule: before disabling anything, grep the repo, the incident log and the issue
tracker for the prior attempt. Resources rarely carry the reason they exist. In more
than one case the "obviously idle" thing was the fix for a past outage.

C2. Reasoning about consumers instead of checking

A negative grep in one directory is a hypothesis. Verify from logs, metrics, or the
running system before concluding nothing uses a thing. Where a control is available, run
the same query shape against something you know is live, to prove the query can return
a non-zero answer at all.

C3. Shipping a safety net without running it

A guard that fails silently is worse than no guard. Execute it end to end, including its
failure path, and watch it act.

Paid for: a scale-down job was deployed in a container image that did not include the
CLI it invoked. Every run errored; the error was being discarded; the job looked healthy
and did nothing. A design review had approved it - a reading gate cannot detect a missing
binary.

C4. Assuming the change is still live when you read the result

Declarative tooling re-applies the committed state. A deploy, a sync, or a teammate can
revert your change without telling you, and your verification will then measure the old
world.

Rule: read the live configuration back from the running system as the first step of
any verification, and say that you did.

C5. Running up a BigQuery bill while looking for savings

Looks like: exploring the export is free.
Actually: you are billed on bytes scanned, and a detailed export on a large account is
big. Wide date ranges and SELECT * over it cost real money.
Rule: dry-run first (bq.py --dry-run prints the estimate), cap with
maximum_bytes_billed, keep windows narrow, and read table metadata rather than querying
when metadata will do.

C6. Scripts that only ran on your machine

Portability bugs show up in cost tooling constantly because it is written quickly and
run rarely: shell builtins that differ between versions, sed/date flags that differ
between GNU and BSD, a binary present locally and absent in the container.

Rule: run the thing in the environment it will actually run in, once, before trusting
it.

references/where-the-money-hides.md

Where the money actually hides

Ordered roughly by how much is usually there and how safely it comes out. For each:
what to look for, how to price it, and the specific thing that makes the naive estimate
wrong.

A hypothesis to test early, not a fact to assume: idle capacity is often a bigger share
of a bill than busy capacity. It is worth checking first because it is cheap to check and
safe to fix - but establish your own account's distribution before believing it. Run
sku-run-rate.sql and look. An account dominated by egress, model APIs or storage has a
different shape, and starting from the wrong prior wastes the whole engagement.

---

1. Non-production environments running 24/7

Staging, test, dev, QA, preview and demo environments usually cost the same as
production and are used during working hours at best.

Check: every environment's replica count / instance count, and the actual request or
message volume it served in the last 30 days.
Price: full cost if it can be deleted; the idle fraction if it becomes on-demand.
Trap: the realisable number is much lower than the gross. An environment used two
hours a day does not save 92% - you still pay for the storage, the addresses, the cold
starts and the hours somebody forgets to turn it off.

⚠ On Kubernetes, know which billing model you are on before pricing anything. They
behave oppositely:

• Autopilot generally bills pod resource requests, so scaling a workload to zero removes
  the charge directly. Not universally: Autopilot pods that select particular hardware are
  billed on a node basis instead.
• Standard bills nodes. Deleting pods changes nothing by itself - you keep paying for the
  VMs until the cluster autoscaler removes them, and it can only scale down to the pool's
  minimum. Pods that cannot be rescheduled elsewhere will also hold a node up.

The label on the cluster is not the answer: a Standard cluster can host Autopilot workloads,
so determine the billing mode of the workload, not of the cluster. Either way, on a
node-billed workload the saving is "can a node be removed", not "can a pod be stopped" -
verify against the node count after the change.

If you make it on-demand, you need three things or it will cost more than it saves:
a one-command way to bring it up, a lease with an automatic reaper so a forgotten
session cannot bill a month, and documentation at the point of confusion - because the
failure mode is silent. A developer whose environment is off usually sees a hang, not an
error, and will spend an hour debugging the wrong end.

Measure the cold start against whatever startup-probe or health-check budget exists
before declaring it usable.

2. Always-on serverless instances

Serverless platforms bill very differently depending on whether CPU is allocated continuously
or only during requests. The "CPU always allocated" / "no CPU throttling" setting is what
switches a service to instance-based billing. A minimum-instance setting is a separate
control - it keeps instances warm and they accrue idle charges, but it does not by itself
change the billing mode.

Check: minimum instances and, separately, the CPU-allocation / billing setting. They are
two different things and conflating them produces wrong prices. A request-billed service
can still have minimum instances, and those idle instances still bill - at a lower idle rate,
on their own SKU. Cloud Run jobs are always instance-billed regardless.
Price: the instance-based SKU lines, split by region - and the request-billed idle
lines, which a search for "instance-based" alone will miss.

Trap 1: a service that is genuinely never idle saves ~nothing from min-instances=0,
because it would hold the instance anyway. Measure the real gap between requests first.

⚠ Trap 2, and this one can break production. Turning CPU throttling back on is
sometimes a large one-line win - and sometimes an outage. CPU-always-allocated exists so a
container can do work outside a request: background goroutines, async flushes, queue
consumers, retries and telemetry after the response is sent. Throttled, that work is
suspended between requests and may never finish.

Before changing it, establish that the service does nothing outside the request lifecycle

• from its code and its traces, not from its name. "Batch-style" is a guess about a
  service; it is not a property you can read off the billing export. If you cannot establish
  it, leave the flag alone and take the saving elsewhere.

3. Cluster and control-plane fees

Every cluster carries a fixed fee whether or not anything runs on it. Accounts
accumulate clusters - one per team, one for a migration, one from a marketplace install.

Check: the cluster count against the workload. Could two clusters hold everything?
Price: fee per cluster per month, plus the node pool for any cluster you can empty.
Trap: a free-tier allowance may cover one cluster's fee, which disguises the cost of
the others. Judge on gross (see the capped-pot rule).

4. Self-inflicted observability volume

Frequently the fastest safe win, because nothing reads most of it.

Check: which metric sources are enabled on your clusters, and which metric families
actually appear in dashboards or alerts. Per-container and per-node metric collectors can
produce an enormous sample volume for data nobody queries. Same for log sinks that
duplicate what another system already stores.
Price: samples or GiB ingested, converted at the published rate.
Trap: pricing models differ within observability, so one rule does not cover it.
Cloud Monitoring metrics are byte-priced with a monthly free allotment, so dollars
mislead and you should measure usage units (traps A3, A4). Managed Service for
Prometheus is priced per sample ingested and has no free allotment, so its dollars track
volume directly from the first sample. Check which one you are looking at before applying
either rule, and read usage.amount and usage.pricing_unit rather than assuming. Also confirm what reads the data before disabling: the answer is often
"one dashboard nobody opens", but occasionally it is an alert that matters.
Identify the source precisely. Group ingested samples by metric family before
blaming a component; it is easy to accuse the wrong collector and disable something
useful while the real volume continues.

5. Oversized managed databases and caches

Check: CPU and memory utilisation over 30 days against the provisioned tier; and
whether a managed cache could be an in-cluster pod.
Price: the tier delta.
Trap: managed services include failover, auth, TLS and backups. Moving a cache
in-cluster trades real money for real operational risk - a cold cache after any eviction
or upgrade, and a stampede onto the database behind it. Price the risk, do not just
price the instance.

6. Orphans

The purest wins: nothing breaks, because nothing is using them.

Check: unattached persistent disks; static/reserved IP addresses not bound to a
running resource (an address attached to a stopped instance still bills); old
snapshots and images; load balancers with no healthy backend; idle NAT gateways.
Price: full cost.
Trap: an address may be referenced by configuration even while unattached, and
releasing it means it cannot be reclaimed. Check for the address as a literal across the
codebase and secrets before releasing. The safe move for an address that must keep its
value is to reserve it, not release it.

7. Artifact and image storage

Registries accumulate every image ever built and are rarely pruned.

Check: storage per repository and whether any cleanup policy exists.
Price: GB-month over the free allowance.
Trap: run the cleanup policy in dry-run first and read what it would delete.
Keeping "recent tags" is not enough - a rollback target may be old, and a digest pinned
in a manifest may not carry a tag at all.

8. Network: egress, connectors, inter-region

Check: egress by destination; serverless VPC connectors (they run minimum instances
of their own); cross-region and cross-zone traffic between chatty services.
Price: per-GB egress; per-instance connector cost.
Trap: connectors can be partly cancelled by a free-tier discount on their instance
type, so the realisable saving is uneven across them and removing one may shift the
discount rather than bank it. Price each one individually.

9. Scheduled jobs nobody reviews

Check: every cron/scheduler entry, its frequency, and whether its twin in another
environment is paused. A job running every minute in staging against a paid API is a
common silent cost.
Price: invocation count × downstream cost - usually the API calls dominate, not the
scheduler.

10. The model bill, once infrastructure is tidy

⚠ Know which side of the line each model bill sits on. Models you consume through Google

• Vertex AI, the Gemini API, and partner models such as Claude purchased via Vertex - are
  Google-billed and do appear in this export under their own SKUs. Models you buy directly
  from the vendor do not appear at all. So a "total AI spend" from the export alone can both
  understate (missing direct vendor invoices) and, if you then add those invoices without
  checking, double-count the partner models already in it. Enumerate the SKUs first, then
  add only what is genuinely absent, and say that you combined sources.

In AI-heavy accounts the model API quickly becomes the largest line, and infrastructure
optimization has a floor. Once the infra lever is spent, the remaining levers are
model tier, prompt caching, output-token discipline, and not making the
call at all.

Treat this as demand-side work and measure it per unit of useful output (per request,
per job, per customer), not per month - a monthly total conflates price and volume and
will tell you a cheaper model "did nothing" in a month when usage grew.

references/monthly-review.md

Mode A - the monthly review

A recurring ~30-minute pass. Its job is not to find savings - that is Mode B. Its job
is to answer four questions honestly and quickly:

1. What did we actually bill, and did it move?
2. Did anything we previously fixed come back?
3. Did anything new appear?
4. Is there one thing worth doing this month?

Run it after the month is fully settled - usually a few days in. Running it on the 1st
produces a confident wrong answer (trap A1).

The loop

0. Settle gate - settle-gate.sql

Check the flat-rate control reads its known gross value for every day of the month being
reviewed, and note the first day where it does not. Exclude that day and everything after it.
If the month is not settled, stop and come back - do not "adjust for lag".

Then do the half people skip. A passing control says the day is not obviously partial;
it does not certify it, because services export on their own schedules. Put the per-service
export_lag_p95 from your calibration note next to this month's spend by service, and for any
service that is both slow and material, check its own daily series before quoting it. A
service that arrives late and bills ~nothing can be ignored; one that arrives late and is a top
line cannot.

1. Invoice reconciliation - invoice-reconcile.sql

Last complete month vs the one before, as the user will see it on the invoice. Use
assets/queries/invoice-reconcile.sql, which differs from every other query here on
purpose:

• it groups on invoice.month, not on a usage date - a row can land on a different
  invoice than its usage date implies, because late-arriving usage is billed on the next one
• it does not filter cost_type, because tax and adjustments are on the invoice

regular gross → tax → adjustments → credits → invoice total

Quote the invoice figure, not a usage-date sum. People reconcile against the invoice, and
a report that does not match it gets discarded - and the two genuinely differ.

Note if tax looks incomplete (it is exported late) and say so rather than quietly
under-reporting.

2. Run rate - sku-run-rate.sql, credit-shape.sql

The last settled 7 days, on gross, normalised ×30.44. This is what next month looks
like if nothing changes - more useful than the month just closed, because mid-month
changes are only partly reflected in a monthly total.

Then re-apply credits deliberately rather than by one subtraction, because a flat
"minus the pot" quietly contradicts the rules above:

• capped pot → subtract it once, and never below the eligible spend. If projected
  gross for that SKU is 50 and the allowance is 70, the answer is 0, not −20.
• proportional discount → scale with the projection, do not subtract a constant.
• tiered allowance → if the 7-day window sat inside the free allowance, extrapolating
  it projects $0 for a service that will start billing mid-month. Project the month's
  usage against the allowance, then price it.
• fixed monthly fees (cluster fees, subscriptions) → carry them as monthly constants
  rather than ×30.44 of a daily slice.

A single number is still fine to publish. Just build it from those four, and say which
input is doing the work.

Split demand-driven lines (model APIs, egress, anything that scales with usage) from
structural lines (instances, fees, storage). They behave differently and mixing them
makes both unreadable.

3. Step detection - month-over-month.sql, then by-project.sql, then step-detect.sql

For every service that moved more than ~10% or more than a material absolute amount,
pull the daily series and find the step date. Then name the cause. Three outcomes, all
acceptable, but say which:

• named - matched to a deploy, PR, config change or audit entry
• demand - no config change; volume moved
• unattributed - you could not name it. Write it down as unattributed rather than
  guessing; an unexplained step is itself a finding worth carrying forward.

4. New-or-resumed SKU check - new-skus.sql

The highest-value five minutes in the whole review, and the one most people skip.

List every SKU that was ~zero last month and is not now. New lines are how costs
start, and they are invisible in a service-level month-over-month view because they are
small at first and buried inside a service that already had spend.

Report them as "this was reliably zero until date D" and resist annualising a lumpy new
series (trap A8).

5. Regression sweep

Re-verify that previous savings are still in force, by reading the live
configuration, not the repo:

• things set to zero replicas - still zero?
• disabled components - still disabled?
• deleted resources - still absent?
• minimum instances / CPU-allocation flags - unchanged?

Declarative tooling re-applies committed state, and a change made by hand is one deploy
away from being undone. This check is why the monthly review earns its place: savings
decay silently.

6. One opportun...