Orchestrate GPU training jobs across cloud platforms end-to-end

Remote GPU Job Orchestration treats renting a GPU box as being a short-term tenant on someone else's machine: the job is to detach the work, make the result outlive the instance, and stop the meter safely - not provision a cluster. It is platform-agnostic at the core (10 operating principles plus a 6-phase lifecycle) and platform-specific at the edges, with one profile per platform (profiles/<platform>.md) owning the exact paths, proxies, billing verbs, and spot semantics for AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms, or a bare SSH/Slurm/K8s box.

Use it for deploying, training, monitoring, or troubleshooting a long-running job on a rented instance you don't own - training, eval, ablation sweeps, batch inference, large data processing, single or multi-instance. It is explicitly not for local single-GPU training or in-box multi-GPU DDP (use torchrun/accelerate directly), managed multi-cloud price-shopping (use SkyPilot's own skill), or zero-ops serverless inference (use Modal); a companion skill, verifying-dl-experiments, owns whether a resulting metric is actually true, while this skill only owns running the job.

The ten invariants include: minimize paid wall-clock by smoke-testing on CPU before renting; trust artifacts you loaded, not log lines claiming success ("synced/saved/done" can lie under a silently-failed write); know exactly which storage mount survives a stop versus a terminate per platform, since the data you need often lives on the volatile one; storage fails on inodes before bytes, so monitor df -i not just df -h; and checkpoint-to-durable-location plus unconditional load-latest-on-startup is the one mechanism surviving an SSH drop, Slurm walltime kill, K8s reschedule, or spot preemption. The 6-phase workflow - environment audit, SSH/credentials, wrapper-plus-CPU-smoke gate, detached launch, durable four-layer monitoring, and aggregate-verify-teardown - ends each phase in a runnable check, and an "Iron Law" blocks any release/terminate/destroy action until checkpoints are pulled to local and verified by load, with explicit user approval, since most platforms' meter-stopping action is irreversible.

up (rent+reach) -> push (code/data) -> run (detached+checkpointing)
-> watch (durable monitor) -> pull (results+verify) -> down (stop meter)

A per-platform quick-reference table tracks what survives stop vs. destroy, spot grace windows, and mirror requirements for AutoDL, RunPod, vast.ai, Lambda, and Chinese platforms - AutoDL is the sole exception where "stop" (关机) keeps the data disk. A separate references/training/ layer covers debugging the training run itself (OOM, distributed hangs, precision/NaN, throughput, checkpoint-resume bugs, convergence, data-pipeline correctness) once the box is actually running. It cross-references several separately-installed companion skills: verifying-dl-experiments for metric validity, huggingface-skills:hf-cli for resumable transfer verbs, and superpowers:dispatching-parallel-agents for independent ablation fan-out across instances.

Developers and researchers renting metered GPU hardware for training or large batch jobs who need the job to survive SSH drops, spot preemption, and disk exhaustion, and who need to stop paying the meter without losing results - especially on platforms (Chinese cloud rentals, bare SSH boxes) the mainstream orchestrators don't cover.