A GPU can quietly lose most of its bandwidth and never throw an error. Utilization looks normal. Jobs are just slower, and stay slower for weeks.
example: pcie link at x4 on an x16 slot since boot, found three weeks in
Backed by
Your AI infra is always quietly degrading.
Harbor finds the cause and the fix.
Harbor traces slow and failed workloads across compute, network, and storage. Every answer comes with the evidence behind it.
Runs entirely inside your cluster.
Incidenttrain-run-4471throughput -31%
Workload
samples/sec 1,840 → 1,270
Node
node-14 · 8x H100
Device
gpu-03 · pcie root
Link state
negotiated x4 @ 8GT/srated x16 @ 16GT/s
The failure pattern
The symptom is never on the layer that caused it.
A multi-node run dies minutes after the actual fault. By the time anyone looks, the component that caused it reads healthy again and the evidence is gone.
example: nccl timeout at 02:14, thermal history normalized by 02:24
Storage read latency creeps up and nothing alerts, because storage is technically up. Meanwhile the GPUs it feeds sit idle and the team debugs the wrong layer.
example: read latency 4x baseline, gpus starved on the two nodes it feeds
Teams debug these by elimination: “it’s not this, not this, not this, so it must be that.” Harbor exists so nobody has to debug by elimination.
How it works
From a slow run to the component that caused it.
01
See everything on one panel.
Every node, GPU, NIC, switch port, and storage device, with the numbers that matter: negotiated versus rated link speed, per-port errors, storage read latency, thermals. One health view instead of a wall of graphs.
Fleet overview12 nodes · healthy
node-03
8x H100
util 82%
node-04
8x H100
util 76%
node-05
8x H100
util 91%
node-06
8x H100
util 68%
node-07
8x H100
util 74%
node-08
8x H100
util 88%
node-09
8x H100
util 79%
node-10
8x H100
util 84%
node-11
8x H100
util 71%
node-12
8x H100
util 80%
node-13
8x H100
util 77%
node-14
8x H100
util 78%
02
Alerts that end in answers.
When a workload degrades, Harbor alerts you the way anything else would. The difference is what arrives with it: the causal chain, from the slow run down to the component behind it, with the readings at every hop. This one ends at a link running below its rated width, a number most dashboards never surface.
Incident diagnosisnode-14
- Workloadtrain-run-4471throughput -31% vs baseline
- Nodenode-148x H100
- Devicegpu-03 · pcie rootroot complex 0000:4a:00
- Link statenegotiated x4 @ 8GT/srated x16 @ 16GT/s
03
From diagnosis to resolution.
Every diagnosis ends in a recommended fix: a config change or a physical action, named. When the workload and the hardware belong to different parties, Harbor routes the incident to the side of the boundary that owns it, so it does not stall between two teams. After the fix is applied, Harbor verifies it took. Every action stays operator-gated: Harbor recommends, your people decide.
Ownership routingnode-14
node-14 · gpu-03 · pcie x4Physical fault
Config / ML
Software team
tenant side
Physical / hardwareOwns fix
Operator / provider
whoever can touch the machine
Named before the incident crosses a company boundary, so it does not stall between two teams that each assume it is the other’s.
Sample diagnosis
One catch, end to end.
The incident from the top of the page, at full depth. Step through how Harbor got from a slow run to a verified fix.
Incident diagnosis · node-14hop 1 / 6
01 · Workloadtrain-run-4471 → node-14
a run slows. the regression isolates to one node out of twelve.
- samples/sec, baseline
- 1,840
- samples/sec, now
- 1,270
- candidate nodes
- 12 → 1
A replay of one diagnosis, hop by hop, with the readings Harbor used at each step.
Trust model
Evidence, not scores.
Every Harbor diagnosis shows its work. When telemetry proves the cause, Harbor asserts it and shows the exact readings that do. When proof is not there, Harbor hands your team the evidence and a shortlist of suspects, so the search starts from signal instead of a guess. No percentages attached to either.
gpu-03 · pcieAsserted
Link width negotiated below rated. Cause proven by direct readings.
LnkSta x4 · LnkCap x16confirms cause
nccl · intermittentUnconfirmed
The window closed before proof. Harbor preserved the evidence and narrowed the field, so the team starts with the two likeliest suspects.
timeout events / 24h3
pre-fault thermal historypreserved
shortlistnic-1 link · switch port 12
Signature library
Things Harbor catches that dashboards don’t.
Harbor ships with a growing library of failure signatures, each grounded in the telemetry that proves it. A few of them, placed on the layer where they live:
One engine across the stack
- Workloadtraining throughput regression traced to hardware · inference p99 climbs, the model gets blamed, the cause is a layer down
- Frameworkcollective timeout, pre-failure history preserved
- Driverxid event patterns preceding job failure
- GPUecc error accumulation · thermal asymmetry, one hot one cold
- PCIe & fabriclink below rated width · device off the bus mid-job
- Networkbandwidth degradation via crc and drop trends
- Storageread latency growth starving gpus downstream
And when a failure matches no known signature, Harbor still assembles the cross-layer evidence, so the search starts from signal.
Who it’s for
Built for teams on the hook for a GPU fleet.
Teams running their own clusters
Racks you built for throughput and privacy, bare metal rented from a datacenter, reserved neocloud capacity: owned or rented, if you operate the GPUs, their failures are yours. You did not hire a datacenter team to babysit them. Harbor is the single health panel and the diagnosis engine your two-person infra function needs.
Regulated & air-gapped
When the cluster lives somewhere you cannot SSH into (sovereign deployments, export-controlled programs, client-operated sites), Harbor runs entirely inside the environment, produces auditable diagnoses, and tells each side of the boundary what is theirs to fix. Nothing leaves the room.
GPU cloud providers
Your hardware is your revenue. Harbor gives you continuous per-node health verification and bare-metal-level diagnosis, so a device falling off the bus is your finding, not your customer’s, and hardware quality becomes something you can prove, not promise.
Works with what you have
Grafana shows you gauges. Harbor tells you why.
Harbor sits above the telemetry you already collect and adds the layer none of it is: causal diagnosis across the stack. Keep your dashboards. Stop needing to stare at them.
Your telemetry
GPU telemetry · DCGM, nvidia-smiHost & node exportersPrometheusFabric counters · Ethernet, InfiniBandStorage metricsBMC / IPMI
Harbor
Causal diagnosis across the stack
Incident resolved
node-14 · gpu-03 · pcie x4
Deployment
Built to pass your security review.
The checklist your technical gatekeeper screens with, answered up front.
The full detail for your security review- Fully self-hostedExecutable plus Helm / Terraform, deployed by your team.
- Zero data egressRuns air-gapped. No telemetry leaves the environment.
- Read-only against your clusterEvery action stays operator-gated. Harbor diagnoses, you decide.
- Uses your existing instrumentationDCGM, Prometheus, IPMI / BMC where you grant it, at diagnostic-grade frequency.
- Kubernetes and SlurmFits the scheduler you already run.
See it work
Stop debugging by elimination.
Book 30 minutes. We’ll run a live diagnosis and show you how it deploys.