Debugging graphics without guessing

// status: seedling

Graphics bugs are unusually good at making a wrong program look plausible. A stale buffer can look correct for three frames. A missing barrier can disappear when a debugger slows the GPU down. A colour-space error can be dismissed as “art direction”. The cure is a small scientific loop: state one hypothesis, capture evidence at the stage where it could fail, change one variable, and keep the smallest reproducible scene.

This is the workflow I want beside OpenGL - learning log, Vulkan - learning log, Compute shaders, and Deferred vs forward rendering. It also applies to a generated Marching Cubes mesh, a culling pass from GPU-Driven Visibility for Large Worlds, or the reticle in Ray Picking Through the Rendering Pipeline.

Start at the first wrong stage

The frame is a pipeline, not a single image:

1
2
3CPU inputs → resource upload → shader compilation / pipeline state
→ draw or dispatch → intermediate images and buffers
→ lighting / post-process → UI → presentation

When the screen is wrong, find the earliest stage that is wrong. If a mesh viewer shows the right vertices but the final colour is black, stop investigating vertex generation. If the indirect command has indexCount = 0, a fragment shader cannot be the cause.

Capture a frame

Use a frame debugger such as RenderDoc on a reproducible frame. The names vary by tool, but the useful sequence is stable:

1. Capture after the bug is visible, and also capture a known-good frame if possible.
2. Find the first draw or dispatch that touches the bad result.
3. Inspect pipeline state: shaders, resource bindings, viewport/scissor, rasterization, depth/stencil, blend, and render targets.
4. Inspect the input mesh or buffer before the call.
5. Inspect the output attachment immediately after the call.
6. Use the mesh/texture viewer, pixel history, and shader debugger for one suspicious pixel or primitive.
7. Save the capture with the exact binary/shader assets used to produce it.

Do not begin by staring at the final image. The event browser and resource history tell you whether the value was never written, overwritten later, or sampled with the wrong interpretation.

For a generated mesh, inspect caseIndex, triangleCount, offsets, and the indirect command as well as the vertex buffer. For GPU culling, inspect the visible count and one offending bound. For TAA, inspect current colour, velocity, reprojected UV, and history rejection separately.

Make invisible data visible

The fastest graphics debugger is often a deliberately ugly shader. Add toggles for:

• normals mapped from [-1, 1] to [0, 1];
• depth as linear eye distance, not raw hardware depth;
• albedo, roughness, metallic, and ambient occlusion channels individually;
• motion vectors as a signed colour wheel;
• UVs as a repeating grid;
• object / primitive / chunk IDs as flat colours;
• culling state (frustum, occluded, resident, LOD) as categorical colours;
• triangle winding and face-normal direction;
• overdraw or light count;
• NaN and infinity detection.

A tiny fragment helper catches “black because NaN” bugs:

1
2
3
4
5
6bool bad(float x) { return isnan(x) || isinf(x); }
bool bad(vec3 x) { return bad(x.x) || bad(x.y) || bad(x.z); }

vec3 debugFinite(vec3 colour) {
    return bad(colour) ? vec3(1.0, 0.0, 1.0) : colour;
}

For buffers, render a selected range as a line or read it back into a small CPU-side diagnostic structure. A printf-style GPU facility can help on some APIs and tools, but a structured debug buffer with one record per selected invocation is more portable and less noisy.

API validation before visual debugging

OpenGL

Create a debug context and enable GL_KHR_debug / the core debug output path. Install glDebugMessageCallback, label objects with glObjectLabel, and insert groups around passes with glPushDebugGroup / glPopDebugGroup. Filter known-benign messages only after reading them. glGetError is useful as a coarse assertion, but it does not explain shader memory hazards, stale state, or most undefined behaviour.

OpenGL state is global and sticky. At the start of a suspicious pass, print or label the state you rely on: framebuffer, viewport, scissor, depth test/write, blend equation, cull mode, bound VAO, program, textures, images, SSBOs, and indirect buffers. Direct State Access reduces accidental binds; it does not make an old blend state disappear.

Vulkan

Enable VK_LAYER_KHRONOS_validation in development, including synchronization validation and GPU-assisted validation when the target and build support them. Give objects debug names and use command-buffer labels so validation and captures identify “Marching Cubes emit” rather than VkBuffer 0x1234.

Validation catches invalid usage; it cannot prove that a frustum plane has the sign you intended. Treat a clean validation log as “the API contract is probably obeyed”, then use a capture and debug views for algorithmic correctness. Keep validation enabled in CI smoke tests even if a release build disables it.

Shader toolchains deserve their own checks:

1
2GLSL/HLSL source → compiler warnings → SPIR-V / native shader
→ reflection (bindings, offsets, formats) → validation → capture

For Vulkan shaders, run a SPIR-V validator in the build and compare reflected descriptor layouts with the host structures. A vec3 padding mismatch, wrong descriptor set, or stale specialization constant is easier to catch before the frame reaches the GPU.

Synchronization bugs: draw the dependency graph

If a pass writes a resource and a later pass reads it, write the edge down:

1
2
3
4compute classify: cases / counts        (shader writes)
→ scan: counts                          (shader reads, offsets writes)
→ emit: cases / offsets / field          (shader reads, vertices writes)
→ indirect draw: vertices / commands    (vertex + indirect reads)

For each edge, identify the execution scope and memory visibility scope. A workgroup barrier() cannot order two dispatches. A shader memory barrier inside one invocation cannot make a later command see a buffer unless the API-level dependency is also present. Conversely, a huge global barrier can hide a missing ownership transition while destroying overlap.

In OpenGL, choose glMemoryBarrier bits for the later access (GL_SHADER_STORAGE_BARRIER_BIT, GL_VERTEX_ATTRIB_ARRAY_BARRIER_BIT, GL_COMMAND_BARRIER_BIT, and so on). In Vulkan, specify source/destination stages and access masks for the actual producer and consumer, plus image layout transitions and queue-family ownership where applicable. Keep a debug mode that emits one pass at a time; if removing a pass makes the bug disappear, its resource edge is the next suspect.

The classic diagnostic is to add a CPU wait or a debugger capture. If that “fixes” the frame, suspect synchronization or lifetime, not that the shader needed more time. The slowdown changed the schedule; it did not make an invalid dependency valid.

Coordinate, format, and depth sanity checks

Many visual bugs are convention mismatches:

• OpenGL default NDC z is [-1, 1]; Vulkan uses [0, 1].
• Pixel origins and viewport y orientation are API/configuration dependent.
• Hardware depth is non-linear under perspective.
• Normal maps and G-buffer normals need a documented tangent/encoding convention.
• sRGB textures should be decoded once and written to a correctly configured target.
• std140, std430, and host-language structs have different alignment rules.
• A matrix's transpose and multiplication order depend on the math library convention.

Put a known calibration object in the scene: an axis triad, a unit cube, a checkerboard, and a near/far depth ramp. If the cube is the wrong size, the bug is in transforms or units; if only the checkerboard is wrong, investigate UVs/filtering; if depth ramps backwards, inspect projection/reversed-Z state before touching lighting.

Performance without superstition

Measure CPU and GPU separately. A CPU timer around submitFrame mostly measures how quickly commands were queued; a GPU timestamp around a pass measures execution but can be invalidated by clock changes or a disjoint event. Use per-pass timestamps, a short warm-up, several frames, and a stable camera path. Record resolution, feature switches, GPU model, driver, and validation state.

Break cost into:

• front-end / submission — command count, pipeline changes, descriptor updates;
• vertex / geometry — transform, culling, vertex bandwidth;
• raster / overdraw — covered samples and depth rejection;
• fragment / compute — arithmetic, texture latency, divergence;
• memory — attachment bandwidth, cache misses, readback;
• synchronization — waits, barriers, queue idle time.

The debug view should answer the hypothesis. If the claim is “culling saved fragment work”, compare overdraw and GPU pass timings with culling on/off at the same camera, not only CPU frame time. If the claim is “the scan is expensive”, measure count, scan, and emit independently and vary the dirty-cell count.

A repeatable bug template

Keep a small issue note with:

1
2
3
4
5
6
7
8
9Observation: what is wrong, in one sentence.
Expected invariant: the value or relationship that must hold.
First bad stage: earliest pass where the invariant fails.
Capture: file, frame/event, GPU/driver, shader revision.
Hypothesis: one explanation, not a list of ten.
Test: one controlled change.
Result: evidence for / against the hypothesis.
Fix: code and validation that prove it.
Regression case: smallest scene that would fail again.

This sounds formal until a driver-specific flicker returns six weeks later. A capture and a one-line invariant beat a page of “it looked weird near the mountain.”

Things that tripped me up

• The final image is a lossy witness. Inspect intermediate attachments and buffer contents at the first bad event.
• A debugger changes timing. Treat “works under RenderDoc” as evidence for a race or lifetime bug, not a fix.
• Validation is necessary but not sufficient. API validity does not establish the math or visual algorithm.
• State leaks across draws. Label stateful resources and set every state that a pass relies on.
• Black often means NaN. Add explicit finite-value debug colours before changing lighting constants.
• Raw depth lies about distance. Linearise it using the same projection and reversed-Z convention as the renderer.
• Readback can hide a GPU bug. A synchronous readback serialises the pipeline; use it for diagnosis, then restore the asynchronous path and keep the test.
• One vendor is not a specification. Test at least one stricter or differently scheduled implementation before relying on undefined behaviour.
• Performance conclusions need a controlled scene. Changing resolution, validation, and camera motion at once makes the result uninterpretable.

References

• RenderDoc documentation and user guide
• Khronos Vulkan validation layers and best practices
• Vulkan Guide — validation and debugging (Khronos)
• OpenGL KHR_debug extension specification (Khronos)
• OpenGL 4.6 Core Specification — shader memory access and synchronization (Khronos)
• Vulkan API Specification — synchronization and memory dependencies (Khronos)
• SPIR-V Tools — validation and optimization (KhronosGroup)
• NVIDIA Nsight Graphics — frame debugging and profiling

Back to Notes · see also Compute shaders · OpenGL - learning log · Vulkan - learning log · GPU-Driven Visibility for Large Worlds · Ray Picking Through the Rendering Pipeline