# ==Debugging graphics== without guessing <p class="doc-sub">// status: seedling</p> Graphics bugs are unusually good at making a wrong program look plausible. A stale buffer can look correct for three frames. A missing barrier can disappear when a debugger slows the GPU down. A colour-space error can be dismissed as “art direction”. The cure is a small scientific loop: state one hypothesis, capture evidence at the stage where it could fail, change one variable, and keep the smallest reproducible scene. This is the workflow I want beside [[OpenGL - learning log]], [[Vulkan - learning log]], [[Compute shaders]], and [[Deferred vs forward rendering]]. It also applies to a generated [[Marching Cubes]] mesh, a culling pass from [[GPU-Driven Visibility for Large Worlds]], or the reticle in [[Ray Picking Through the Rendering Pipeline]]. ## Start at the first wrong stage The frame is a pipeline, not a single image: ```text CPU inputs → resource upload → shader compilation / pipeline state → draw or dispatch → intermediate images and buffers → lighting / post-process → UI → presentation ``` When the screen is wrong, find the earliest stage that is wrong. If a mesh viewer shows the right vertices but the final colour is black, stop investigating vertex generation. If the indirect command has `indexCount = 0`, a fragment shader cannot be the cause. | Symptom | First evidence to inspect | Common causes | |---|---|---| | nothing appears | draw call, viewport, depth state, index count | wrong transform, cull winding, empty indirect command, bad scissor | | geometry is inside-out | mesh viewer, face culling, normal view | index winding, negative scale, transposed matrix | | geometry flickers | depth/overdraw view, repeated captures | z-fighting, missing barrier, uninitialised data, temporal history | | one object is missing | bounds / culling overlay, object ID | too-small bound, wrong space, stale residency, Hi-Z false occlusion | | colours are wrong | texture format, sRGB state, intermediate attachment | double decode, wrong channel order, HDR range, tonemap placement | | lighting is wrong | normal/roughness/depth debug views | tangent basis, normal encoding, non-linear depth, G-buffer mismatch | | only another GPU fails | validation output, shader reflection, spec rules | undefined behaviour, layout mismatch, vendor-specific tolerance | | frame is slow | CPU/GPU timestamps, pass timings, counters | synchronization stall, bandwidth, overdraw, shader divergence | ## Capture a frame Use a frame debugger such as [RenderDoc](https://renderdoc.org/) on a reproducible frame. The names vary by tool, but the useful sequence is stable: 1. Capture after the bug is visible, and also capture a known-good frame if possible. 2. Find the first draw or dispatch that touches the bad result. 3. Inspect pipeline state: shaders, resource bindings, viewport/scissor, rasterization, depth/stencil, blend, and render targets. 4. Inspect the input mesh or buffer before the call. 5. Inspect the output attachment immediately after the call. 6. Use the mesh/texture viewer, pixel history, and shader debugger for one suspicious pixel or primitive. 7. Save the capture with the exact binary/shader assets used to produce it. Do not begin by staring at the final image. The event browser and resource history tell you whether the value was never written, overwritten later, or sampled with the wrong interpretation. For a generated mesh, inspect `caseIndex`, `triangleCount`, `offsets`, and the indirect command as well as the vertex buffer. For GPU culling, inspect the visible count and one offending bound. For TAA, inspect current colour, velocity, reprojected UV, and history rejection separately. ## Make invisible data visible The fastest graphics debugger is often a deliberately ugly shader. Add toggles for: - normals mapped from `[-1, 1]` to `[0, 1]`; - depth as linear eye distance, not raw hardware depth; - albedo, roughness, metallic, and ambient occlusion channels individually; - motion vectors as a signed colour wheel; - UVs as a repeating grid; - object / primitive / chunk IDs as flat colours; - culling state (`frustum`, `occluded`, `resident`, `LOD`) as categorical colours; - triangle winding and face-normal direction; - overdraw or light count; - NaN and infinity detection. A tiny fragment helper catches “black because NaN” bugs: ```glsl bool bad(float x) { return isnan(x) || isinf(x); } bool bad(vec3 x) { return bad(x.x) || bad(x.y) || bad(x.z); } vec3 debugFinite(vec3 colour) { return bad(colour) ? vec3(1.0, 0.0, 1.0) : colour; } ``` For buffers, render a selected range as a line or read it back into a small CPU-side diagnostic structure. A `printf`-style GPU facility can help on some APIs and tools, but a structured debug buffer with one record per selected invocation is more portable and less noisy. ## API validation before visual debugging ### OpenGL Create a debug context and enable `GL_KHR_debug` / the core debug output path. Install `glDebugMessageCallback`, label objects with `glObjectLabel`, and insert groups around passes with `glPushDebugGroup` / `glPopDebugGroup`. Filter known-benign messages only after reading them. `glGetError` is useful as a coarse assertion, but it does not explain shader memory hazards, stale state, or most undefined behaviour. OpenGL state is global and sticky. At the start of a suspicious pass, print or label the state you rely on: framebuffer, viewport, scissor, depth test/write, blend equation, cull mode, bound VAO, program, textures, images, SSBOs, and indirect buffers. Direct State Access reduces accidental binds; it does not make an old blend state disappear. ### Vulkan Enable `VK_LAYER_KHRONOS_validation` in development, including synchronization validation and GPU-assisted validation when the target and build support them. Give objects debug names and use command-buffer labels so validation and captures identify “Marching Cubes emit” rather than `VkBuffer 0x1234`. Validation catches invalid usage; it cannot prove that a frustum plane has the sign you intended. Treat a clean validation log as “the API contract is probably obeyed”, then use a capture and debug views for algorithmic correctness. Keep validation enabled in CI smoke tests even if a release build disables it. Shader toolchains deserve their own checks: ```text GLSL/HLSL source → compiler warnings → SPIR-V / native shader → reflection (bindings, offsets, formats) → validation → capture ``` For Vulkan shaders, run a SPIR-V validator in the build and compare reflected descriptor layouts with the host structures. A `vec3` padding mismatch, wrong descriptor set, or stale specialization constant is easier to catch before the frame reaches the GPU. ## Synchronization bugs: draw the dependency graph If a pass writes a resource and a later pass reads it, write the edge down: ```text compute classify: cases / counts (shader writes) → scan: counts (shader reads, offsets writes) → emit: cases / offsets / field (shader reads, vertices writes) → indirect draw: vertices / commands (vertex + indirect reads) ``` For each edge, identify the execution scope and memory visibility scope. A workgroup `barrier()` cannot order two dispatches. A shader memory barrier inside one invocation cannot make a later command see a buffer unless the API-level dependency is also present. Conversely, a huge global barrier can hide a missing ownership transition while destroying overlap. In OpenGL, choose `glMemoryBarrier` bits for the later access (`GL_SHADER_STORAGE_BARRIER_BIT`, `GL_VERTEX_ATTRIB_ARRAY_BARRIER_BIT`, `GL_COMMAND_BARRIER_BIT`, and so on). In Vulkan, specify source/destination stages and access masks for the actual producer and consumer, plus image layout transitions and queue-family ownership where applicable. Keep a debug mode that emits one pass at a time; if removing a pass makes the bug disappear, its resource edge is the next suspect. The classic diagnostic is to add a CPU wait or a debugger capture. If that “fixes” the frame, suspect synchronization or lifetime, not that the shader needed more time. The slowdown changed the schedule; it did not make an invalid dependency valid. ## Coordinate, format, and depth sanity checks Many visual bugs are convention mismatches: - OpenGL default NDC z is `[-1, 1]`; Vulkan uses `[0, 1]`. - Pixel origins and viewport y orientation are API/configuration dependent. - Hardware depth is non-linear under perspective. - Normal maps and G-buffer normals need a documented tangent/encoding convention. - sRGB textures should be decoded once and written to a correctly configured target. - `std140`, `std430`, and host-language structs have different alignment rules. - A matrix's transpose and multiplication order depend on the math library convention. Put a known calibration object in the scene: an axis triad, a unit cube, a checkerboard, and a near/far depth ramp. If the cube is the wrong size, the bug is in transforms or units; if only the checkerboard is wrong, investigate UVs/filtering; if depth ramps backwards, inspect projection/reversed-Z state before touching lighting. ## Performance without superstition Measure CPU and GPU separately. A CPU timer around `submitFrame` mostly measures how quickly commands were queued; a GPU timestamp around a pass measures execution but can be invalidated by clock changes or a disjoint event. Use per-pass timestamps, a short warm-up, several frames, and a stable camera path. Record resolution, feature switches, GPU model, driver, and validation state. Break cost into: - **front-end / submission** — command count, pipeline changes, descriptor updates; - **vertex / geometry** — transform, culling, vertex bandwidth; - **raster / overdraw** — covered samples and depth rejection; - **fragment / compute** — arithmetic, texture latency, divergence; - **memory** — attachment bandwidth, cache misses, readback; - **synchronization** — waits, barriers, queue idle time. The debug view should answer the hypothesis. If the claim is “culling saved fragment work”, compare overdraw and GPU pass timings with culling on/off at the same camera, not only CPU frame time. If the claim is “the scan is expensive”, measure count, scan, and emit independently and vary the dirty-cell count. ## A repeatable bug template Keep a small issue note with: ```text Observation: what is wrong, in one sentence. Expected invariant: the value or relationship that must hold. First bad stage: earliest pass where the invariant fails. Capture: file, frame/event, GPU/driver, shader revision. Hypothesis: one explanation, not a list of ten. Test: one controlled change. Result: evidence for / against the hypothesis. Fix: code and validation that prove it. Regression case: smallest scene that would fail again. ``` This sounds formal until a driver-specific flicker returns six weeks later. A capture and a one-line invariant beat a page of “it looked weird near the mountain.” ## Things that tripped me up - **The final image is a lossy witness.** Inspect intermediate attachments and buffer contents at the first bad event. - **A debugger changes timing.** Treat “works under RenderDoc” as evidence for a race or lifetime bug, not a fix. - **Validation is necessary but not sufficient.** API validity does not establish the math or visual algorithm. - **State leaks across draws.** Label stateful resources and set every state that a pass relies on. - **Black often means NaN.** Add explicit finite-value debug colours before changing lighting constants. - **Raw depth lies about distance.** Linearise it using the same projection and reversed-Z convention as the renderer. - **Readback can hide a GPU bug.** A synchronous readback serialises the pipeline; use it for diagnosis, then restore the asynchronous path and keep the test. - **One vendor is not a specification.** Test at least one stricter or differently scheduled implementation before relying on undefined behaviour. - **Performance conclusions need a controlled scene.** Changing resolution, validation, and camera motion at once makes the result uninterpretable. ## References - [RenderDoc documentation and user guide](https://renderdoc.org/docs/) - [Khronos Vulkan validation layers and best practices](https://github.com/KhronosGroup/Vulkan-ValidationLayers) - [Vulkan Guide — validation and debugging (Khronos)](https://docs.vulkan.org/guide/latest/) - [OpenGL `KHR_debug` extension specification (Khronos)](https://registry.khronos.org/OpenGL/extensions/KHR/KHR_debug.txt) - [OpenGL 4.6 Core Specification — shader memory access and synchronization (Khronos)](https://registry.khronos.org/OpenGL/specs/gl/glspec46.core.pdf) - [Vulkan API Specification — synchronization and memory dependencies (Khronos)](https://registry.khronos.org/vulkan/specs/latest/html/) - [SPIR-V Tools — validation and optimization (KhronosGroup)](https://github.com/KhronosGroup/SPIRV-Tools) - [NVIDIA Nsight Graphics — frame debugging and profiling](https://developer.nvidia.com/nsight-graphics) --- Back to [[Notes/Index|Notes]] · see also [[Compute shaders]] · [[OpenGL - learning log]] · [[Vulkan - learning log]] · [[GPU-Driven Visibility for Large Worlds]] · [[Ray Picking Through the Rendering Pipeline]]