Solving a GPU driver crash in Unreal Engine 5 (part 1)

Epic games has released Unreal engine 5 with many great features for users. At iRender, we also receive many love from Unreal engine 5 users because of our powerful GPU as well as the intuitive workflow. However, there are some issues related to GPU driver crash in Unreal Engine 5 and we will try to help you along with our users to troubleshoot in this article.

Sometime when you work in a GPU-intensive project, a GPU crash could happen. At that time, a window like this is likely to pop up.

Then, you might see a Unreal Engine Crash Reporter window like this.

When an application use too much memory, Windows, with its auto-protection mechanism, will kill the GPU driver. This leads to an application crashed.

If a GPU crashes, you will get some callstack and logs such as “GPUCrash – exiting due to D3D device being lost – D3D Hung” and “DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG.”

There are many reasons for a GPU crash, such as:

• • • • Running out of memory (OOM)
      • Times out while doing an expensive operation (TDR event)
      • A bug in the engine code
      • A bug in the driver
      • A bug in the Operating System
      • A problem in the hardware

Because there are various reasons, it’s really hard to detect which is the main cause of Unreal Engine crashing. In the following of the article, let’s see some debugging options and some preventative measures you can take.

You can use the call stack generated by Crash Reporter and log files to see what causes a normal crash in Unreal. However, if a GPU crash occurs, it does not really point out the reason behind. Therefore, it’s kind of pointless to see the call stack.

To output logs with a more useful information, you can use two debugging command line arguments:

• • • • -gpucrashdebugging collects GPU progress and tracks the current GPU state when debugging GPU crashes.
      • -d3ddebug provides information about the D3D pipeline.

After running the engine with each of these command lines (you should run them separately) and getting the crash, you can find the logs in [Your Project]/Saved/Logs folder.

The first reason for the GPU crash could be it runs out of memory. You can easily check the usage of GPU memory with the Windows Task Manager > Performance tab.

In here, you can click to select your GPU (1) and see its available memory and how much it is currently consuming (2).

You can watch the GPU memory amount being consumed and how many GPU are available. If you are close to the GPU limit, it can cause the crash. You can try the following methods to troubleshoot:

• • • • Close other programs that may be consuming large amounts of GPU memory.
      • Simplify the scene using lower resolution textures, lower resolution meshes, culling to reduce objects in the scene, and so on.
      • Use a lower screen resolution.
        • While working in the editor, you can use the Level Viewport Screen Percentageto render at a lower resolution.
      • While working in the editor, if you have multiple viewports open, close all but one.
      • Avoid disabling primary features like Niagara or Ray Tracing.
        • Bypassing these components changes many things, which could lead to invalid conclusions as to the cause of the GPU crash.

In the part 2, we will continue to explore GPU Time Outs Caused by Expensive Operations and Investigate Bugs in the Engine Code, Drivers and Operating Systems

Source + image: docs.unrealengine.com