Brie Doom Sources Docs

Performance

This page presents and discusses the performance of the port.

While contracts provide a way to find bugs in the code, they are checked in runtime. This has a performance cost.

To measure the performance impact we run the game several times with different sets of contractss enabled. However, we should run the program with the same inputs each time. To achieve this we can use Doom’s demo mechanism.

A demo is a record of game inputs per each frame. We can record the demo once and play several times with different executables having different contracts enabled. We have recorded a demo of playing the first level and we will use it as an example in the following sections.

To measure the performance we will measure the time it took to run the game logic and draw the result on the screen for each frame during the gameplay. The multiplicative inverse of this measure gives the more commonly used measure called frame rate expressed in frames per second (FPS).

Now we know what to measure, let’s discuss the how.

Removing contracts from the code

Game development focuses mainly on the fun of the game. This means that during development, the developed game is play-tested often.

Since contract checking has a performance cost, we need to run the game without contract checking. This can be achieved by compiling without the -keep parameter: 

ec -finalize -c_compile

Running the finalized version with assertions discarded gives the following times to render each frame:

no_contracts Produced by GNUPLOT 5.2 patchlevel 2 0 10000 20000 30000 40000 50000 60000 70000 0 200 400 600 800 1000 1200 1400 Time (microseconds) Frame no_contracts wipe 1 wipe 1 level 1 level 1 wipe 2 wipe 2 intermission 1 intermission 1 wipe 3 wipe 3 level 2 level 2 35 FPS reference 35 FPS reference

The demo starts with the initial wipe sequence (wipe 1), then the first level is played (level 1). When the level exit button is pressed, there is another wipe sequence (wipe 2) and the level intermission screen is drawn (intermission 1). After another wipe (wipe 3) the next level starts (level 2).

The 35 FPS reference line shows the ideal time to render each frame to achieve the intended 35 FPS of the original game. We see that all sections are below the reference line, so game with contracts disabled is performant enough.

We see that wipes and intermissions are much cheaper to draw than the level gameplay. Because of that all next plots will show only the level section of the first level.

Keep assertions, but do not check them

In the next sections we will measure the impact of different types of contracts. For that we will need to add the -keep parameter when compiling. In this section we will compare the performance of the program with assertions removed from the executable (no -keep) and with assertions kept in the executable but not executed (with -keep). 

ec -finalize -keep -c_compile
keep_vs_no_keep Produced by GNUPLOT 5.2 patchlevel 2 10000 20000 30000 40000 50000 60000 70000 80000 100 200 300 400 500 600 700 800 900 1000 1100 Time (microseconds) Frame keep_vs_no_keep No Keep No Keep Keep Keep 35 FPS reference 35 FPS reference

We can see that just enabling -keep without enabling the checks decreases the performance 2.06 times (average FPS goes down from 54 to 26).

This is because the -keep parameter enables the collection of some debug information and disables some optimizations.

Enabling assertions will decrease the performance even further. This means that playtesting the game with contracts enabled will have a very noticeable performance degradation, which can make such testing infeasible.

Check all contracts

In this section we will compare the cost of checking all contracts present in the game.

all Produced by GNUPLOT 5.2 patchlevel 2 0 500000 1x10 6 1.5x10 6 2x10 6 2.5x10 6 3x10 6 100 200 300 400 500 600 700 800 900 1000 1100 Time (microseconds) Frame all No Keep No Keep Keep, disable Keep, disable Keep, enable Keep, enable 35 FPS reference 35 FPS reference

The plot shows that checking all contracts has a high cost. This decreases the performance 21.0 times (26 FPS with contracts compiled but disabled compared 1.24 FPS with contracts compiled and checked).

Contract impact per cluster

In this section we will compare the impact of enabling contracts per cluster. This means that all contracts are disabled except the contracts of one specific cluster.

per_cluster Produced by GNUPLOT 5.2 patchlevel 2 0 200000 400000 600000 800000 1x10 6 1.2x10 6 1.4x10 6 1.6x10 6 1.8x10 6 100 200 300 400 500 600 700 800 900 1000 1100 Time (microseconds) Frame per_cluster math math pointers pointers render render root root sound sound status bar status bar 35 FPS reference 35 FPS reference

This plot shows that contract checking of clusters render and math have the biggest impact on the performance. Indeed, render cluster is responsible for the 3D rendering which is the most computationally expensive part of the game. The math cluster is responsible for the implementation of the fixed point arithmetics which is used throughout the game.

Let us zoom into the other clusters:

per_cluster2 Produced by GNUPLOT 5.2 patchlevel 2 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000 550000 100 200 300 400 500 600 700 800 900 1000 1100 Time (microseconds) Frame per_cluster2 pointers pointers root root sound sound status bar status bar 35 FPS reference 35 FPS reference

Out of the clusters present in this plot the most expensive cluster is the root cluster which is responsible for the main game logic. Most notably this includes the collision detection (did something bump into an obstacle), line of sight checks (can someone see something or is there an obstacle blocking the sight).

Conclusion

Game development is often done iteratively: implement something, test it, keep if it is fun to play. With the focus being on the fun, less time is left for correctness of the code.

While Design by Contract can help developers to develop correct programs, it does not replace testing. Sadly, it is currently not possible to integrate contract checking into playtesting without sacrificing performance for this project.

It is still possible to check contracts after the playtesting session by recording the inputs and replaying them later with contract checking enabled.

To speed things up, it is possible to disable contracts for parts of the system which are shown to be more or less bug-free. If the development of the core 3D rendering engine is finished and was tested, why test it again? We can shorten the feedback loop by checking contracts of the actively developed parts of the program.