sTauB
2005-07-28, 13:41:48
1. Teil der Übersetzung zu: Average fps - wirklich das Maß aller Dinge?
http://www.3dcenter.org/artikel/average_fps/
There are facts which should not be touched. Different natural laws are one example for this category. In the area of PC’s the benchmarking obtains a similar role – more precisely said the final result of each benchmark: The average value of all frames.
For measuring the performance of today’s hardware the reader usually gets a picture by the average fps value. In the benchmarking scene most important is only who will be first with the highest value. This victory looses its actual value if the playability (above 40fps) of a game is not achievable.
According to the definition of “average fps” the maximum and minimum framerates will get lost by the average calculation. The unfair difference is: Very high fps are usually not recognized by players, however, low fps will leave a negative impression. If the fact is omitted, that the average fps result delivers a value of 40fps, the performance of the PC will be degraded dramatically according to impressions while playing the game.
A scene with a low number of explosions and environmental details might deliver far above 100fps. However, scenes which inherits the players full concentration and controllability might fall below critical 20fps. The final benchmark result, the average frames per second, will not be changed noticeable by these effects. An additional benchmarking problem is that in many cases the CPU and GPU load will be different if compared to a real game. These benchmarks have a lack of the usual AI and physical calculations, which are always present during gameplay.
Maybe it has come to the time to change some common rules. By how much do current average-fps-results illustrate the playability with a certain hardware? If the average fps value falls by 20%, caused e.g. by Anti-Aliasing”, does this also mean that the playability will decrease by the same amount? Should a benchmark tester not give more attention to the playability of a game?
If the answer is “Yes”, then this article will hold a possible solution. Instead of the overall average value, the absolute minimum fps are also measureable, which give a much better clue about the playability of a game; and this at every possible graphical resolution.
Additionally, this article will demonstrate the influence of the CPU for the most common settings of the Radeon X800 XT-PE (1280x960 - 4xFSAA – 16xAF). For comparison and orientation purposes, which should illustrate the load distribution between CPU and GPU, two other settings have also been taken into account: 1024x768 (1xFSAA – 1xAF) and 1600x1200 (6xFSAA – 16xAF).
As during benchmarks with Doom3 and Far Cry the graphics card will limit quite early there have been additional settings: 1280x1024 - 16xAF for Doom3 and 1280x960 2xFSAA – 16xAF for Far Cry to show graphics settings with a more or less equal distribution of GPU and CPU load.
As the complete framehistory of each benchmark can be made visible, the maximum and minimal framerates are also obtainable besides the average framerates. However this article will mainly deal with the minimal framerate, which will be calculated from the three lowest fps values. For this the protocol function of Fraps helped in obtaining the single fps values.
At this point a short intervention for the theoretical-statistical background of the minimum framerate: Minimum framerates have already been used for benching games for a longer time, however there are only a few articles which used these values. That minimum framerates are one of the essential values for benchmarking is well-known among hardware enthusiasts.
Nevertheless a very common mistake in most cases of minimum framerate benchmarking to use the result from these benchmurk programs as a comparable value. Most benchmark programs only deliver the lowest fps value originated by only one single picture. An average fps result is also only one number, which is however calculated from hundreds or thousands frames. From a statistical point of view it is not a wise decision to trust the value of a single frame – this would be the same if only one person will be interviewed for the question on Sunday.
Also wouldn’t this single value express really, what the [Anhänger] of the minimum framerate theory would expect to get. This is caused by the fact that a single value can only produce a statement how fast (or slow) a game would be in tenth a second. As such a short lag is usually not critical it is more important how slow the game is in e.g. its slowest 3 seconds.
With this [2 fliegen mit einer klappe]: At first ……..
Measurements have been taken out according to this scheme, therefore the “minimum frames” always represent the average of the worst three seconds – and not a absolute single value.
Testsystem:
Unfortunately it was not known during the testing phase of this article how the benchmarks could be recorded in the original speed and not in the system-performance dependent speed in the case of Doom3, Unreal Tournament 2004 and Far Cry.. Therefore the diagrams for these games will be stretched according to the actual time needed for the benchmark
Seite2
While the average fps decrease from 1024x768 to 1280x1024 with 16xAF by 9%, it is only 2% among the minimum framerates. If additionally 4xAA is being activated at these resolutions the average frames decrease by 32% while the minimum framerate does this by 45%.
The differences between different resolutions and settings are according to the minimum frame results by approximately 10% higher but show roughly the same result as with the average framerates. However, with 1280x104 16xAF 4xAA there is a significant different between average framerates (36% decrease) an minimum fps (52% decrease). As Doom3 even at 1280x1024 4xAA 16xAA is already only limited by the graphics card there are no significant differences between the minimum and average framerates for both resolutions.
http://www.3dcenter.org/artikel/average_fps/
There are facts which should not be touched. Different natural laws are one example for this category. In the area of PC’s the benchmarking obtains a similar role – more precisely said the final result of each benchmark: The average value of all frames.
For measuring the performance of today’s hardware the reader usually gets a picture by the average fps value. In the benchmarking scene most important is only who will be first with the highest value. This victory looses its actual value if the playability (above 40fps) of a game is not achievable.
According to the definition of “average fps” the maximum and minimum framerates will get lost by the average calculation. The unfair difference is: Very high fps are usually not recognized by players, however, low fps will leave a negative impression. If the fact is omitted, that the average fps result delivers a value of 40fps, the performance of the PC will be degraded dramatically according to impressions while playing the game.
A scene with a low number of explosions and environmental details might deliver far above 100fps. However, scenes which inherits the players full concentration and controllability might fall below critical 20fps. The final benchmark result, the average frames per second, will not be changed noticeable by these effects. An additional benchmarking problem is that in many cases the CPU and GPU load will be different if compared to a real game. These benchmarks have a lack of the usual AI and physical calculations, which are always present during gameplay.
Maybe it has come to the time to change some common rules. By how much do current average-fps-results illustrate the playability with a certain hardware? If the average fps value falls by 20%, caused e.g. by Anti-Aliasing”, does this also mean that the playability will decrease by the same amount? Should a benchmark tester not give more attention to the playability of a game?
If the answer is “Yes”, then this article will hold a possible solution. Instead of the overall average value, the absolute minimum fps are also measureable, which give a much better clue about the playability of a game; and this at every possible graphical resolution.
Additionally, this article will demonstrate the influence of the CPU for the most common settings of the Radeon X800 XT-PE (1280x960 - 4xFSAA – 16xAF). For comparison and orientation purposes, which should illustrate the load distribution between CPU and GPU, two other settings have also been taken into account: 1024x768 (1xFSAA – 1xAF) and 1600x1200 (6xFSAA – 16xAF).
As during benchmarks with Doom3 and Far Cry the graphics card will limit quite early there have been additional settings: 1280x1024 - 16xAF for Doom3 and 1280x960 2xFSAA – 16xAF for Far Cry to show graphics settings with a more or less equal distribution of GPU and CPU load.
As the complete framehistory of each benchmark can be made visible, the maximum and minimal framerates are also obtainable besides the average framerates. However this article will mainly deal with the minimal framerate, which will be calculated from the three lowest fps values. For this the protocol function of Fraps helped in obtaining the single fps values.
At this point a short intervention for the theoretical-statistical background of the minimum framerate: Minimum framerates have already been used for benching games for a longer time, however there are only a few articles which used these values. That minimum framerates are one of the essential values for benchmarking is well-known among hardware enthusiasts.
Nevertheless a very common mistake in most cases of minimum framerate benchmarking to use the result from these benchmurk programs as a comparable value. Most benchmark programs only deliver the lowest fps value originated by only one single picture. An average fps result is also only one number, which is however calculated from hundreds or thousands frames. From a statistical point of view it is not a wise decision to trust the value of a single frame – this would be the same if only one person will be interviewed for the question on Sunday.
Also wouldn’t this single value express really, what the [Anhänger] of the minimum framerate theory would expect to get. This is caused by the fact that a single value can only produce a statement how fast (or slow) a game would be in tenth a second. As such a short lag is usually not critical it is more important how slow the game is in e.g. its slowest 3 seconds.
With this [2 fliegen mit einer klappe]: At first ……..
Measurements have been taken out according to this scheme, therefore the “minimum frames” always represent the average of the worst three seconds – and not a absolute single value.
Testsystem:
Unfortunately it was not known during the testing phase of this article how the benchmarks could be recorded in the original speed and not in the system-performance dependent speed in the case of Doom3, Unreal Tournament 2004 and Far Cry.. Therefore the diagrams for these games will be stretched according to the actual time needed for the benchmark
Seite2
While the average fps decrease from 1024x768 to 1280x1024 with 16xAF by 9%, it is only 2% among the minimum framerates. If additionally 4xAA is being activated at these resolutions the average frames decrease by 32% while the minimum framerate does this by 45%.
The differences between different resolutions and settings are according to the minimum frame results by approximately 10% higher but show roughly the same result as with the average framerates. However, with 1280x104 16xAF 4xAA there is a significant different between average framerates (36% decrease) an minimum fps (52% decrease). As Doom3 even at 1280x1024 4xAA 16xAA is already only limited by the graphics card there are no significant differences between the minimum and average framerates for both resolutions.