Authors:
Linda Null ,julia Lobur
Chapter:
Performance Measurement And Analysis
Exercise:
Exercises
Question:7 | ISBN:9780763704445 | Edition: 3
Question
7. What would you say to a vendor that tells you that his system runs 50% of the SPEC benchmark kernel programs twice as fast as the leading competitive system? Which statistical fallacy is at work here?
Answer
The statement made by the vendor about their system running 50% of the SPEC benchmark kernel programs twice as fast as the leading competitive system is an example of the "Simpson's Paradox" statistical fallacy.
- Simpson's Paradox occurs when a trend or pattern observed in different groups of data reverses or disappears when the groups are combined. In this case, the fallacy arises because the vendor is comparing the performance of their system on different subsets of the benchmark programs rather than considering the overall performance across all the programs.
- To evaluate the claim accurately, it is important to examine the performance of both systems on the entire SPEC benchmark suite, rather than only focusing on specific subsets of programs. By cherry-picking specific subsets of programs where their system outperforms the competition, the vendor may be presenting a skewed or misleading perspective.
Hence, it is essential to request comprehensive and unbiased performance data that considers the performance across the entire range of benchmark programs to make a fair comparison between the systems.