August 29, 2003

Cache efficiency for the SPEC 2000 benchmarks 

 by Hans de Vries 

     Cache efficiency for the SPEC 2000 Benchmarks

The SPEC 2000 benchmarks are subject to much debate in the scientific community. Are they broken? Do they just depend on memory bandwidth?  Do they fit entirely in the cache?  The recent publication of new benchmarks for the hp server rx5670 gives us a chance to produce some metrics. This small server is a four processor machine with a single memory controller. The memory bandwidth is 6.4 GByte /second. We look at the scores for four different configurations:

1)  Single 1000 MHz Itanium 2 with 3.0 MByte L3 on Chip Cache     CINT2000     CFP2000

2)  Four    1000 MHz Itanium 2 with 3.0 MByte L3 on Chip Cache     CINT2000     CFP2000

3)  Single 1500 MHz Itanium 2 with 6.0 MByte L3 on Chip Cache     CINT2000     CFP2000

4)  Four    1500 MHz Itanium 2 with 6.0 MByte L3 on Chip Cache     CINT2000     CFP2000

We define the Cache efficiency here as 100% if four processors finish just as fast as a single processor. Cache efficiency is said to be "0%"  if four processors take four times as long to finish the benchmark: This means that the run-time is entirely determined by the throughput of the single memory controller. 

We give the performance ratio's for one and four processor configurations. The ratio should be 1.5 (1500/1000) if the application fits entirely in the caches. It should be higher than 1.5 if it fits better in the 6.0 MByte cache than in the 3.0 MByte cache. The ratio is lower than 1.5 if the memory controller becomes a bottleneck. A ratio of 1.0 effectively means that the performance is entirely determined by the memory controller throughput: 1000 MHz processors run just as fast as the 1500 MHz processors. Some small differences are due fact that a newer version of the compiler is used for the 1500 MHz Itanium 2 configurations

We'll see that we get very different results for the Integer and Floating Point benchmarks.

    Cache efficiency for  Integer SPEC 2000 benchmarks.

*) different compiler versions are used for the 1000 MHz and 1500 MHz results

     Cache efficiency for  Integer SPEC 2000 benchmarks.

The caches seem to work quite well for the integer benchmarks. 10 out of 12 benchmarks have a cache efficiency of 98% or higher for the 3.0 MByte caches. This becomes 11 out of 12 for the 6 MByte cache (97%+). The Integer benchmarks do a lot of work on the data they load into the cache. Tasks likes Compilation and Compression have a high re-use of the data they work on. Caching works fine here even though the memory footprints as given by the SPEC committee are much larger than the cache itself.

Benchmark 181.mcf becomes number 11 of the highly efficient ones. We see performance improvements of 2.74 x and 2.86 x for the one and four processor versions. The only one left (254.gap) becomes actually less efficient, 92% to 85%. Four processors running this benchmark simultaneously are hindered more by the throughput of the single memory controller then they benefit from the increased cache size.  

     Cache efficiency for Floating Point SPEC 2000 benchmarks.

*) different compiler versions are used for the 1000 MHz and 1500 MHz results

      Cache efficiency for Floating Point SPEC 2000 benchmarks.

We see a very different picture for the Floating Point benchmarks however. Only 3 out of 14 have a cache efficiency of 97% and higher for the 3.0 MByte cache.  This becomes 5 out of 14 for the 6.0 MByte cache. Numbers 4 and 5 are the "infamous" 179.art and 178.galgel. 179.art becomes entirely cache resident in the 6.0 Megabyte cache. 178.galgel has a memory footprint much larger then the cache. The re-use of data however makes if very efficient in a 6 Megabyte cache. 

Most of these Scientific and Technical benchmark operate on large data-structures. The re-use becomes very high if a significantly part of the data structure fits into the cache.  For instance: If the data-structure is a 3D-volume then it may be enough  to hold three planes in the cache. This because new values are often calculated with the directly surrounding points:

New Data [ i, j, k ]  = Function { Data [ i-1, j-1, k-1 ] ..... Data [ i+1, j+1, k+1 ]   }

The re-use of data then becomes 26 / 27 =  96% where 27 is the number of points in the 3x3x3 sub-cube, only one new point needs to be loaded from memory if the 3x3x3 sub-cube moves a single position through the volume.

We see that 2 benchmarks are completely bandwidth starved at 1500 MHz for a single processor: 171.Swim and 172.mgrid

8 out of 14 benchmarks become less cache efficient if we go from 1000 MHz / 3 MB cache Itanium 2's to the newer 1500 MHz/ 

6 MB Itanium 2's  The Memory Controller bottleneck more then cancels the advantage of the larger caches.

If you're looking to use the Itanium 2 for Large Scale scientific or Technical calculations then you'll have to look at SGI with is propriety memory controllers. SGI uses two SHUB Memory Controllers for every four processors. The fully loaded 64 processor Altix 3000 has a 65% higher throughput then the half loaded 32 processor Altix 3000.

The memory footprint of the SPEC2000 benchmarks is less then 200 MByte to be able to run on systems with 256 MByte DRAM. Heavier applications using multiple Gigabyte structures are likely to see much greater degradations. AMD's distributed memory solution based on HyperTransfer links is likely to pay of in these cases. A four processor 2200 MHz Opteron may reach a similar SPEC2000_rate performance as a four way 1500 MHz Itanium 2 even though the latter has a much higher single processor score.  Again, larger floating point memory footprints may skew the results even further.    

    Regards,  Hans