IBM and HP BW-EML Benchmark Result Analysis

Posted by Alyce Lindquist on August 24, 2013

More by this author

by Boris Gelman

I noticed that IBM has recently submitted results in the SAP NetWeaver BW Enhanced Mixed Load (EML) benchmark. This benchmark was created by the SAP Performance Benchmarking team almost two years ago to more accurately reflect how customers are using SAP BW today. As a result, the benchmark includes ad-hoc queries on detailed data (DSO’s) as well as cubes, and simultaneous data loading at a rate of 10 delta loads every five minutes throughout the testing. In short, the BW EML benchmark comes much closer than past BW benchmarks to representing real-world BW scenarios.

The first BW EML result came in almost a year and a half ago, submitted by HP on an HP AppSystem for SAP HANA, and I think it’s worth taking a few minutes to compare these results. I worked very closely with the HP team and am familiar with the configuration, as well as the execution of the benchmark. I have been less familiar with the recently submitted results from IBM on DB2 for iSeries, but I will take a few minutes to make some points that jump out at me.

Comparison #1: Data Volume

The HP results tested a BW scenario with 1 Billion rows of data, whereas the IBM results submitted utilized only 500 Million rows – half the data. As any BW administrator knows, data volume is probably the single largest factor in query and load performance. The more data a query has to navigate, the more rows must be stored in secondary indexes and materialized query tables, the longer maintenance operations take, the longer loads must run, the more performance is potentially impacted in an adverse manner.

The HP on HANA result had twice the data but does not require secondary indexes, materialized query tables, aggregates, or any such tuning mechanisms to achieve fast results. This leads to a simpler, less maintenance-intensive configuration, while still maintaining query and load performance.

To analyze the two performance results on more equal footing, one method is to apply what is commonly known as data size scaling factors. Given we have 2 varying dataset sizes, one with 500 million records in IBM system and another with 1 billion records in HP system, we can apply scaling factor of 5 to the IBM results and the scaling factor of 10 to the HP results to help normalize the performance comparison by varying size:

  • HP – 65,990 x 10 = 659,990 navigation steps per hour
  • IBM – 66,900 x 5 = 334,500 navigation steps per hour

This shows that the HP result is actually two times better when data size is taken into consideration.

Comparison #2: CPU Performance

As someone that has worked with IBM servers in the past, I can vouch for their performance. So it was no surprise to me to see that, according to most measures of performance, be it SAPS ratings or LINPACK or SPEC benchmarks for example, the recently-released IBM Power7+ 4.06GHz has advantages over the Intel Xeon E7-4870 (Westmere EX), which has been out for almost three years now.

Not only does the Power7+ run at a higher clockspeed than the Xeon (4.06GHz vs 2.4GHz), have 10MB of L3 cache per core vs the Xeon¹s 3MB of L3 cache per core (30MB per socket), but it is also capable of four threads per core vs the Xeon’s two threads per core, a fact which allows for much more throughput for enterprise applications. Calculations can vary dramatically based on workload and function, but I have seen comparisons claiming the Power7+ 4.06 is from two to four times faster than the Xeon E7-4870 2.4.

Comparison #3: Two-Tier vs. Three-Tier

As every Basis Administrator, DBA or System Administrator knows, combining an application server onto the same physical hardware as it’s database server allows for a dramatic reduction in networking overhead between application and database. This is why the vast majority of SAP SD benchmarks submitted are two-tier, not three tier. (In fact, IBM has not submitted any three-tier SD benchmarks on their flagship Power platform since 2008)

The fact that the IBM benchmark result was a two-tier result, and the HP result was a three-tier result is a significant one that would make a meaningful impact on performance, and should not be overlooked.

Summary

As we know, the benchmarks only become valuable when multiple vendors participate with multiple platforms. I still believe that to be the case, and I am glad that IBM stepped up to the plate with these results.

I look forward to more results, running on both HANA and other database platforms, in the near future.

VN:F [1.9.22_1171]
Average User Rating
Rating: 4.0/5 (1 vote cast)
IBM and HP BW-EML Benchmark Result Analysis, 4.0 out of 5 based on 1 rating

8601 Views