Hibernate Team Posted Several Batoo JPA vs Hibernate Benchmarks - Proving Batoo Is Fast

Well, a short while ago the Batoo JPA was announced on TheServerside.com. In parallel an article was published on HighScalability.com. Both sites created a very good attention. HighScalability.com has become the host Hibernate team posted benchmarks that provided evidence in favor of their new competition.

As expected, Hibernate Team was so much surprised to see a product as in the same league as their product comes along with a bold promise - Over 15 times faster then Hibernate.


Starting with current Hibernate Lead Steve Ebersole they quickly ran the benchmarks and saw some numbers that suggested him Batoo was only twice as fast then Hibernate. Content with what he sees, he failed to note that the number he was looking into was including the times spent by the database, and failed to look at the numbers where Hibernate and Batoo JPA developers make a difference. When the calculations are made, he has seen the truth that what he actually posted was in line with Batoo JPA's statement - Over 15 times faster at JPA layer and twice as fast including the database. Having been informed of the true interpretation of the numbers he published, he started to base his argument on insults rather then facts.

That's where Steve Ebersole stepped out and fellow Hibernate Developer Anthony Patricio came along with assertions on the benchmark being too much batch oriented - a fair argument. Then Batoo modified the benchmark and  re-ran it. The numbers did not change much, still over 15 times faster... Then they advocated that the three settings were needed to 'tune' Hibernate to do basic CRUD operations while Batoo JPA needed no tuning at all be it in several entities persisted / updated in a single transaction or single parent entity persisted in a single transaction.

Then Anthony put the blame on using an in memory database. While for an experienced JPA developer changing a datasource in spec complaint persistence.xml is a piece of cake, he refused to run the tests as is. Meanwhile Batoo JPA released a JSF application 'HelloJSF', along with ready to use mysql-persistence.xml that used a 'proper database'. Again looking at HelloJSF connected to JBoss ExampleDS, he again refused to run the application which has JMeter test along and proves that the very same JSF application with presentation -> business -> persistence -> RDBMS layers, Batoo JPA clocked twice as many requests with plain switch from default JBoss Hibernate JPA. The funny thing is a datasource cannot be provided within an application as it is a JTA environment. He again could configure the ExampleDS to point at the database of his choice. He simultaneously asserted that the Batoo JPA's great speed is thanks to using a ByteCode instrumentation. Explaining that the Batoo JPA's speed is archived by line by line code written clean, with performance in mind, and applying the best practices along with ByteCode instrumentation, he seemed to back on that idea as it was never brought up afterwards. But in between the lines he was offended by Batoo JPA is being that fast the Hibernate was badly coded - so seems like he was more of defending his name rather then Hibernate.

He then claimed that Batoo JPA was lacking the L2 Cache. A small argument on the cache issue, he then agreed that "about second level cache, it's a false friend but many people will use it just to boost some benchmark results". He kept on arguing that using an in-memory database is just as wrong. Obviously he wanted to add a bit more complexity so that the high resources spent by JDBC serialization would overshadow heavy weight Hibernate middleware.

Then again to convince the community, Batoo JPA released a Spring Application where the datasource could be coded into Spring application - HelloRest. This application demonstrated a REST service that interfaced a CRUD and Query service that is hardwired to external MySQL database. Even in README, the project suggested that a remote MySQL should be set up for the sake of best similarity to a real world example. A Comprehensive JMeter test case was provided, so that we did not count on Batoo counters. Again Hibernate team to date, has not run the tests.

Instead, Anthony took a different approach and wanted to take the argument to homeland, issued a blog article Decrypting another JPA benchmark. In that article, he refused to base his arguments on the built-in profiler which indeed utilizes the Sun JVM built in profiling support that the VisualVM profiler also uses. But this profiler profiles the benchmark down to line level and with 10ns snapshots. He also refused to employ a respected profiler, nor industry standard benchmarking tool - JMeter. Rather he counted on 'vmstat' to count the precious CPU cycles. That alone explains the bias and performance perception of the Hibernate Team. Even the argument was based on response time rather then throughput + response time. He tought taking the argument to his own blog would pay off and he censored my comments. Bad call!

He posted 5 tests that he has done, but what was unfortunate that he taken the DB times into the calculation and failed to calculate the ratios. The calculated ratios were:

Format:  (Hibernate Seconds x CPU Load) / (Batoo Seconds x CPU Load)

Shoot 1: (13 * 200) / (9 * 210) = 137.6%
Shoot 2: (44 * 64) / (36 * 55) = 142.2%
Shoot 3: (301 * 29) / ( 292 * 22) =135.9%
Shoot 4: (990 * 6) / (1075 * 5) = 110.5%
Shoot 5: Numbers are not provided properly.

A second look into Shoot 4 revealed that rounding actually mattered a lot since the numbers are based on poor vmstat output. With the calculation below, obviously the Shoot 4 performance ration is somewhere between 110.5 ~  132.5%.

Say Hibernate CPU 6 is rounded down from 6.49 and Batoo CPU is rounded up from 4.51
Shoot 4: (990 * 6.49) / (1075 * 4.51) = 132.5%

On Batoo JPA Project, he issued "Is this is interesting ? Yes, Should we respect that ? Yes, Is this useful ? No, sorry this is not solving real performance issues at all." But his postings revealed otherwise. Hibernate alone spent half an RDBMS would consume. Uhh too much for a middleware right? We thought so a year ago.

He then concluded his findings "I don’t care these CPU optimizations I prefer FEATURES!". A quick look at the features revealed another important aspect. Hibernate had more then one third of the issues still open! Obviously Hibernate team was so much caught up with providing features that do not exist in JPA spec, (last time I checked, Hibernate was represented on JPA JCP Board, you smell an attempt to vendor lock-in like I do) but even all those features not only brought Hibernate performance down to its knees, but also left Hibernate users with ~3000 open issues. That being more then 1 / 3 of every issue submitted to Hibernate was never addressed. Again on that remark Anthony stormed with accusation of dishonesty and providing false numbers.


Upon provision of evidence from Hibernate JIRA, he is yet to comment on the issue. While he concluded his postings to features are what matters, he put the blame on extra features, which of course false - the ~2500 of ~3000 open issues were in the Hibernate  Core components.
While strongly advocating Hibernate performance was good enough, a few performance related issues popped up  Hibernate JIRA post to Batoo JPA's debut.
We simply challenged Hibernate "Please prepare the test you like, I challenge Hibernate in every setup, in every application. Batoo JPA can beat Hibernate by far". This we will yet to see and no hesitation at all.

Now if you are ready to give Batoo JPA and boost your application to the performance it deserves, we invite you to take it out and test drive.