# Make comparison chart more simple **URL:** \ **Category:** Uncategorized\ **Created:** [July 12, 2017, 11:17am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149 "2017-07-12T11:17:42Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 12, 2017, 11:17am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/1 "2017-07-12T11:17:42Z")
I find current comparison chart a bit confusing. I think we don’t need all results displayed from comparing benchmark. Maybe only last commit result from benchmark we want to compare with is enough on graph. This could be simply plotted as one line graph and would be much more clear what is being compared. You can get the feel about differences between benchmarks but it is not that clear which commit results are being compared. I propose something like this : ![](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/319c6a993635fad3074b56b9398bc98505a07e0d.png) This is much simpler in every way. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 12, 2017, 9:56pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/2 "2017-07-12T21:56:43Z")
I think there are few things we need to do first, before we even consider this: - Restrict “comparison targets” why am I allowed to compare 🍎s to 📙s At a minimum display the relevant comparison at the top. - With “prepared statement” vs “without” is just a wash and just causing confusion. My first question here is: Do we have any bench where the difference is significant? If not just suppress one of them for now or add a checkbox at bottom of graph to select prepared vs not. - This looks confusing: ![](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/51a69fbf3482d17a47d88201e21ba7f65c3b47c8.png) When we are missing commit just render a straight dotted line after last result. I really want to keep a “proper” comparison looking at trends in both benchmarks over time, but we do need to clean up some usability here. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 13, 2017, 2:10am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/3 "2017-07-13T02:10:51Z")
Also, maybe a simple comparison UI to see all the comparison tests available? For example same comparison, one of the heavy active record ones, can run on various commits of Ruby. Then you can tell how much Ruby can help. What do you think @noahgibbs ? ---
**Author:** ![noahgibbs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/noahgibbs/32/91_2.png) [@noahgibbs](https://community.rubybench.org/u/noahgibbs)\ **Post date:** [July 13, 2017, 4:08am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/4 "2017-07-13T04:08:26Z")
Certainly being able to compare across Ruby versions could be nice. Especially with some of the JIT stuff we’re looking at for near-future Ruby versions, there could be all kinds of effects on ActiveRecord… Even a lot of the GC work might be quite relevant. The usability stuff you mention (restricting comparisons, etc.) sounds good in concept. I’m not sure if there’s a simple way to group the tests for comparison, though. ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 13, 2017, 8:05am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/5 "2017-07-13T08:05:56Z")
Grouping comparable benchmarks is something that definitely needs to be done. I want to make it possible through admin UI, since I think it’s the most convenient way to do it. So you would have comparison candidates from same group in dropdown. I haven’t found any benchmark which results significantly differ whether with or without prepared statements. So I agree that displaying both produces bit of a noise. I will manage that. Oh yeah, that strange issue occurs when benchmark doesn’t have enough results to display. I will make it display minimum of last results these two benchmarks have. > I really want to keep a “proper” comparison looking at trends in both benchmarks over time I’ve assumed when we want to compare benchmark A to benchmark B we are only interested in looking benchmark A over time. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 13, 2017, 2:02pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/6 "2017-07-13T14:02:58Z")
> [@bmarkons](#): > > I’ve assumed when we want to compare benchmark A to benchmark B we are only interested in looking benchmark A over time. Nahh I want to look at multiple trends. At the moment I can select up to 2000 commits, what I would like to do is be able to select a time frame there as well (1 year, 2 years, 3 years, 5 year), then we can pick say 2000 commits in the N year period and graph them. At that point we can compare **trends**. And show for example when AR worked on perf, compared to when Sequel did. Additionally, we will easily be able to tell when time is invested in “catching up” on performance debt. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 20, 2017, 10:42pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/7 "2017-07-20T22:42:47Z")
An news here? I really want something I can tweet out at people 😊 ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 21, 2017, 8:34am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/8 "2017-07-21T08:34:28Z")
Have a look @sam 🙂 [https://rubybench.org/rails/rails/commits?result\_type=activerecord/postgres\_scope\_all&compare\_with=sequel/postgres\_scope\_all](https://rubybench.org/rails/rails/commits?result_type=activerecord/postgres_scope_all&compare_with=sequel/postgres_scope_all) Right now I am working on [grouping benchmarks](https://github.com/ruby-bench/ruby-bench-web/issues/232) so we have only appropriate benchmarks in select box for comparison. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 21, 2017, 9:29pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/9 "2017-07-21T21:29:09Z")
Nice! Are these benches correct? Is AR really that close to Sequel for this benchmark? [https://rubybench.org/rails/rails/commits?result\_type=active\_record/postgres\_discourse&display\_count=500&compare\_with=sequel/postgres\_discourse](https://rubybench.org/rails/rails/commits?result_type=active_record/postgres_discourse&display_count=500&compare_with=sequel/postgres_discourse) Seems a bit odd to me cause we were seeing huge discrepancies in other benches. cc @jeremyevans ---
**Author:** ![jeremyevans](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/jeremyevans/32/102_2.png) [@jeremyevans](https://community.rubybench.org/u/jeremyevans)\ **Post date:** [July 21, 2017, 11:43pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/10 "2017-07-21T23:43:28Z")
I didn’t look in detail, but it may be just that the query is more complex than other benches and most of the time is spent in the database and not in ruby. Sequel does allocate less than a third of objects that AR does, and is 10% faster, but if the database is the bottleneck, then the performance difference between Sequel and AR is likely to be small. Thanks, Jeremy ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 24, 2017, 11:21am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/11 "2017-07-24T11:21:53Z")
I’ve come across an odd issue while I was trying to reproduce these results locally. After I ran both benchmarks directly on my machine (not in docker) I got Sequel 2x faster than ActiveRecord: - sequel/postgres\_discourse - 312 ips - activerecord/postgres\_discourse - 162 ips While running same benchmarks in a standard way (inside docker) gave me smaller differences in results: - sequel/postgres - 198 ips - activerecord/postgres - 153 ips I am super confused with this. Maybe we are missing something in setup. I have verified benchmark correctness (same SQL query and same string generated) by running it directly on local machine. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 24, 2017, 3:11pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/12 "2017-07-24T15:11:49Z")
Very interesting, How is the DB setup? You have to make sure you run a volume for the DB on the host, otherwise you pay a huge price on filesystem access, also be sure to disable fsync and other stuff … see: > for some of our speed hacks. ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 25, 2017, 10:02am UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/13 "2017-07-25T10:02:41Z")
In our current setup DB is being run in separate linked container. > So I need to make these DB containers use volumes on the host? ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 25, 2017, 1:32pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/14 "2017-07-25T13:32:59Z")
I think by default it runs a volume on the host. Maybe do a “raw” benchmark as well then you can compare all 3 which will give you a great picture of what is going on ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 25, 2017, 1:52pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/15 "2017-07-25T13:52:47Z")
Do we want the same setup for pg gem (and mysql2) like for AR and Sequel - tracking over time? [https://github.com/ruby-bench/ruby-bench-suite/pull/76#issuecomment-314076173](https://github.com/ruby-bench/ruby-bench-suite/pull/76#issuecomment-314076173) ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [July 25, 2017, 2:06pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/16 "2017-07-25T14:06:54Z")
I guess this is the correct thing to do, but keep in mind I have seen very very little perf movement in these gems, they are pretty much as fast as they are going to be. You got to get something in asap though to fully debug this. ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [July 25, 2017, 2:39pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/17 "2017-07-25T14:39:45Z")
Sounds reasonable. I will work towards having raw benchmark running in docker setup to be able to debug this. Later we can setup automatic run on repo push if we want. ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [August 9, 2017, 4:35pm UTC](https://community.rubybench.org/t/make-comparison-chart-more-simple/149/18 "2017-08-09T16:35:41Z")
I did some setup change, I’ve decided to try with docker-compose. > Results I’ve got with this setup : - Raw : 230 ips - Sequel : 190 ips - ActiveRecord : 150 ips It seems overhead with AR is 2x bigger than with Sequel.