# GSOC 2017 - Long Running Ruby and Rails Benchmarks **URL:** \ **Category:** Uncategorized\ **Created:** [March 23, 2017, 9:35am UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130 "2017-03-23T09:35:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [March 23, 2017, 9:35am UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/1 "2017-03-23T09:35:28Z")
God day everyone! I am student interested in improving RubyBench during gsoc 2017. Before I submit application I would like to get clear understanding of current state of RubyBench project. I read a portion of this [discussion](http://community.rubybench.org/t/gsoc-project-improving-rubybench/99/1) with @shahsaurabh0605 regarding the same project last year. It would be helpful if any of you could summarize what was achieved during last summer, because top level goals are the same as last year. I tried to find project in gsoc 2016 archive but with no success. Cheers 🙂 ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [April 2, 2017, 12:31pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/2 "2017-04-02T12:31:18Z")
Hello bmarkons, So sorry for the late answer here ! I’m not sure whether you’ll have enough time to send a proposal but answering anyway to your question. Last year, it looks like shahsaurabh0605 implemented [Sequel benchmarks](https://github.com/ruby-bench/ruby-bench-web/pull/171) and also begun working on detecting regressions here : [Detecting benchmark regressions by shahsaurabh0605 · Pull Request #174 · ruby-bench/ruby-bench-web · GitHub](http://github.com/ruby-bench/ruby-bench-web/pull/174). If you have any ideas to improve Ruby Bench, feel free to put them in your proposal. At the very least, it looks like the last pull reference request needs some work. Also, it looks like there is [a lack for other Ruby implementations](http://community.rubybench.org/t/would-be-possible-to-add-truffleruby-to-the-benchmarks/124). Maybe @sam and @system could chime in ; even though you are not mentoring this year, this would be awesome to have your points of view here. 😊 Have a nice day and sorry again for the late answer ! Thanks ! ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [April 3, 2017, 12:57pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/3 "2017-04-03T12:57:28Z")
> [@GSOC Project : Improving RubyBench](http://community.rubybench.org/t/gsoc-project-improving-rubybench/99/7): > > Note, just going to mention this here. I think the bisecting stuff is very interesting and important, however feel that the most important thing is getting our benchmarks in order. I feel there is a huge gap around our Rails benchmarks, in particular have a look at say: [https://rubybench.org/rails/rails/commits?result\_type=activerecord/mysql2\_scope\_all\_with\_default\_scope&display\_count=200](https://rubybench.org/rails/rails/commits?result_type=activerecord/mysql2_scope_all_with_default_scope&display_count=200) What if this was implemented raw against the mysql gem, how fast would it be? How many objects would b… to me is still the #1 #2 and #3 issues I want sorted, our benchmarks are not honest enough, there is no way to tell how much performance is being left on the table. ---
**Author:** ![noahgibbs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/noahgibbs/32/91_2.png) [@noahgibbs](https://community.rubybench.org/u/noahgibbs)\ **Post date:** [May 5, 2017, 3:46pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/4 "2017-05-05T15:46:04Z")
Hey, Marko! Sure, I’m fine with having discussions here rather than in email 🙂 ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 5, 2017, 4:20pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/5 "2017-05-05T16:20:17Z")
Awesome! 🎉 This way anyone can drop some thoughts on subjects we will discuss. I read the portion of last summer gsoc topic and it helped me get the feel of project flow, so thats where idea of having conversation here comes from. Looking forward to ask first questions 🕶 ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [May 6, 2017, 2:01pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/6 "2017-05-06T14:01:23Z")
Hi there ! Congratulations Marko ! Try to be as comfortable as possible with the Ruby Bench code base before the coding period starts and obviously, feel free to ask us questions ! 🙂 Also posting to mention that if you want, we can open a Basecamp project to have a Campfire, a place for centralizing documents, to-dos, etc. That’s up to you to Noah and Marko, I don’t mind on my side and I think Jon doesn’t neither. ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 6, 2017, 3:33pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/7 "2017-05-06T15:33:43Z")
Thanks Robin 🙂 👍 for basecamp project. ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [May 7, 2017, 3:41pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/8 "2017-05-07T15:41:18Z")
Okay, fine, you should’ve received an invitation for the Basecamp project ! ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 9, 2017, 8:13pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/9 "2017-05-09T20:13:20Z")
I changed readme a bit in rubybench guideline repo which was quite poor 😟 Pls take a look [Make README more descriptive by bmarkons · Pull Request #23 · ruby-bench/ruby-bench · GitHub](https://github.com/ruby-bench/ruby-bench/pull/23) btw, I managed to run ruby benchmarks locally in docker container today 🎉 ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 11, 2017, 1:55pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/10 "2017-05-11T13:55:35Z")
What are prepared statements? I see results with prepared statements and without them. ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [May 11, 2017, 4:17pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/11 "2017-05-11T16:17:14Z")
Prepared statements are SQL statements that have placeholders in them to inject data. For example, this is a prepared statement: ```sql SELECT * FROM articles WHERE id = $1; ``` Here, `$1` is a placeholder for a given value (it will be replaced with 10 for instance). You may also see the `?` syntax for placeholders. Prepared statements are mainly used in Ruby on Rails to cache SQL queries as you often run the same queries but only with different values. This is also a good practice to avoid SQL injections as the value will be properly escaped (if you are dealing with user inputs for example). If you have a Rails application, looking at the server logs, you’ll see SQL prepared statements. Hope it’s clear enough, feel free to tell me if I’m not crystal clear ! 😃 ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 11, 2017, 4:47pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/12 "2017-05-11T16:47:51Z")
Thanks @robin850 🙂 So in performance aspect, prepared statements are expected to run faster because of caching behind, right? ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [May 11, 2017, 5:01pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/13 "2017-05-11T17:01:58Z")
Yes exactly, because Active Record won’t have to rebuild the query ; it just needs to inject the values. 🙂 ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 12, 2017, 8:49am UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/14 "2017-05-12T08:49:18Z")
I see we currently have benchmarks for rails, ruby, bundler, discourse and sequel in our suite. Is this suite supposed to support only certain gems or as many as possible? In case we are trying to support as many as possible with benchmarks, I was thinking if duplicating benchmarks to our suite (since benchmarks for ruby and discourse already exist on official repos) is good approach? The idea is that benchmarks stay on the official repos instead of copying it to ruby-bench-suite. Workflow example would be like, if you are gem developer and you want benchmarks to be executed you would just submit pull request with some configuration, and after ruby-bench approve it and merge it, benchmarks you wrote as a gem developer would be executed like any other. Do you think it is doable? Pls tell me if I’m missing something. 😬 ---
**Author:** ![robin850](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/robin850/32/29_2.png) [@robin850](https://community.rubybench.org/u/robin850)\ **Post date:** [May 12, 2017, 8:32pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/15 "2017-05-12T20:32:41Z")
Maybe @sam and @noahgibbs can answer you better than I would on the subject but I guess the goal is to support only major projects. The real problem isn’t benchmarks but running them somewhere and if Ruby Bench tries to support more and more projects, there may be scaling issues because the resources are a bit limited. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [May 12, 2017, 8:47pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/16 "2017-05-12T20:47:58Z")
> [@robin850](#): > > Yes exactly, because Active Record won’t have to rebuild the query ; it just needs to inject the values. Sort of, in the real world ™ any large install of Postgres will use [pgbouncer](https://pgbouncer.github.io/) in transaction pooling mode. This means that prepared statements simply are not an option. PG starts performing really inconsistently with tons of connections and Rails loves creating connections. > [@robin850](#): > > but I guess the goal is to support only major projects. Yes I would like to only support major projects here, and for the ORM tests Sequal, AR, Raw are a good enough trio. No need to add more for now. @bmarkons I am curious, did you write a few of the benchmarks in raw PG and Sequel yet (even 1 is a good start), how does performance compare on local? Let’s try to stay laser focused on getting a great answer to the question above. ---
**Author:** ![noahgibbs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/noahgibbs/32/91_2.png) [@noahgibbs](https://community.rubybench.org/u/noahgibbs)\ **Post date:** [May 12, 2017, 8:52pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/17 "2017-05-12T20:52:20Z")
I’m new to Ruby Bench too. But the way most benchmarking works is that you don’t want as much as possible - you want to focus on the things you consider important. Having a huge amount of stuff that doesn’t matter tends to clutter up your understanding. There’s a great talk by Matt Gaudet on how to benchmark Ruby 3 that may help: [[EN] Ruby3x3: How are we going to measure 3x? \/ Matthew Gaudet](https://www.youtube.com/watch?v=kJDOpucaUR4) What he’s getting at there is that your benchmarks should be broad enough to measure all the stuff you care about, but specific enough not to be too confusing. If we measured hundreds of different gems but they were all a kind of mix of Matt’s eight ideas, we wouldn’t be gaining anything new. ---
**Author:** ![sam](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/sam/32/5_2.png) [@sam](https://community.rubybench.org/u/sam)\ **Post date:** [May 15, 2017, 1:32pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/18 "2017-05-15T13:32:52Z")
5 posts were split to a new topic: [Rewriting the scope\_all benchmark in Sequel and Raw](https://community.rubybench.org/t/rewriting-the-scope-all-benchmark-in-sequel-and-raw/138) ---
**Author:** ![bmarkons](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/bmarkons/32/87_2.png) [@bmarkons](https://community.rubybench.org/u/bmarkons)\ **Post date:** [May 14, 2017, 7:38pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/19 "2017-05-14T19:38:25Z")
> Let’s try to stay laser focused on getting a great answer to the question above. @sam Maybe I sounded wrong, though I tried to understand rubybench project direction rather than do it atm 🙂 Now I know that the plan is to support only major projects in near future 👌 > there may be scaling issues because the resources are a bit limited @robin850 yeah, it seems like scaling would be an issue 😰 @noahgibbs thanks for this great talk. I guess Matt was talking about measuring certain number of different gems for the purpose of measuring ruby 3 performance. I had in mind rubybench as a platform where gem developers are getting feedback on gem performance they are developing. Thank you guys for explanation 🙂 ---
**Author:** ![noahgibbs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/noahgibbs/32/91_2.png) [@noahgibbs](https://community.rubybench.org/u/noahgibbs)\ **Post date:** [May 14, 2017, 9:27pm UTC](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130/20 "2017-05-14T21:27:21Z")
I don’t know of any current plans to use Ruby Bench for that. But we’d definitely like it as a resource for the Ruby and Rails core teams to see regressions quickly and track them down accurately. Presumably also one or two other major projects like Discourse 😉 [Next page](https://community.rubybench.org/t/gsoc-2017-long-running-ruby-and-rails-benchmarks/130.md?page=2)