# GSOC Project : Improving RubyBench **URL:** \ **Category:** Uncategorized\ **Created:** [March 13, 2016, 1:22pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99 "2016-03-13T13:22:00Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [June 9, 2016, 6:40pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/41 "2016-06-09T18:40:33Z")
I am currently going through the current ActiveRecord framework. Do I need to completely transform this into Sequel model? Also, I did not understand what `sam` meant by this: "I feel there is a huge gap around our Rails benchmarks, in particular have a look at say: [https://rubybench.org/rails/rails/commits?result\_type=activerecord/mysql2\_scope\_all\_with\_default\_scope&display\_count=200](https://rubybench.org/rails/rails/commits?result_type=activerecord/mysql2_scope_all_with_default_scope&display_count=200) Can you elaborate? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 10, 2016, 1:28am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/42 "2016-06-10T01:28:29Z")
> [@shahsaurabh0605](#): > > I am currently going through the current ActiveRecord framework. Do I need to completely transform this into Sequel model? Yes you’ll need to. Look at what each ActiveRecord benchmark is benchmarking for and write the same benchmark using Sequel. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 10, 2016, 1:30am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/43 "2016-06-10T01:30:24Z")
> [@shahsaurabh0605](#): > > Also, I did not understand what sam meant by this: "I feel there is a huge gap around our Rails benchmarks, in particular have a look at say: [RubyBench - Long Running Ruby Benchmark](https://rubybench.org/rails/rails/commits?result_type=activerecord/mysql2_scope_all_with_default_scope&display_count=200) > > Can you elaborate? Further into the post, the elaboration can be found 🙂 > What if this was implemented raw against the mysql gem, how fast would it be? How many objects would be allocated? > What if this was implemented in Sequel? How fast would it be? How many objects would be allocated? ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [June 11, 2016, 6:34pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/44 "2016-06-11T18:34:23Z")
I have just created the sequel equivalent of the first activerecord benchmark [bm\_activerecord\_create\_string\_columns.rb](https://github.com/ruby-bench/ruby-bench-suite/blob/master/rails/benchmarks/bm_activerecord_create_string_columns.rb). Have a look at [this](https://github.com/shahsaurabh0605/Sequel-benchmarks/blob/master/bm_sequel_create_string_columns.rb). Are there any changes to be made? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 12, 2016, 12:52am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/45 "2016-06-12T00:52:58Z")
Please open a PR on ruby-bench-suite so that I can comment directly 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 12, 2016, 1:04am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/46 "2016-06-12T01:04:39Z")
Aside from some minor comments I have, you can start working on the other sequel benchmarks 🙂 ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [June 28, 2016, 10:02am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/47 "2016-06-28T10:02:30Z")
Here are some of the things we can do before moving on to the next step : 1. In our discussions while adding the sequel benchmarks in `ruby-bench-suite` we removed sqlite3 gem from the gemfile. Similarly for the rails benchmarks I think sqlite3 gem is for local testing only and not used anywhere. 2. In the guides for running rails benchmarks instructions are corresponding to sqlite3 gem. As you pointed out, we expect the users to have mysql2 or postgres setup installed. So we may as well change that. 3. If we decide to remove sqlite3 then we can also remove sqlite3 dependencies from the dockerfile in `ruby-bench-docker`. I did not include those dependencies in sequel dockerfile. 4. Should I update [readme.md](http://readme.md) in `ruby-bench-docker` to run sequel benchmarks? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 28, 2016, 10:19am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/48 "2016-06-28T10:19:26Z")
Yea just nuke sqlite3. > [@shahsaurabh0605](#): > > Should I update readme.md in ruby-bench-docker to run sequel benchmarks? Yup please do 🙂 Actually can you write a bash script to run all the commands? ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [June 28, 2016, 10:25am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/49 "2016-06-28T10:25:45Z")
> [@system](#): > > Actually can you write a bash script to run all the commands? You mean a bash script to run all the sequel benchmarks? Another one for rails benchmarks? Similarly all those for different benchmarks? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 29, 2016, 1:06am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/50 "2016-06-29T01:06:37Z")
A bash script that basically runs the docker commands for each benchmark (etc `rails_releases`, `rails_trunk`) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [June 29, 2016, 4:08am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/51 "2016-06-29T04:08:27Z")
@shahsaurabh0605 The next important thing I would like you to work on is getting a simply benchmark regression notifier built into RubyBench. Have it track the benchmark results of the last X number of commits and report a regression if a new result exceeds the previous baseline. Couple of things to think about while building this feature: 1. False positives: I’ll probably handle false positives by running the benchmarks again and confirming the regression before reporting it. 2. Where to report? I think getting an issue opened on `ruby-bench-suite` will be a good start Let me know if you need more help! ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [June 30, 2016, 7:12am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/52 "2016-06-30T07:12:57Z")
For benchmark regressions, I think we can add a method in `benchmark_runs` controller. So, as soon as the new commit is pushed, new benchmark is created and after that we fetch benchmark results from the last X commits from database and compare. Can this be the right way? Also, what should be the previous baseline? For the last X commits there can be results which are quite larger or quite smaller. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [July 4, 2016, 1:50pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/53 "2016-07-04T13:50:29Z")
> [@shahsaurabh0605](#): > > For benchmark regressions, I think we can add a method in benchmark\_runs controller. So, as soon as the new commit is pushed, new benchmark is created and after that we fetch benchmark results from the last X commits from database and compare. Can this be the right way? Yup pretty much. > [@shahsaurabh0605](#): > > Also, what should be the previous baseline? For the last X commits there can be results which are quite larger or quite smaller. I’ve never implemented something like this but what I will try out first is some sort of standard deviation and then set a certain percentage as the threshold. The percentage can be anything reasonable as that can be tuned along the way. ---
**Author:** ![prathamesh](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/prathamesh/32/68_2.png) [@prathamesh](https://community.rubybench.org/u/prathamesh)\ **Post date:** [July 5, 2016, 2:51pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/54 "2016-07-05T14:51:11Z")
We can also build the functionality for handling false positives into the system. We can add UI where the admin can re-run the benchmarks for false positives and only after that issue will be created on the `ruby-bench-suite` repo. Thoughts? ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [July 6, 2016, 1:53pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/55 "2016-07-06T13:53:10Z")
I did this activity to understand the suitable standard deviation and regression parameter: For a particular benchmark in rails, I picked up 75 benchmark results as below: 950,945,932,932,939,932,943,936,939,938,924,948,942,948,943,946,932,942,945,945,951,937,939,937,951,929,939,953,938,941,935,959,950,928,956,942,974,936,933,930,955,953,940,939,938,952,936,940,939,950,933,944,927,933,937,933,949,938,945,943,936,928,955,941,937,942,939,929,932,966,935 Here in our case the benchmark results don’t vary a lot. Average of the above result turns out to be 941.16 and standard deviation 9.23. I would want to know when can we say that a particular benchmark is undergoing a regression? Can we set something like an average + 2 standard deviations undergoes a regression? @prathamesh ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [July 7, 2016, 8:59am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/56 "2016-07-07T08:59:21Z")
> [@shahsaurabh0605](#): > > Can we set something like an average + 2 standard deviations undergoes a regression? We can try that out first. We’ll tune it as we try it out ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [July 10, 2016, 6:35pm UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/57 "2016-07-10T18:35:05Z")
This is to inform you that I am currently admitted in the hospital because of dehydration. I am on glucose right now, that’s why I won’t be able to work for next 3 - 4 days as my doctor says. I will try to get back on track as soon as possible. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [July 11, 2016, 12:47am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/58 "2016-07-11T00:47:17Z")
No worries. Your health is more important, no rush in getting back to working on this project. Take care! ---
**Author:** ![shahsaurabh0605](https://sea2.discourse-cdn.com/flex016/user_avatar/community.rubybench.org/shahsaurabh0605/32/77_2.png) [@shahsaurabh0605](https://community.rubybench.org/u/shahsaurabh0605)\ **Post date:** [July 18, 2016, 10:25am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/59 "2016-07-18T10:25:58Z")
> [@system](#): > > 1. Where to report? I think getting an issue opened on ruby-bench-suite will be a good start To open an issue on ruby-bench-suite I need a user which can open the issue making a curl post request. Do we open an issue using a bot or is there an alternative? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/rubybench/original/1X/4ae23b7eff9a1b039e126f05dd734ab09c8ab4b5.png) [@system](https://community.rubybench.org/u/system)\ **Post date:** [July 18, 2016, 11:07am UTC](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99/60 "2016-07-18T11:07:29Z")
Yea just use a bot to open an issue [Previous page](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99.md?page=2) [Next page](https://community.rubybench.org/t/gsoc-project-improving-rubybench/99.md?page=4)