GSOC Project : Improving RubyBench

41

I am currently going through the current ActiveRecord framework. Do I need to completely transform this into Sequel model?

Also, I did not understand what sam meant by this:
"I feel there is a huge gap around our Rails benchmarks, in particular have a look at say:
https://rubybench.org/rails/rails/commits?result_type=activerecord/mysql2_scope_all_with_default_scope&display_count=200

Can you elaborate?

42

Yes you’ll need to. Look at what each ActiveRecord benchmark is benchmarking for and write the same benchmark using Sequel.

43

Further into the post, the elaboration can be found :slight_smile:

What if this was implemented raw against the mysql gem, how fast would it be? How many objects would be allocated?

What if this was implemented in Sequel? How fast would it be? How many objects would be allocated?

44

I have just created the sequel equivalent of the first activerecord benchmark bm_activerecord_create_string_columns.rb. Have a look at this. Are there any changes to be made?

45

Please open a PR on ruby-bench-suite so that I can comment directly :slight_smile:

46

Aside from some minor comments I have, you can start working on the other sequel benchmarks :slight_smile:

47

Here are some of the things we can do before moving on to the next step :

  1. In our discussions while adding the sequel benchmarks in ruby-bench-suite we removed sqlite3 gem from the gemfile. Similarly for the rails benchmarks I think sqlite3 gem is for local testing only and not used anywhere.
  2. In the guides for running rails benchmarks instructions are corresponding to sqlite3 gem. As you pointed out, we expect the users to have mysql2 or postgres setup installed. So we may as well change that.
  3. If we decide to remove sqlite3 then we can also remove sqlite3 dependencies from the dockerfile in ruby-bench-docker. I did not include those dependencies in sequel dockerfile.
  4. Should I update readme.md in ruby-bench-docker to run sequel benchmarks?
1 Like
48

Yea just nuke sqlite3.

Yup please do :slight_smile:

Actually can you write a bash script to run all the commands?

49

You mean a bash script to run all the sequel benchmarks? Another one for rails benchmarks? Similarly all those for different benchmarks?

50

A bash script that basically runs the docker commands for each benchmark (etc rails_releases, rails_trunk)

51

@shahsaurabh0605

The next important thing I would like you to work on is getting a simply benchmark regression notifier built into RubyBench. Have it track the benchmark results of the last X number of commits and report a regression if a new result exceeds the previous baseline.

Couple of things to think about while building this feature:

  1. False positives: I’ll probably handle false positives by running the benchmarks again and confirming the regression before reporting it.
  2. Where to report? I think getting an issue opened on ruby-bench-suite will be a good start

Let me know if you need more help!

52

For benchmark regressions, I think we can add a method in benchmark_runs controller. So, as soon as the new commit is pushed, new benchmark is created and after that we fetch benchmark results from the last X commits from database and compare. Can this be the right way?

Also, what should be the previous baseline? For the last X commits there can be results which are quite larger or quite smaller.

53

Yup pretty much.

I’ve never implemented something like this but what I will try out first is some sort of standard deviation and then set a certain percentage as the threshold. The percentage can be anything reasonable as that can be tuned along the way.

54

We can also build the functionality for handling false positives into the system. We can add UI where the admin can re-run the benchmarks for false positives and only after that issue will be created on the ruby-bench-suite repo.

Thoughts?

55

I did this activity to understand the suitable standard deviation and regression parameter:
For a particular benchmark in rails, I picked up 75 benchmark results as below:

950,945,932,932,939,932,943,936,939,938,924,948,942,948,943,946,932,942,945,945,951,937,939,937,951,929,939,953,938,941,935,959,950,928,956,942,974,936,933,930,955,953,940,939,938,952,936,940,939,950,933,944,927,933,937,933,949,938,945,943,936,928,955,941,937,942,939,929,932,966,935

Here in our case the benchmark results don’t vary a lot. Average of the above result turns out to be 941.16 and standard deviation 9.23. I would want to know when can we say that a particular benchmark is undergoing a regression? Can we set something like an average + 2 standard deviations undergoes a regression?

@prathamesh

56

We can try that out first. We’ll tune it as we try it out

1 Like
57

This is to inform you that I am currently admitted in the hospital because of dehydration. I am on glucose right now, that’s why I won’t be able to work for next 3 - 4 days as my doctor says. I will try to get back on track as soon as possible.

2 Likes
58

No worries. Your health is more important, no rush in getting back to working on this project. Take care!

1 Like
59

To open an issue on ruby-bench-suite I need a user which can open the issue making a curl post request. Do we open an issue using a bot or is there an alternative?

60

Yea just use a bot to open an issue