GSOC 2017 - Long Running Ruby and Rails Benchmarks


#9

I changed readme a bit in rubybench guideline repo which was quite poor :worried:

Pls take a look Make README more descriptive by bmarkons · Pull Request #23 · ruby-bench/ruby-bench · GitHub

btw, I managed to run ruby benchmarks locally in docker container today :tada:


#10

What are prepared statements? I see results with prepared statements and without them.


#11

Prepared statements are SQL statements that have placeholders in them to inject data. For example, this is a prepared statement:

SELECT * FROM articles WHERE id = $1;

Here, $1 is a placeholder for a given value (it will be replaced with 10 for instance). You may also see the ? syntax for placeholders. Prepared statements are mainly used in Ruby on Rails to cache SQL queries as you often run the same queries but only with different values. This is also a good practice to avoid SQL injections as the value will be properly escaped (if you are dealing with user inputs for example).

If you have a Rails application, looking at the server logs, you’ll see SQL prepared statements.

Hope it’s clear enough, feel free to tell me if I’m not crystal clear ! :smiley:


#12

Thanks @robin850 :slight_smile:

So in performance aspect, prepared statements are expected to run faster because of caching behind, right?


#13

Yes exactly, because Active Record won’t have to rebuild the query ; it just needs to inject the values. :slight_smile:


#14

I see we currently have benchmarks for rails, ruby, bundler, discourse and sequel in our suite. Is this suite supposed to support only certain gems or as many as possible?

In case we are trying to support as many as possible with benchmarks, I was thinking if duplicating benchmarks to our suite (since benchmarks for ruby and discourse already exist on official repos) is good approach?

The idea is that benchmarks stay on the official repos instead of copying it to ruby-bench-suite. Workflow example would be like, if you are gem developer and you want benchmarks to be executed you would just submit pull request with some configuration, and after ruby-bench approve it and merge it, benchmarks you wrote as a gem developer would be executed like any other.

Do you think it is doable? Pls tell me if I’m missing something. :grimacing:


#15

Maybe @sam and @noahgibbs can answer you better than I would on the subject but I guess the goal is to support only major projects. The real problem isn’t benchmarks but running them somewhere and if Ruby Bench tries to support more and more projects, there may be scaling issues because the resources are a bit limited.


#16

Sort of, in the real world ™ any large install of Postgres will use pgbouncer in transaction pooling mode. This means that prepared statements simply are not an option. PG starts performing really inconsistently with tons of connections and Rails loves creating connections.

Yes I would like to only support major projects here, and for the ORM tests Sequal, AR, Raw are a good enough trio. No need to add more for now.

@bmarkons I am curious, did you write a few of the benchmarks in raw PG and Sequel yet (even 1 is a good start), how does performance compare on local?

Let’s try to stay laser focused on getting a great answer to the question above.


#17

I’m new to Ruby Bench too. But the way most benchmarking works is that you don’t want as much as possible - you want to focus on the things you consider important. Having a huge amount of stuff that doesn’t matter tends to clutter up your understanding.

There’s a great talk by Matt Gaudet on how to benchmark Ruby 3 that may help: [EN] Ruby3x3: How are we going to measure 3x? \/ Matthew Gaudet

What he’s getting at there is that your benchmarks should be broad enough to measure all the stuff you care about, but specific enough not to be too confusing. If we measured hundreds of different gems but they were all a kind of mix of Matt’s eight ideas, we wouldn’t be gaining anything new.


#18

5 posts were split to a new topic: Rewriting the scope_all benchmark in Sequel and Raw


#19

Let’s try to stay laser focused on getting a great answer to the question above.

@sam Maybe I sounded wrong, though I tried to understand rubybench project direction rather than do it atm :slight_smile: Now I know that the plan is to support only major projects in near future :ok_hand:

there may be scaling issues because the resources are a bit limited

@robin850 yeah, it seems like scaling would be an issue :cold_sweat:

@noahgibbs thanks for this great talk. I guess Matt was talking about measuring certain number of different gems for the purpose of measuring ruby 3 performance. I had in mind rubybench as a platform where gem developers are getting feedback on gem performance they are developing.

Thank you guys for explanation :slight_smile:


#20

I don’t know of any current plans to use Ruby Bench for that. But we’d definitely like it as a resource for the Ruby and Rails core teams to see regressions quickly and track them down accurately. Presumably also one or two other major projects like Discourse :wink:


#24

Hi @bmarkons, sorry I’m late to the party. Let me know if you need me to run anything on the servers. Currently we have 2, 1 bare metal server sponsored by Ruby Together which runs the benchmarks and another DO droplet that hosts ruby-bench-web


#25

Hi @tgxworld :clap: sure, thanks! :slight_smile: So I could run benchmarks directly on server, or you would run it for me - because of the access?


#26

@tgxworld will sort out access for you, you can not be blocked on this.


#27

Maybe a good idea to also check against the pg or mysql2 gem raw query. That’s about as raw as you’re going to get in Ruby.


#28

Hey @tgxworld,

I guess you should provide me with access on rubybench production and hetzner servers where benchmarks are being run :smile:

I see sequel benchmarks have been added last year but I am not sure why can’t be seen on UI? I am starting to work toward this:

So firstly I will be working on displaying sequal results for postgres_scope_all bench on that graph, before I start backfilling pg benchmarks.


#29

Hi everyone,

I have submitted final report on work during this summer.

Big thanks to every one of you!

Cheers


#30

Congratulations Marko ! :tada: You’ve done a great work during this summer, thank you very much !


#31

Great work @bmarkons. I’m really thankful for all the things which you’ve done for this project :slight_smile: