Rewriting the scope_all benchmark in Sequel and Raw


#9

Can you update that post with all your results to date on this specific test?

Also, can you run it on Rails 2/3/4 for a quick and dirty comparison.

(I know it will be eventually backfilled, but I am curious) :wink:


#10

sure @sam, I’m on it :thumbsup:


#11
// Activerecord

{
  "label":"activerecord/postgres_scope_all",
  "version":"3.2.22.5",
  "iterations_per_second":76.49321589608218,
  "iterations_per_second_standard_deviation":2.614610951534639,
  "total_allocated_objects_per_iteration":43179
}

{
  "label":"activerecord/postgres_scope_all",
  "version":"4.2.8",
  "iterations_per_second":80.02798516365223,
  "iterations_per_second_standard_deviation":3.748688654181643,
  "total_allocated_objects_per_iteration":21153
}

{
  "label":"activerecord/postgres_scope_all",
  "version":"5.1.1",
  "iterations_per_second":77.37787266441069,
  "iterations_per_second_standard_deviation":3.8770773823300333,
  "total_allocated_objects_per_iteration":20131
}

// Sequel

{
  "label":"sequel/postgres_scope_all",
  "version":"4.46.0",
  "iterations_per_second":342.1849354356875,
  "iterations_per_second_standard_deviation":5.2603133966378355,
  "total_allocated_objects_per_iteration":6033
}

// Raw pg

{
  "label":"raw_pg/postgres_scope_all",
  "version":"0.18.1",
  "iterations_per_second":810.0042539142364,
  "iterations_per_second_standard_deviation":7.530824647552864,
  "total_allocated_objects_per_iteration":4004
}

^ Here it is @sam :slight_smile:

Btw, I didn’t managed to run benchmark for rails 2, I couldn’t require rails gem in benchmark script. Not sure where’s the problem :confused:


#12

Awesome, now that you have this data. I want the graph amended so it can display a comparison of all these 5 results! Start hacking away at the web UI and the backfill.

Only focus on this single test, this is the only test you care about till we can render the 5 lines here and get it backfilled :wink:

https://rubybench.org/rails/rails/commits?result_type=activerecord/postgres_scope_all&display_count=2000

If you have to drop the whole “prepared” vs “non prepared” stuff and always do “prepared” for now so be it.

cc @tgxworld

Don’t worry about Rails 2 for now.


#13

@sem Just to make myself sure. Since this is graph with commit data, you want me to include here commits for sequel and pg. Therefore it would have 3 lines (AR, sequel, pg) and graph would look something like this:

Maybe better place to display results for older Rails releases would be here?
https://rubybench.org/rails/rails/releases?result_type=activerecord/postgres_scope_all


GSOC 2017 - Long Running Ruby and Rails Benchmarks
#14

Yes, absolutely, this is what I want you to work towards!

Possibly, let’s not worry about that for now, just for now worry about getting the graph with the 3 lines going and backfilling all the info. Work with @tgxworld on gaining access to prd.


GSOC 2017 - Long Running Ruby and Rails Benchmarks
#15

How are you going with this? Are you blocked on anything from our side?


#16

@sam, I am not blocked on anything. Right now I am working on web to display graphs to compare. My first goal is to display sequel along with AR because sequel has been already added. Here’s wip PR on web.


#17

Fantastic, focus on getting stuff merged and on the site as soon as you can. I prefer to have a slightly janky UI than wait weeks till we have the UI in perfect state.


#18

@sam
Little peek. This is with data from production imported in local. UI is not perfect and has some issues but before I get into fixing it I’ll make sure sequel benchmark graphs are being displayed.


Though I am not sure why sequel is not being displayed since it looks like been added :confused:


@tgxworld pls take a look at this one :arrow_down: :pray:


#19

Cool!!! how far do you feel we are from having something on rubybench.org?


#20

Hey @sam,

PR is some time in reviewing process. According to me it’s ready to be merged. When it’s merged we could compare benchmarks on rubybench.

Jeremy configured sequel repo with webhook to send push events on rubybench, so next time when he pushes something we should have sequel repo with it’s benchmarks on rubybench :tada:


#21

@bmarkons Merged and deployed :slight_smile:


#22

Is there anything you need from me to get the Sequel benchmarks? We should probably start back-filling the benchmarks.


#23

Great! Thanks so much for help :slight_smile:

Is there anything you need from me to get the Sequel benchmarks?

We should have first sequel runs displayed on Jeremy’s next push.

We should probably start back-filling the benchmarks.

You mean running them?


#24

Yup you can look for the last 2000 commit ids and run the benchmarks for them :slight_smile:


#25

Status update

We are receiving push events from sequel repo properly. We have very first sequel benchmark results on RubyBench. Some errors prevented all benchmarks to be displayed but I am working on it.

Once we have this one merged I will run benchmarks for sequel last 2000 commits. After that we should be able to compare them with AR.

Edit

Everything works properly for sequel benchmarks, all are present on RubyBench :tada:
The only thing left is to run suite for last 2000 commits manually.

https://rubybench.org/jeremyevans/sequel/commits


#26

I’ve run sequel suite for last 2000 commits. Everything is fine for now, it will definitely need some time to finish :slight_smile:

https://rubybench.org/jeremyevans/sequel/commits?result_type=sequel/postgres_scope_all&display_count=2000


#27

@bmarkons can you have a look into

https://rubybench.org/jeremyevans/sequel/commits?result_type=sequel/postgres_finders_find_by_attributes&display_count=500

The commit hash do not seem to be displayed correctly.


#28

Sure, I’ll take a look :+1:

Edit

Fixed here
https://github.com/ruby-bench/ruby-bench-web/pull/214