Ability to easily download and run benchmarks locally

7

Making it easier to run benchmarks sounds great :slight_smile:
Here are my thoughts. I’m unsure if a gem or the repository + some scripts/harness is better. The gem is probably a bit harder to tweak if needed.

The downside is that the ruby-bench/ruby-bench-suite repo contains many files for ruby, rails, PG and bundler benchmarks.

I think this doesn’t matter much, the repository is only 10MB.

What would matter for me and I think for other Ruby implementers and probably more people is:

  • Easy to run the benchmark: automated setup of dependencies or checks for them is nice.
  • Give me a full command line for running the benchmark (the setup can be done via some scripts, that’s fine), so that it can easily be tweaked (e.g., by passing extra options to the Ruby interpreter). Specifically, I don’t want to execute any other Ruby process to run it, so e.g., I can profile/debug/instrument/etc just the process that matters easily.
  • STDOUT/STDERR should be the defaults, or clearly documented where they are redirected. That’s where my debugging/profiling output ends up often.
  • Ability to run without Docker, it’s just so much more convenient if e.g., I recompile my Ruby interpreter, change files here and there, etc.
  • Ability to run a fixed workload (e.g., 10000 iterations of the benchmark), this makes it easier to compare profiles.

And some orthogonal concern:

  • Each benchmark run should be verified if it produces the correct result. If I do an incorrect optimization, it should tell me it broke the benchmark, not that it got faster.
1 Like