Ability to easily download and run benchmarks locally

#1

Hi everyone :wave:

My name is Osama and I’m currently working on rubybench.org to bring it up-to-date and make some improvements.

One feature I’d like to implement is to make it easy to download and run benchmarks locally. This would come in handy for example when someone wants to debug/fix a performance regression, they can quickly grab the script and use it.

I can see 2 ways to make this work:

First way is to create a gem which can be invoked via command line to install the required dependencies to run the benchmarks. Once the dependencies are installed, you can provide the gem with a URL (which you could copy from rubybench.org) or path to a local copy of the script you want to benchmark and it’ll benchmark the script for you. The script will be benchmarked against a local clone of the repository you’re fixing (e.g. rails, ruby etc.). The command would look something like this:

rubybench-runner rails $SCRIPT_URL --path ~/rails

(suggestions for a better name are welcome)

Once the benchmarks are complete the gem would print the results to the terminal in a nice format and maybe it’ll have an option to save the results to a file.

The other way is to create a runner script in GitHub - ruby-bench/ruby-bench-suite: Benchmark suite for RubyBench.org that can take care of installing dependencies and benchmarking scripts similar to how the gem would work. Then the user would need to clone the repo (which contains all the scripts we have for rubybench.org) and launch the runner script inside the repo and pass to the launcher the name of the benchmark script they wish to run.

The downside is that the ruby-bench/ruby-bench-suite repo contains many files for ruby, rails, PG and bundler benchmarks. If you’re working on rails you probably don’t care about the ruby files and vice versa. Also it contains files that are only used by rubybench.org server to publish results on the website.

I’m in favor of the gem approach because I think it’s cleaner, easier to use and more user-friendly than calling a script inside a cloned repo.

Some benchmarks depend on having a postgres (or mysql etc.) server running, the problem here is that installing dependencies such as postgres differs between between operating systems and it even differs between Linux distros. Is it safe to assume the user already has such fundamental dependencies installed?

I think it makes sense to have a page on rubybench.org that explains how to run the benchmarks locally and lists what dependencies are required.

Does this sound good to you? Any concerns?

cc @sam / @tgxworld

1 Like
#2

The gem approach sounds amazing. As far as, say, postgres, would it make sense to do it with a Docker container, the same way Ruby-Bench runs on the web site? That would not only take care of installation and avoid creating databases on local Postgres, it would also guarantee the same version was used.

2 Likes
#3

I think it is reasonably safe to go with the gem approach and trust that dependencies are good, I guess on boot it can do a test of pre-reqs and error out if something is wildly off. That said I would not really lock us to a particular version of postgres and instead just say must be newer than say 9.6 or something like that. Then if there is a mistmatch simply explain that in the runner:

The bench on Ruby bench ran using PG version 10.5, you are running 9.6

Something like that.

1 Like
#4

Thanks Noah. While using docker would certainly solve this particular problem, there are other downsides that can be barrier to entry such as having to install docker itself and download/build docker images which can take a while. I think the vast majority of people who will use this feature will have the necessary dependencies (e.g. postgres) already installed on their systems.

#5

Fair enough. But then you’ll need to duplicate any configuration that’s currently done via Docker and keep it up to date. If you’re fine with that, then sounds good :slight_smile:

1 Like