My name is Osama and I’m currently working on rubybench.org to bring it up-to-date and make some improvements.
One feature I’d like to implement is to make it easy to download and run benchmarks locally. This would come in handy for example when someone wants to debug/fix a performance regression, they can quickly grab the script and use it.
I can see 2 ways to make this work:
First way is to create a gem which can be invoked via command line to install the required dependencies to run the benchmarks. Once the dependencies are installed, you can provide the gem with a URL (which you could copy from rubybench.org) or path to a local copy of the script you want to benchmark and it’ll benchmark the script for you. The script will be benchmarked against a local clone of the repository you’re fixing (e.g. rails, ruby etc.). The command would look something like this:
rubybench-runner rails $SCRIPT_URL --path ~/rails
(suggestions for a better name are welcome)
Once the benchmarks are complete the gem would print the results to the terminal in a nice format and maybe it’ll have an option to save the results to a file.
The other way is to create a runner script in GitHub - ruby-bench/ruby-bench-suite: Benchmark suite for RubyBench.org that can take care of installing dependencies and benchmarking scripts similar to how the gem would work. Then the user would need to clone the repo (which contains all the scripts we have for rubybench.org) and launch the runner script inside the repo and pass to the launcher the name of the benchmark script they wish to run.
The downside is that the
ruby-bench/ruby-bench-suite repo contains many files for ruby, rails, PG and bundler benchmarks. If you’re working on rails you probably don’t care about the ruby files and vice versa. Also it contains files that are only used by rubybench.org server to publish results on the website.
I’m in favor of the gem approach because I think it’s cleaner, easier to use and more user-friendly than calling a script inside a cloned repo.
Some benchmarks depend on having a postgres (or mysql etc.) server running, the problem here is that installing dependencies such as postgres differs between between operating systems and it even differs between Linux distros. Is it safe to assume the user already has such fundamental dependencies installed?
I think it makes sense to have a page on rubybench.org that explains how to run the benchmarks locally and lists what dependencies are required.
Does this sound good to you? Any concerns?