@system I would like to add a few micro benchmarks, which lead me to thinking about organisation of benchmarks.
I feel we need a single place for all our benchmarks that accepts PRs for improving on them, you can then add/backfill and improve benchmarks. If the results are keyed on SHA1 of the benchmark file it becomes trivial to invalidate old benchmarks.
How should we organise all this stuff? How is it organised now?
I’m currently modifying driver.rb to POST the results. The good thing about doing this is that as Ruby add more benchmarks, we’ll be running them for long running benchmarks without me having to know that new benchmarks have been added. The downside is that releases benchmarks will not be aware unless I rebuild the docker containers.
Are we looking at something central like GitHub - acangiano/ruby-benchmark-suite: A set of Ruby benchmarks for testing Ruby implementations.? Owning our benchmarks would also allow us to switch to Benchmark IPS. Perhaps we could work with Ruby core to move the benchmark suite into a separate repo?
I think it is critical of rubybench to own all the benchmarks, it allows us to control the evolution and update schedule. Not saying start from scratch, just have a repo we own that we selectively merge into as needed.
got it. I suppose I’ll make the switch to Benchmark/ips too?
if you prefer, IPS is a bit magic, my pref is to keep it super simple, how would you wire ips up?
I’ll include benchmark ips in all the scripts. My main motivation is for benchmark ips to solve spikes in our results. I’ll try to wipe up a quite PR for this so that it’ll be easier for discussion. 
@sam Looks like using ips would mean more work and rerunning of our benchmarks. I’m not too keen on the latter. Therefore, I’ve ported the benchmarks over to our own repo. GitHub - ruby-bench/ruby-bench-suite: Benchmark suite for RubyBench.org