Memory usage on aobench looks a bit odd

Looking at: http://rubybench.org/tgxworld/ruby/releases?result_type=app_aobench

memory usage is at say 96kb on 2.2.0

When running locally I notice

sam@ubuntu ruby-bench % ps aux | grep aob
sam       1301 79.3  0.1  34448  7388 pts/18   Sl+  10:46   0:41 ruby bm_app_aobench.rb

so usage is at 34kb

Something is fishy here, why is there such a discrepancy?

Are memory benchmarks supposed to be stable? I tried running multiple times and it seems like the memory usage varies quite a bit. For releases, we’re running it 5 times and taking the min.

Can you reproduce:

Locally ?

Memory should be pretty stable (unless GC triggered a heap growth that can happen), try inside docker, then outside docker (just try reproing numbers for 2.2.0)

need exact script you are running and can you confirm the meaning of the numbers.

@sam Sorry for taking so long to get to this.

Gist script.

ps -o rss= -p #{pid}. According to man ps:
rss RSS resident set size, the non-swapped physical memory that a task has used (inkiloBytes). (alias rssize, rsz).

Let me know if anything is wrong. Thanks :smile:

@sam I think I see what you mean.

guoxiang@guoxiang-GS60-2PC-Ghost ruby (trunk) $ ruby benchmark/bm_app_aobench.rb 
260

guoxiang@guoxiang-GS60-2PC-Ghost ruby (trunk) $ ps aux | grep aob
guoxiang  4969  100  0.0  35428  7400 pts/5    Rl+  15:46   0:38 ruby benchmark/bm_app_aobench.rb
guoxiang  5137  0.0  0.0  15944   932 pts/1    S+   15:46   0:00 grep --color=auto aob

I have a feeling this script is inaccurate.

ahhh yeah, the mem_end - mem_start thing makes little sense, ruby starts out with a bunch of allocated empty space.

instead just looking at RSS at the end should be fine.

@sam Ah ic! I’ve updated the benchmarks to just take the memory at the end of the script