I did this activity to understand the suitable standard deviation and regression parameter:
For a particular benchmark in rails, I picked up 75 benchmark results as below:
Here in our case the benchmark results don't vary a lot. Average of the above result turns out to be 941.16 and standard deviation 9.23. I would want to know when can we say that a particular benchmark is undergoing a regression? Can we set something like an average + 2 standard deviations undergoes a regression?