I did this activity to understand the suitable standard deviation and regression parameter:
For a particular benchmark in rails, I picked up 75 benchmark results as below:
950,945,932,932,939,932,943,936,939,938,924,948,942,948,943,946,932,942,945,945,951,937,939,937,951,929,939,953,938,941,935,959,950,928,956,942,974,936,933,930,955,953,940,939,938,952,936,940,939,950,933,944,927,933,937,933,949,938,945,943,936,928,955,941,937,942,939,929,932,966,935
Here in our case the benchmark results don’t vary a lot. Average of the above result turns out to be 941.16 and standard deviation 9.23. I would want to know when can we say that a particular benchmark is undergoing a regression? Can we set something like an average + 2 standard deviations undergoes a regression?