Jump to navigation

Poupou's Corner of the Web

Looking for perfect security? Try a wireless brick.
Otherwise you may find some unperfect stuff here...

Weblog

Gendarme Speed Up

In the last six months a lot of attention was given to Gendarme's performance. First we updated it's framework to make it easier to skip rules when they are not applicable, e.g. checking for 2.0 features inside a 1.x assembly. Then I spent some time reducing memory allocations in Cecil. Since Gendarme is a big consumer (it reads everything) even small gains show up quickly.

This was an important step to move forward because the some of the new features, like debugging symbols (mdb and pdb on windows) and automatic assembly resolving were sure to require a lot more time (and memory) to process rules (not to mention that the number of rules is itself growing fast). Also to keep its value Gendarme, or at least its default rule set, needs to execute quickly enough that there's no good reason not to execute it at each build. And finally it's also because we're not much more patient than our users - we want results now :-)

So last night I took the time to measure how the new Gendarme 0.2, to be released with Mono 2.0, compares the the old Gendarme 0.0.5.1 released with Mono 1.9. The next table shows the time (in seconds) required to analyze the 72 assemblies that Mono ships for it's 2.0 profile.

Gendarme 0.0.5.1Gendarme 0.2.0.0ChangeMean time per
rule delta
# rulestime# rulestimerules factortime factor
BadPractice 7 9.39688 13 21.933973 186% 233.42% 0.34
Concurrency 3 9.703159 6 14.05999 200% 144.90% -0.89
Correctness 7 11.732152 13 15.228963 186% 129.81% -0.50
Design 25 4.235242 33 4.29234 132% 101.35% -0.04
Exceptions 2 9.467328 7 16.499359 350% 174.28% -2.38
Interoperability4 8.841298 5 14.582129 125% 164.93% 0.71
Maintainability 0 0 6 18.499207 N/A N/A N/A
Naming 11 8.352653 12 2.133514 109% 25.54% -0.58
Performance 13 157.135513 25 39.716426 192% 25.28% -10.50
Portability 4 11.556642 5 42.844831 125% 370.74% 5.68
Security 10 6.434193 10 8.709008 100% 135.36% 0.23
Serialization 0 0 7 1.539076 N/A N/A N/A
Smells 6 652.999564 6 685.303299 100% 104.95% 5.38
UI 3 3.163413 3 0.106041 100% 3.35% -1.02
TOTAL 95 893.018037 151 885.448156 159% 99.15% -3.54
default 89 172.60422 145 99.322118 163% 57.54% -1.25

The number of rules and execution times are given for each rule assembly/category. We can see the impact of the new features when looking at Security, it has the same number of rules but it now requires 35% more time to execute them. We can also see how skipping rules can help in UI where it nows takes only 3% of the previous time to execute the rules (since UI rules don't applies to the class librairies).

The good news is that, globally, executing every rules takes just a bit less time than before. Considering that we added 56 rules in this release this was quite a nice surprise (even if a bit planned I did not anticipate a status quo). But the results are even better if we consider Gendarme's default rule set (i.e. everything but the Smell rules) as it requires less than 60% of the time than the previous version did, even with the extra 56 rules.

Results are still preliminary since Mono 2.0 won't ship until September. Some existing rules don't yet take advantages of the new API and I intend to update them (or at least a few of them) before then and provide an updated table. And who knows maybe I'll even dive into Cecil a littl'bit more ;-)

There is no end in performance (so expect more blog entries on the subject ;-). We need to stay vigilant because even small things, like memory allocations, can make big differences in rules executed thousands of times. We already have ideas to speed up a lot of rules using a new feature. This should happen in the next Novell Hack Week and the results will be visible in Mono 2.1. With a bit of luck we'll have more rules and results in even less time (or at least worked upon that goal).


7/17/2008 18:55:11 | Comments

The views expressed on this website/weblog are mine alone and do not necessarily reflect the views of my employer.