Jump to navigation

Poupou's Corner of the Web

Looking for perfect security? Try a wireless brick.
Otherwise you may find some unperfect stuff here...

Weblog

Gendarme 2.2 Performance

Like the previous 2.0 release I wanted to invest some time to track where time and memory goes - and make sure Gendarme stays fast. My (personal) performance goal for Gendarme 2.2 was that time/memory increases (percentages) should be lower than the added new features (i.e. rules).

Now be warned that this benchmark (like most of them ;-) is not fair. Why ? because I want to compare performance between Gendarme 2.0 and 2.2. This is not the same as comparing Gendarme 2.0 on Mono 2.0 versus Gendarme 2.2 on Mono 2.2 - even if this is what most people will end up doing. E.g. The newer JIT in Mono 2.2 is a lot better, thanks to some pretty cool hackers - but I'll let them (or others) show you how much better it is ;-). So in the following tests both versions of Gendarme are being executed on the Mono 2.2 JIT. Also the older 2.0 was modified to report the initialization time (between a tenth and half of a second) to be similar to what 2.2 does. This is important because the new engines, present in 2.2, spend their time in initialization (from a few to several seconds). So timing this step became critical in 2.2. In other words you should see even better performance than the current numbers shows.

Gendarmeversion2.02.2 preview 1change (%)
Rules#151183121.19%[1]
Applications# assembliestime (sec)time (sec)change (%)
Banshee 1.2 14 6.55 7.58 115.70%[2]
Beagle 20 10.52 11.32 107.59%
F-Spot 22 676.01 586.33 86.73%[3]
Giver 1 1.44 1.41 98.15%
Gnome-Do 0.6.1 3 1.82 1.87 102.45%
Monodoc 12 2.15 2.44 113.73%
Monsoon 4 5.93 6.75 113.83%
Tasque 2 2.12 2.33 109.54%
Tomboy 4 3.65 4.05 110.84%
Mono 2.2 (2.0 profile)
default [4] 85 70.74 154.15 217.92%[5]
all rules 85 697.14 819.39 117.54%[6]
Total1,478.081,597.62108.09%

Notes:

  1. The "real" percentage should be a bit higher since we merged two rules and other got more features/checks.
  2. Don't worry I updated to Banshee 1.4, but that was after I got those numbers :-)
  3. Most of the time still comes from Tao.OpenGl.dll, the net gain too. Nestor is working on the main culprit so it will be even faster in 2.4
  4. The "default" rule set includes every rules except the Smells
  5. That was quite unexpected. Even more since 2.1 was more performent than 2.0 up to 175 rules.
  6. Fixing #5 will help for the final 2.2 but even more important will be the fix for #3 (but that 2.4 stuff ;-)

The final index (108%) is under the rules increase (121%) but the time required to fully analyze Mono 2.2 (2.0 profile) assemblies shows a (or some) bottleneck(s) that needs further work. Analysis (and fixes) have already started this weekend. Expect an updated table for the next preview (or RC).


11/16/2008 21:10:51 | Comments | Permalink

Gendarme 2.2

Mono branched for 2.2 earlier this week and Gendarme follow its wave. A first preview, binaries and win32 installer, are now available on Ohloh. Linux packages (inside mono-tools) for the first preview should be available soon.

This Gendarme release includes the development results of the last four months, including some major events like the Novell Hack Week 3 and the second half of GSoC 2008 work done by Nestor Salceda. Major highlights includes:

  • Improvement to the framework by the addition of engines that can build extra information to extend the model Cecil gives us to work with.
  • New [FxCopCompatibility] attribute to match between Gendarme and FxCop rules. This will be used to support [SuppressMessage] inside assemblies in a future release.
  • New TearDown capability on rules and runners allow rules to defer the reporting of defects until the analysis is completed.
  • New "Audit" severity. This is used to denotate defects that, too often, can't be fixed (i.e. will always be reported) but needs to be audited (e.g. security items).
  • By default the console runner now reports only when both Severity > Low, i.e. skipping Low and Audit severity defects, and Confidence > Low, i.e. skipping Low confidence defects.
    This is to help reduce false positives and get smaller reports with the most important defects. You can override both option, with lower or higher values, with new command switches.
  • The wizard runner is now, by default, limited to 1000 defects on the visible API and will, like the console runner, report only defects when both Severity > Low and Confidence > Low. A new step in the wizard allow users to change those settings before the analysis (defaults can be modified and saved).
  • You can now save your rule selection as the new default in the wizard. The same set will be enable next time you restart the wizard.

32 new rules including:


Gendarme.Rules.BadPractice
Gendarme.Rules.Concurrency
Gendarme.Rules.Correctness
Gendarme.Rules.Design
Gendarme.Rules.Design.Generic
Gendarme.Rules.Design.Linq
Gendarme.Rules.Exceptions
Gendarme.Rules.Interoperability
Gendarme.Rules.Naming
Gendarme.Rules.Performance
Gendarme.Rules.Security.Cas
Gendarme.Rules.Serialization
Gendarme.Rules.Smells

Lots of rules have been updated and, in a few cases, renamed or moved, to provide extended functionalities. Biggest move/merge changes are:

  • CAS-related Security rules were moved into Security.Cas
  • DisposableFieldsShouldBeDisposedRule was moved from Design into Correctness
  • EnumNotEndsWithEnumOrFlagsSuffixRule was merged with UseCorrectSuffixRule (Naming)
  • FinalizersShouldCallBaseClassFinalizerRule was moved from Design into Correctness
  • ImplementGenericCollectionInterfacesRule was moved from Design into Design.Generic

Contributors for this release are: Peter Johanson, Nestor Salceda, Cedric Vivier, Jesse Jones, Alan McGovern and me :-)


11/13/2008 20:18:15 | Comments | Permalink

The views expressed on this website/weblog are mine alone and do not necessarily reflect the views of my employer.