Poupou's Corner of the Web

In the last six months a lot of attention was given to Gendarme's performance. First we updated it's framework to make it easier to skip rules when they are not applicable, e.g. checking for 2.0 features inside a 1.x assembly. Then I spent some time reducing memory allocations in Cecil. Since Gendarme is a big consumer (it reads everything) even small gains show up quickly.

This was an important step to move forward because the some of the new features, like debugging symbols (mdb and pdb on windows) and automatic assembly resolving were sure to require a lot more time (and memory) to process rules (not to mention that the number of rules is itself growing fast). Also to keep its value Gendarme, or at least its default rule set, needs to execute quickly enough that there's no good reason not to execute it at each build. And finally it's also because we're not much more patient than our users - we want results now :-)

So last night I took the time to measure how the new Gendarme 0.2, to be released with Mono 2.0, compares the the old Gendarme 0.0.5.1 released with Mono 1.9. The next table shows the time (in seconds) required to analyze the 72 assemblies that Mono ships for it's 2.0 profile.

	Gendarme 0.0.5.1		Gendarme 0.2.0.0		Change		Mean time per rule delta
	# rules	time	# rules	time	rules factor	time factor
BadPractice	7	9.39688	13	21.933973	186%	233.42%	0.34
Concurrency	3	9.703159	6	14.05999	200%	144.90%	-0.89
Correctness	7	11.732152	13	15.228963	186%	129.81%	-0.50
Design	25	4.235242	33	4.29234	132%	101.35%	-0.04
Exceptions	2	9.467328	7	16.499359	350%	174.28%	-2.38
Interoperability	4	8.841298	5	14.582129	125%	164.93%	0.71
Maintainability	0	0	6	18.499207	N/A	N/A	N/A
Naming	11	8.352653	12	2.133514	109%	25.54%	-0.58
Performance	13	157.135513	25	39.716426	192%	25.28%	-10.50
Portability	4	11.556642	5	42.844831	125%	370.74%	5.68
Security	10	6.434193	10	8.709008	100%	135.36%	0.23
Serialization	0	0	7	1.539076	N/A	N/A	N/A
Smells	6	652.999564	6	685.303299	100%	104.95%	5.38
UI	3	3.163413	3	0.106041	100%	3.35%	-1.02
TOTAL	95	893.018037	151	885.448156	159%	99.15%	-3.54
default	89	172.60422	145	99.322118	163%	57.54%	-1.25

The number of rules and execution times are given for each rule assembly/category. We can see the impact of the new features when looking at Security, it has the same number of rules but it now requires 35% more time to execute them. We can also see how skipping rules can help in UI where it nows takes only 3% of the previous time to execute the rules (since UI rules don't applies to the class librairies).

The good news is that, globally, executing every rules takes just a bit less time than before. Considering that we added 56 rules in this release this was quite a nice surprise (even if a bit planned I did not anticipate a status quo). But the results are even better if we consider Gendarme's default rule set (i.e. everything but the Smell rules) as it requires less than 60% of the time than the previous version did, even with the extra 56 rules.

Results are still preliminary since Mono 2.0 won't ship until September. Some existing rules don't yet take advantages of the new API and I intend to update them (or at least a few of them) before then and provide an updated table. And who knows maybe I'll even dive into Cecil a littl'bit more ;-)

There is no end in performance (so expect more blog entries on the subject ;-). We need to stay vigilant because even small things, like memory allocations, can make big differences in rules executed thousands of times. We already have ideas to speed up a lot of rules using a new feature. This should happen in the next Novell Hack Week and the results will be visible in Mono 2.1. With a bit of luck we'll have more rules and results in even less time (or at least worked upon that goal).

Mono has branched for 2.0! Quite a good news for everyone. It's also a great moment for all gendarmes because it means we branched too!

The Gendarme 0.2 release includes the results of the last six months of hacking, including some major events like the Novell Hack Week (Winter 2008 edition), which changed much of the Gendarme framework, the first "Gendarme Rule Day" and the first part of GSoC 2008 work by Néstor Salceda.

General highlights:

• Performance enhancements, memory usage reduction (mostly inside Cecil but the gains are very visible in Gendarme)
• MDB/PDB support: source files and line numbers are now provided with defects
• New Windows installer: You can now easily enjoy Gendarme, and it's wizard, from Windows computers
• .desktop file for the Wizard runner
• Console runner now uses color when displaying on the console
• Albeit a bit invisible to end users, we started using the new unit test helpers made during GHOP
• Bug fixes: works continue, like previous releases, to remove false positives from existing rules
• and a lot of new rules including...

Gendarme.Rules.BadPractice

• AvoidCallingProblematicMethodsRule
• AvoidVisibleConstantFieldRule
• DoNotForgetNotImplementedMethodsRule
• DisableDebuggingCodeRule
• ObsoleteMessagesShouldNotBeEmptyRule
• ReplaceIncompleteOddnessCheckRule

Gendarme.Rules.Concurrency

• DoNotLockOnThisOrTypesRule
• DoNotLockOnWeakIdentityObjectsRule
• DoNotUseLockedRegionOutsideMethodRule

Gendarme.Rules.Correctness

• DoNotRoundIntegersRule
• ReviewDoubleAssignmentRule
• ReviewSelfAssignmentRule
• ReviewUselessControlFlow
• ReviewUseOfInt64BitsToDoubleRule
• ReviewUseOfModuloOneOnIntegersRule

Gendarme.Rules.Design

• AvoidMultidimensionalIndexerRule
• AvoidRefAndOutParametersRule
• AvoidVisibleNestedTypesRule
• ConsiderConvertingFieldToNullableRule
• ConsiderUsingStaticTypeRule
• ImplementGenericCollectionInterfacesRule
• ImplementIComparableCorreclyRule
• PreferIntegerOrStringForIndexersRule

Gendarme.Rules.Exception

• AvoidArgumentExceptionDefaultConstructorRule
• AvoidThrowingBasicExceptionsRule
• DoNotThrowReservedExceptionRule
• ExceptionShouldBeVisibleRule
• MissingExceptionConstructorsRule

Gendarme.Rules.Interoperability

• DoNotCastIntPtrToInt32Rule

Gendarme.Rules.Maintainability (new)

• AvoidComplexMethodsRule
• AvoidDeepInheritanceTreeRule
• AvoidLackOfCohesionOfMethodRule
• ConsiderUsingStopwatchRule
• PreferStringIsNullOrEmptyRule

Gendarme.Rules.Naming

• DoNotPrefixEventsWithAfterOrBeforeRule

Gendarme.Rules.Performance

• AvoidLargeNumberOfLocalVariablesRule
• AvoidLargeStructureRule
• AvoidRepetitiveCastsRule
• AvoidTypeGetTypeWhenPossibleRule
• AvoidUnusedPrivateFieldsRule
• AvoidUnneededUnboxingRule
• AvoidUnsealedUninheritedInternalClassesRule
• ImplementEqualsTypeRule
• OverrideValueTypeDefaultsRule
• RemoveUnusedLocalVariablesRule
• UseIsOperatorRule
• UseTypeEmptyTypesRule

Gendarme.Rules.Portability

• DoNotHardcodePathsRule

Gendarme.Rules.Serialization (new)

• CallBaseMethodsOnISerializableTypesRule
• DeserializeOptionalFieldRule
• ImplementISerializableCorrectlyRule
• MarkAllNonSerializableFieldsRule
• MissingSerializableAttributeOnISerializableTypeRule
• MissingSerializationConstructorRule
• UseCorrectSignatureForSerializationMethodsRule

Contributors for this release are:
Andres G. Aragoneses, Cedric Vivier, Daniel Abramov, JB Evain, Nestor Salceda, Scott Peterson, Sebastien Pouliot, Seo Sanghyeon.

Availability: Preview packages should be available soon (Gendarme resides inside mono-tools). I'll update the Win32 installer asap (probably tomorrow night), check the Files section of the group to get it. Now is the time for testing and reporting :-) Thanks to everyone!