Jump to navigation

Poupou's Corner of the Web

Looking for perfect security? Try a wireless brick.
Otherwise you may find some unperfect stuff here...

Weblog

CAS - where do we go from here

Now that we know where we stand on Code Access Security (CAS) let's see where we are going...

On the short-term, i.e. starting right now and with results partially visible in the Mono 1.2 release, the main focus will be to add more permissions into the class libraries. That will allow better testing of the features already in the runtime without requiring large changes in the runtime (as it will get more difficult as we get nearer the release) and make the class library more useful to people who want to try/test Code Access Security.

By itself, adding permissions isn't really difficult. Adding them so they are meaningful (i.e. adding security without under or over protection) is harder and will (later) require a careful, tool assisted, audit of all class libraries. Another difficult aspect is that Mono's source code is still evolving (e.g. new features, bug fixes, optimizations...) so we must ensure that permissions added today (and/or reviewed later) will still be effective when a new version of Mono is released months or years later.

So the typical "code-review-ship" trinity won't cut it - at least not if we try to do it in sequence. Neither trying to review all incoming changes (e.g. mono-patches) will likely (alone) yield good results. We need to ensure that most security efforts we put in place will somehow stick there - not quite the holy grail but not a very common situation either.

If this sounds somewhat familiar, and it should, it's because security and quality are very similar attributes in software. So we can reuse most of the same techniques for, a part, of the sticky plan.

Unit tests

Mono use NUnit for it's unit testing. Currently we have more than 13k tests on the default profile (fx 1.1) of our class libraries - and this number exclude the CAS specific tests which are presently excluded from the default execution. The NUnit framework makes it easy to write CAS specific tests and to reuse existing unit tests into CAS tests (e.g. testing execution under no, or near minimal, permissions).

Pros

  • Perfect to catch regressions;
  • Automated (e.g. Makefile target and buildbot);

Cons

  • It only (well mostly) test existing (old) code, i.e. new code requires new tests;
  • It's difficult to get 100% coverage;
  • Current Mono coverage by unit tests vary a lot between assemblies / namespaces. Some parts are very high, some parts are not tested;

Rule tests

Some tools, like Microsoft's fxcop, use a different approach - it ensure that the code (source or binary depending on the tool) follows a set of rules. There are many categories of rules: interop, performance, localization... and, of course, there are security rules.

Pros

  • Existing rules catch problems in new, or modified, code;
  • Adding new rules may find multiple problems in existing code;

Cons

  • Rules aren't all PASS or FAIL - they requires some contextual interpretation;
  • Some rule violations can't be fixed and retain compatibility with the MS framework (i.e. that's also true for fxcop-versus-MS BCL, most rules were "invented" after the BCL code was written and, in many cases, compatibility is more important that satisfying a rule);

Some may recall I encouraged people to create an fxcop-like tool based on Cecil a while back. Sadly that didn't yield much results - except that Duncan found a very cool name for such a tool: Gendarme. I can't wait anymore so I'm transforming my old Cecil-based code into (mostly security) rules. Expect more details about this later...

Are other tests required ?

No (in theory). With perfect/complete unit tests (or rule tests) you'll not need anything more. However there's no easy way to know if the tests are complete (don't confuse that with 100% coverage) and even less ways to know if they are perfect. So, even if reality sucks, we must deal with this as best as possible...

Yes (in practice). Unit and rule tests aren't complete because sometimes you:

  • don't know where a problem may exists, making it impossible to write a unit test;
  • can't make a rule clear and simple enough to catch a problem, which also rules out tools like fxcop/Gendarme.

Such cases cannot be automated by unit tests and rules. Finding and understanding the ultimate/missing tests requires a human mind. This is where we generally stop talking about testx and start talking about reviewing. It's a manual process but it doesn't have to be totally manual - which makes it but funnier and faster. This is why I've been using Cecil to generates lists, e.g. hints where errors are more likely or more dangereous (p/invokes, internal calls, unsafe code...) and to visualize the code (my cecil / dot graphs). Visualization helps me, and hopefully others, to find patterns that may later be turned into rules and, for specific cases, into unit tests (to avoid future regression).

As for the long-term activities I'll likely blog about them after, or nearer to, the release of Mono 1.2. But don't hesitate to contact me if you have any interest in them!


10/11/2005 10:58:44 | Comments

The views expressed on this website/weblog are mine alone and do not necessarily reflect the views of my employer.