Technical demo files (Latest: hash-tree benchmark, 2003-09-16)

These demos are based on my actual implementation and should be representative of the performance that could be expected from it. The hashes generated should also be representative of the final implementation with any of the demo hashes if I do not opt to collapse singleton nodes.

Each demo may be updated every now and then depending on what I have been testing. When more advanced demos and proof of concept apps become available, this page should become more interesting.

Demo Index

• First test/debug/demo application
  Current version : 0.0695 (2003-09-16)
• Sparse File Converter
  Current Version : 0.01 (2003-09-16)

First test/debug/demo application

This is the program I put together to test the classes I am coding for MAFS. The classes under test are : CHashContainer, CHashTreeNode and CHashTreeCtrl. CHashContainer can be considered completed and nearly final, CHashTreeNode is is roughly half-done and CTreeCtrl is not much more than an incomplete skeleton/draft implementation - it was used only because it implements the simple top node level computation.
nodeLevel = ceil(log(cFile.GetLength())/log(2)); // Damn, just disclosed one line of core code!

• The primary functionality testing debug program... these will be updated as new functionality is being tested.
  • New Static-Linked version : http://MoonlightHacks.com/eMule/Releases/HashTreeTest-Static.zip (~150KB)
    • Updated 2003-09-16: (v0.0695)
      • Rebuilt to make sure it uses the same hash tree as Sparse File Converter so it can be used to independently validate the results.
    • Updated 2003-09-06: (v0.06, 0.065)
      • Oops: was doing 7X hashing while I thought it was doing 3X... made it 3X for real this time.
      • Added a dummy hash for preloading.
      • Moved to 128KB block IOs but this did not seem to improve performance much.
      • 14h50 : Oops, just noticed that I did not update the ZIP file with the correct EXE.
        • The new bew version us 0.068 and has the following changes:
          • Base level increased from 10 (1KB) to 14 (16KB) to reduce hash function startup overhead and tree depth, leading to an approximately 20% performance improvement.
          • Reduced hash depth from 3 to 2, another 5% speed gain.
        • Even newer version (v0.0681)
          • REALLY really going from 7X hashing to 3X hashing roughly doubled hash speed.
          • Add critical parameters to the benchmark's output.
        • Of course, with new tree parameters, the hashes are completely different, that's why this must be settled before I can finish this - changing hash once this goes to 'production' status. (eMule integration as either official or as an alpha, beta or final mod of any remotely substantial deployment.)
        • Hash tree generation benchmarks
    • Updated 2003-09-05: (v0.05)
      • Changed the output format.
      • Moved the benchmark to its own (templated) function for convenience.
        • Hash benchmarks now delete their trees immediately after completing instead of piling up RAM allocation and deleting trees after all hashes benchmarks have been completed. Not an issue for 1MB files but could be one if someone tried a 1GB one.
      • Changed preloading hash from MD5 to MD4, display this hash's results to show actual MD4 performance with initial HDD IO.
    • No more extra DLLs required.
    • Includes a skeleton implementation of CHashTreeController.
      • Now supports arbitrary file sizes. (the old implementation had a hard-coded tree top level of 21, effectively limiting the hash tree's scope to the first 2MB of any file. CHashTreeCtrl computes the required top level from the file's size.)
    • Dropped MD2 because it is so awfully slow.
    • Some extra code cleanups.
    • Yay, the hashes are still the same so I did not mess up the inner-workings!
    • Yay, I get the same hashes on various PCs and internal architectural variations, meaning it works and it is not luck!
    • Updated 2003-09-09: (v0.0689) Malfunctionning HDD controller corrupted the file system on my main projects drive. (Promise Ultra100TX2)
      • Had to re-code parts of HashContainer.cpp due to corruption, the only code file damaged.
      • Performance is up 30-50%, mainly due to fixing an implementation mistake in CHashContainer.Hash : the function was "Salting" using pData/nData instead of pSalt/nSalt as I had intended - this was confirmed by trying out the suspected improper previous implementation. (pSalt/nSalt is much smaller than pData/nData.)
  • Old Dynamic-Linked version : http://MoonlightHacks.com/eMule/Releases/HashTreeTest.zip (~30KB)
    • This file will be pulled in a few days (I have only 5MB web space, maybe I will be 30KB short soon...) and I will pretend it never existed. I am leaving it here in the meantime only for the curious.
  • At the moment, it demonstrates the following:
    • CHashContainer
      • Most of the basic functionality.
      • The template's ability to handling various hashes.
    • CHashTreeNode
      • The hashing capabilities.
      • The expectable hashing performance.
      • The template's capability of handling various hashes.
    • CHashTreeCtrl
      • Only skeleton functionality. (new version)

Sparse File Converter

This is a simple and presumably efficient test program for my flat to sparse file conversion routine.

What do sparse file do that normal files don't do?

The sparse files are used for eMule's temporary files. Since these mostly contain large NULL blocks, some file systems like NTFS are able to reclaim those unused parts of the file and make the space available for other uses while it is not required. This means that a 10GB partition could contain 100GB worth of <10% average completed files. If completed files are moved somewhere else upon completion, a relatively small partition could provide ample temporary space for huge downloads.

What it does

You can check it out by clicking the link to the right... http://MoonlightHacks.com/eMule/Releases/SparseTest.zip (Now included in Moonlight's Mod (Upcoming build: 0.42b-0.22g, 2004-03-03).) Since this was used to validate my sparse conversion, it does the following:

1. Hash the file to determine its pre-sparse signature.
2. Convert the file to sparse with forced rescan.
3. Re-hash the file to verify that the hash is still the same to make sure it really worked.

That is all the demo stuff I have currently available. The next major demo app might be ready in about a month from now, somewhere around early October. In the meantime, you can expect a fair amount of documentation to be added as the project's core moves on.

Hits since December 5, 2003:

---