Moonlight's Sparse File Support (MSFS)

What is a sparse file?

A sparse file is a file who's internally unused internal ranges are not allocated to any actual disk space. These ranges will read as NULL (0) when applications read them and will be allocated only when written to. Since eMule partfiles are mostly unused space initially, substantial amounts of free space can be temporarily reclaimed for other uses.

The sparse support is built into Moonlight's Mod (Upcoming build: 0.42g-0.22g, ????).

Bug:

I tried various things to make Sparse possible concurrently with buffer flushes and discovered that this was more annoying to do than I had expected so until I work this out, I am making Sparse available only on paused/stopped files.

Todo:

Complete documentation.

Warning:

Since using sparse and compressed files will trick eMule into downloading more data than it thinks it can, sparse and compressed files support should be used along with the "Pause when disk free space is below XXMB" feature. This should ideally be set to many MBs, something like 4MB + 256KB per active (non-paused/stopped) partfile or more. I personally recommend at least 16MB, preferably 64MB.

New sparse/compressed implementation ready for testing.

Implemented in this new version:

• Newer than new, single worker thread work queue (CWorkQueue and CWorkQueueItem)
  • Uses less RAM and other resources
  • Global shutdown event for cleaner thread termination on exit
  • Another simple generic building block that can be used for other things
• Extended options for:
  • Making new partfiles Sparse
  • Making new partfiles Compressed
  • Allow multiple-select Sparse/Compress operations
  • Set the amount of free disk space at which to pause downloads at
• Download list pop-up menu items: (available only on paused or stopped downloads)
  • Make file sparse
  • Re-Scan sparseness
  • Compress
  • Decompress
  • Cancel (pending and abortable operations)

Changed files

• eMule.c, eMule.cpp, eMuleDlg.cpp:
  • Add Global shutdown event
• CPreferences:
  • Add the data needed for the new options
  • Add the relevant Get functions
  • Add preferences load/save for these options
• CPartFile:
  • Add a CAdvFileAttr class member
  • Added locked range support in CPartFile::FlushBuffer
  • Added calls to MakeSparse/MakeCompressed after partfile creation
  • Added low disk space file pausing at the end of CPartFile::FlushBuffer
    • implemented by bluecow in eMule-0.30c.
  • GetRealFileSize() now returns the really real On-Disk file size after compression/sparse

    Warning:

    This may break some mods. (Then again, so can compression and sparse files.)

• CPPgTweaks:
  • Added MakeSparse on create checkbox
  • Added MakeCompressed on create checkbox
  • Added allow multiple file selection operations checkbox
  • Added low disk space control
• CDownloadListCtrl:
  • Added extra popup menu for file operations
• CFileDetailDialog
  • The file status also includes Sparse/Compressed status
• OpCodes.h
  • Added #defines for the new popup menu items' actions
• eMule.rc, resource.h
  • Strings, minor GUI alterations, IDs, etc.

New files:

• AdvFileAttr.h, AdvFileAttr.cpp : CAdvFileAtr (replaces MakeSparse.h)
• WorkQueue.h, WorkQueue.cpp : static CWorkQueue used by CWorkQueueItem used by CAdvFileAttr
• Moonlight.ico : Extended option icon

How to integrate the original micro-mod?

The old quick-hack implementation required only three changes in PartFile.cpp:

• adding #include MakeSparse.h
• adding a call to MakeSparse when the creation of a new partfile or re-opening of an existing one succeeds.

  
  CString partfull(RemoveFileExtension(m_fullname));
  if (!m_hpartfile.Open(partfull,CFile::modeCreate|CFile::modeReadWrite|CFile::shareDenyWrite|CFile::osSequentialScan)){
  	theApp.emuledlg->AddLogLine(false,GetResString(IDS_ERR_CREATEPARTFILE));
  	SetStatus(PS_ERROR);
  } else MakeSparse(m_hpartfile, GetFileName(), false); // Moonlight: Try to make new files sparse.
  

• adding a call to MakeSparse when opening an existing partfile succeeds in LoadPartFile(...)

  
  // open permanent handle
  CString searchpath(RemoveFileExtension(m_fullname));
  CFileException fexpPart;
  if (!m_hpartfile.Open(searchpath, CFile::modeReadWrite|CFile::shareDenyWrite|CFile::osSequentialScan, &fexpPart)){
  	CString strError;
  	strError.Format(GetResString(IDS_ERR_FILEOPEN), searchpath, GetFileName());
  	TCHAR szError[MAX_CFEXP_ERRORMSG];
  	if (fexpPart.GetErrorMessage(szError, ELEMENT_COUNT(szError))){
  		strError += _T(" - ");
  		strError += szError;
  	}
  	theApp.emuledlg->AddLogLine(false, strError);
  	return false;
  } else MakeSparse(m_hpartfile, GetFileName(), false); // Moonlight: Try to make old files sparse.
  

Hits since December 5, 2003:

---