Moonlight's Miscellaneous Code Snippets Collection (MCSC)

As I dig around eMule's source code to find inspiration for new coding concepts, snoop around to add support for personal tweaks and clean ways to add new features, I stumble upon various bits of "This could be [more] useful" code and this page is where I will paste these simple changes/additions.

Current snippets:

• Moonlight's Save Upload Queue Wait Time (MSUQWT, last updated: 2004-02-04)
• Global shutdown event
• File Range Locking
• Moonlight's Guaranteed Minimum Download Slot Speed (MGMDSS, Experimental)
• Multiple Selection A4AF
• CWorkQueue, CWorkQueueItem : A simple asynchroneous task queue system.

Warning:

All full files and code fragments are taken from Moonlight's Mod (Upcoming build: 0.42b-0.22g, 2004-03-03) and may contain changes unrelated to the ones enumerated here. For the newest complete source code files, download the zipped sources from Moonlight's Mod (Upcoming build: 0.42b-0.22g, 2004-03-03).

Global shutdown event

As eMule starts becomes increasingly more multithreaded, a more deterministic way of notifying sleeping threads (waking them up) is to use a global event and make the threads use a CMultiLock object, relying on the "Any Lock" wake-up of CMultiLock::Lock. Since ShutdownEvent is a 'manual' event, once signalled all threads waiting on it will wake up, detect the Shutdown lock and quit ASAP.

• emule.h

  // Moonlight: Global shutdown event to wake up threads.
  class CShutdownEvent : private CEvent {
  public:
      CShutdownEvent(void) : CEvent(FALSE, TRUE), m_Shutdown(false)   {}
      virtual ~CShutdownEvent     (void)  {}
      void Signal                 (void)  {CEvent::SetEvent(); m_Shutdown = true;}
      bool isShuttingDown         (void)  {return m_Shutdown;}
      CSyncObject *   operator&   (void)  {return this;}
  private:
      bool    m_Shutdown;
  };
  extern CShutdownEvent ShutdownEvent;    /// Declares a global shutdown object.
  

• emule.cpp

  CShutdownEvent ShutdownEvent;   // Moonlight: Global shutdown event to wake up threads.
  

• emuleDlg.cpp

  [...]
  void CemuleDlg::OnClose()
  {
  [...]
      theApp.m_app_state  = APP_STATE_SHUTINGDOWN;
      ShutdownEvent.Signal();     // Moonlight: Global shutdown event signalled on exit.
      Sleep(100);                 // Moonlight: Give the threads 100ms to get a clue.
  [...]
  }
  

File Range Locking

I do not know if this bit of code has anything to do with it or if it is 0.30c, but I appear to be getting fewer (as in none so far) corrupt chunks since I began implementing this lock in a more proper manner. Maybe I am just luckier with my sources or the sources that used trashy clients have upgraded/fixed.

Since:

2003-11-06: Added a 'force' parameter fo FlushBuffer - to force flush during destruction: any block that fails destruction flush should be put back into the gap list, avoiding corruption due to thinking blocks were downloaded when they actually never got to the disk.

• PartFile.h

      
  class CPartFile : public CKnownFile {
  [...]
  public:
      void    FlushBuffer(bool bForced = false);  // Moonlight: Sparse/Compressed - force dumping the data on file lock failure.
  [...]
  

• PartFile.cpp

  CPartFile::~CPartFile(){
      delete m_AdvAttr;   // --> Moonlight: Sparse&Compressed support
  
      // Barry - Ensure all buffered data is written
      try{
          FlushBuffer(true);  // Moonlight: Forced flush - If we cannot write data, set the gap list to avoid corruption false-positives.
      }
      catch(CFileException* e){
          e->Delete();
      }
  [...]
  }
  
  void CPartFile::PauseFile(){
      if (status==PS_COMPLETE || status==PS_COMPLETING)
          return;
      FlushBuffer(true);  // Moonlight: Forced flush - If we cannot write data, set the gap list to avoid corruption false-positives.
  [...]
  }
  
  void CPartFile::CreatePartFile(){
  [...]
      if (!m_hpartfile.Open(partfull,CFile::modeCreate|CFile::modeReadWrite|CFile::shareDenyNone|CFile::osSequentialScan)){   // --> Moonlight: Sparse&Compressed support - shareDenyNone
          AddLogLine(false,GetResString(IDS_ERR_CREATEPARTFILE));
          SetStatus(PS_ERROR);
      }
  [...]
  }
  
  uint8 CPartFile::LoadPartFile(LPCTSTR in_directory,LPCTSTR in_filename, bool getsizeonly)
  {
  [...]
      if (!m_hpartfile.Open(searchpath, CFile::modeReadWrite|CFile::shareDenyNone|CFile::osSequentialScan, &fexpPart)){   // --> Moonlight: Sparse&Compressed support - shareDenyNone
          CString strError;
          strError.Format(GetResString(IDS_ERR_FILEOPEN), searchpath, GetFileName());
          TCHAR szError[MAX_CFEXP_ERRORMSG];
          if (fexpPart.GetErrorMessage(szError, ELEMENT_COUNT(szError))){
              strError += _T(" - ");
              strError += szError;
          }
          AddLogLine(false, _T("%s"), strError);
          return false;
      }
  [...]
  }
  
  // Moonlight: Sparse/Compressed - bForce = true : discard the data when the lock fails, force data out of the buffer.
  void CPartFile::FlushBuffer(bool bForced)
  {
  [...]
  // Buffer queue commit loop - almost completely rewritten to handle locked ranges.
  // --> Moonlight: Sparse (and anything else that may temporarily lock ranges) support
          POSITION pos1, pos2; // pos1 = next to process, pos2 = currently being processed.
          for (pos1 = m_BufferedData_list.GetHeadPosition(); ( pos2 = pos1 ) != NULL;)
          {
              // Get the item and update the pos
              item = m_BufferedData_list.GetNext(pos1);
  
              // These are needed a few times.;
              uint32 lenData = item->end - item->start + 1;
  
              // Try locking the range, leave the data in the queue if an error occur.
              bool    isLocked = false;
              int     nTries = 25;
              do {
                  try {
                      m_hpartfile.LockRange(item->start, lenData);
                      isLocked = true;
                  } catch ( CFileException* e ) {
                      if( e->m_cause == e->lockViolation ) {  // Give the lock a chance to clear
                          e->Delete();
                      } else throw e; // Some other exception, rethrow it.
                      Sleep(0);   // Give whichever process that is locking the file a chance to clear its lock.
                  }
              } while (!isLocked && bForced && --nTries); // Loop until the lock is acquired or the counter expires when forced to dump.
  
              if (isLocked) {
                  m_hpartfile.Seek(item->start, CFile::begin);
                  m_hpartfile.Write(item->data, lenData);
                  m_hpartfile.UnlockRange(item->start, lenData);
              
                  // SLUGFILLER: SafeHash - could be more than one part
                  for (uint32 curpart = item->start/PARTSIZE; curpart <= item->end/PARTSIZE; curpart++)
                      changedPart[curpart] = true;
                  // SLUGFILLER: SafeHash
  
                  // Decrease buffer size
                  m_nTotalBufferData -= lenData;
  
                  // Release memory used by this item
                  delete [] item->data;
                  delete item;
  
                  // Delete the flushed record.
                  m_BufferedData_list.RemoveAt(pos2);
  
              } else {
                  theApp.AddDebugLogLine(false, "Failed to obtain lock in %s, offset %i, length %i.", m_hpartfile.GetFilePath(), item->start, lenData);
              }
          }
  [...]
  }
  

Moonlight's Guaranteed Minimum Download Slot Speed (MGMDSS, Experimental)

I wrote this from curiosity. I read some people complaining about how eMule lets download slots starve when previous download slots are using up all the download bandwidth and how this occasionally cause their clients to lose a download slot for a rare file. Since I only use eMule for rare files, I figured that if I help people get their rare files, it should eventually help me to get my own rare files. That's what Moonlight's Save Upload Queue Wait Time (MSUQWT, last updated: 2004-02-04) was all about and the code here is also in the same category. I often see failed upload sessions for rare files and I suppose these failed sessions are due to this issue. After digging in for a while, I decided that it would be worth a look and this is how these code changes came to be.

What it does: these code changes set aside 20% of your download bandwidth to be evenly distributed across all active download slots. It also splits up the download list into three priority bins to reduce list checking overhead. (A gain currently mostly offset by the cross-checking I put in there while I am not 100% certain everything is in there and in perfect working condition.)

Note:

eMule appears to have speed management issues with low download speed limits on fast networks (LAN) very similar to the ones it has at high speeds. In both cases, it causes it to drop connections for no apparent reasons. At low download speed limits, the senders have highly irregular transmission speed

For the source code files, download the newest source zip on the Moonlight's Mod (Upcoming build: 0.42b-0.22g, 2004-03-03) page.

• PartFile.h - add an extra parameter to CPartFile::Process to specify the reserved bandwidth.

  class CPartFile : public CKnownFile {
  [...]
      public:
          uint32  Process(uint32 reducedownload,uint8 m_icounter, uint32 downspeedreserved);  // Moonlight: GMDSSL
  [...]
  };
  

• PartFile.cpp :
  • CPartFile::Process : Add the reserved bandwidth to the limit.
  • CPartFile::SetDownPriority : Perform the priority bin change when changing priority.

• uint32 CPartFile::Process(uint32 reducedownload, uint8 m_icounter/*in percent*/, uint32 downspeedreserved)  // --> Moonlight: GMDSS
  {
  [...]
      if(m_icounter < 9){ // Moonlight: this was probably meant to be every second... <10 means 11 iterations, 1.1 second.
  [...]
                      if(reducedownload){
                          uint32 limit = (reducedownload * cur_datarate + downspeedreserved * 100) /1000;         // --> Moonlight: GMDSS
                          //if (cur_datarate < downspeedreserved) limit = downspeedreserved / 10;
                          if(limit<1000 && reducedownload == 200)
                              limit +=1000;
                          else if(limit<1)
                              limit = 1;
                          cur_src->socket->SetDownloadLimit(limit);
                      }
                  }
              }
          }
      }
      else{
  [...]
                                  if (reducedownload && cur_src->GetDownloadState() == DS_DOWNLOADING){
                                      uint32 limit = (reducedownload * cur_datarate + downspeedreserved * 100) /1000;         // --> Moonlight: GMDSS
                                      //if (cur_datarate < downspeedreserved) limit = downspeedreserved / 10;
                                      if (limit < 1000 && reducedownload == 200)
                                          limit += 1000;
                                      else if (limit < 1)
                                          limit = 1;
                                      cur_src->socket->SetDownloadLimit(limit);
                                  }
                                  else
                                      cur_src->socket->DisableDownloadLimit();
  [...]
  }
  
  void CPartFile::SetDownPriority(uint8 np){
      theApp.downloadqueue->PrioBinRemoveFile(this);  // Moonlight: GMDSS
      m_iDownPriority = np;
      theApp.downloadqueue->PrioBinAddFile(this); // Moonlight: GMDSS
      theApp.downloadqueue->SortByPriority();
      theApp.downloadqueue->CheckDiskspace(); // SLUGFILLER: checkDiskspace
      UpdateDisplayedInfo(true);
      SavePartFile();
  }
  

• DownloadQueue.h : Add priority bins and bin management/debugging functions.

  class CDownloadQueue: public CLoggable
  {
  [...]
  public:
      // Moonlight: GMDSS - Manage priority bins, mostly for internal and CPartFile::SetDownPriority usage.
      bool    PrioBinAddFile(CPartFile *pPartFile);
      bool    PrioBinRemoveFile(CPartFile *pPartFile);
      // Moonlight: GMDSS - The two functions below are mainly for debugging/testing purposes.
      void    PrioBinCrossCheck(void);
      CString &PrioBinName(uint8 prio);
  [...]
  private:
      // Moonlight: GMDSS - Guaranteed Minimum Download Slot Speed, bin the files by priority for anything that discriminates by priority.
      CTypedPtrList<CPtrList, CPartFile*> m_binPrioHigh;
      CTypedPtrList<CPtrList, CPartFile*> m_binPrioNormal;
      CTypedPtrList<CPtrList, CPartFile*> m_binPrioLow;
      uint32  m_downactive;   // Number of active download slots.
  [...]
  };
  

• DownloadQueue.cpp : Implement the extra function and bin managment.
  • CDownloadQueue::CDownloadQueue : initialize m_downactive
  • CDownloadQueue::Process
    • Compute the amount of reserved bandwidth to pass to CPartFile::Process
    • Change the counter to have a 10sec turnabout time for the full bin check. (would be faster to clear the bins and rebuild them. I'm doing it this way to make it easier to determine if I have missed anything.

• CDownloadQueue::CDownloadQueue(CPreferences* in_prefs,CSharedFileList* in_sharedfilelist){
  [...]
      m_downactive = 0;   // Moonlight: GMDSS
  }
  
  void CDownloadQueue::Process(){
      
      ProcessLocalRequests(); // send src requests to local server
  
      uint32 downspeed = 0;
  
      // Moonlight: GMDSS - reserved per-slot speed = 25/100 (25%) of total download speed evenly distributed across active DL slots.
      uint32 downspeedmax = app_prefs->GetMaxDownload();
      uint32 downspeedreserved = (downspeedmax * 1024 * 25/100) / (m_downactive + (m_downactive?0:1));
  
      if ( downspeedmax != UNLIMITED && datarate > 1500){
          downspeed = (downspeedmax*1024*100 * 75/100)/(datarate+1); //(uint16)((float)((float)(app_prefs->GetMaxDownload()*1024)/(datarate+1)) * 100);
          if (downspeed < 50)
              downspeed = 50;
          else if (downspeed > 200)
              downspeed = 200;
      }
  
      while(avarage_dr_list.GetCount()>0 && (GetTickCount() - avarage_dr_list.GetHead().timestamp > 10*1000) )
          m_datarateMS-=avarage_dr_list.RemoveHead().datalen;
      
      if (avarage_dr_list.GetCount()>1){
          datarate = m_datarateMS / avarage_dr_list.GetCount();
      } else {
          datarate = 0;
      }
  
      uint32 datarateX=0;
      ++udcounter %= 100; // Moonlight: GMDSS - Want to do a full bin check every now an then so provide a 10sec loop.
  
      //filelist is already sorted by prio, therefore I removed all the extra loops..
      for (POSITION pos =filelist.GetHeadPosition();pos != 0;filelist.GetNext(pos)){
          CPartFile* cur_file =  filelist.GetAt(pos);
          if ((cur_file->GetStatus() == PS_READY || cur_file->GetStatus() == PS_EMPTY)/* && cur_file->GetDownPriority() == PR_HIGH*/){
              datarateX += cur_file->Process(downspeed, udcounter, downspeedreserved);
          }
          else{
              //This will make sure we don't keep old sources to paused and stoped files..
              cur_file->StopPausedFile();
          }
      }
  
      TransferredData newitem = {datarateX, ::GetTickCount()};
      avarage_dr_list.AddTail(newitem);
      m_datarateMS+=datarateX;
  
      if (udpartcount == 4){
          if (theApp.serverconnect->IsUDPSocketAvailable()){
              if((!lastudpstattime) || (::GetTickCount() - lastudpstattime) > UDPSERVERSTATTIME){
                  lastudpstattime = ::GetTickCount();
                  theApp.serverlist->ServerStats();
              }
          }
      }
  
      // Moonlight: I suppose this was meant to be every 1sec instead of 1.1sec.
      if (udpartcount == 9){
          if (theApp.serverconnect->IsUDPSocketAvailable()){
              if ((!lastudpsearchtime) || (::GetTickCount() - lastudpsearchtime) > UDPSERVERREASKTIME)
                  SendNextUDPPacket();
          }
      }
      // SLUGFILLER: checkDiskspace
      if ((!lastcheckdiskspacetime) || (::GetTickCount() - lastcheckdiskspacetime) > DISKSPACERECHECKTIME)
          CheckDiskspace();
      // SLUGFILLER: checkDiskspace
  }
  

Multiple Selection A4AF

When I select multiple files to "Switch sources to any other file", I typically do not want the sources to end up on any other file on which I used "Send to any other file". The stock 0.30c code does not have multiple-selection A4AF to begin with so here it is:

• PartFile.h

  class CPartFile : public CKnownFile {
  [...]
  public:
      void    SetA4AFAnyOther(bool flagAnyOther)  {m_A4AFAnyOther = flagAnyOther;}    // Moonlight: Multiple Selection A4AF to Any Other file.
      bool    GetA4AFAnyOther(void)               {return m_A4AFAnyOther;}            // Moonlight: Multiple Selection A4AF to Any Other file.
  [...]
  private:
      bool    m_A4AFAnyOther;     // Moonlight: Multiple Selection A4AF to Any Other file.
  [...]
  };
  

• PartFile.cpp

  void CPartFile::Init(){
      m_A4AFAnyOther = false;     // Moonlight: Multiple Selection A4AF to Any Other file.
  [...]
  }
  

• DownloadListCtrl.cpp
  • Change the A4AF "To Any Other" to support multiple selection and set the A4AF 'do not assing to this' flag in CPartFile so swapping sources will not end up on a file we're trying to remove sources from.
  • Change the A4AF "To This File" to support multiple selection. This implementation tries to evenly redistribute swapped sources across selected files.

• BOOL CDownloadListCtrl::OnCommand(WPARAM wParam,LPARAM lParam ){
  [...]
                  case MP_ALL_A4AF_TO_THIS: {
                          SetRedraw(false);
                          if (!selectedCount) break;
                  // Moonlight: Multiple Selection A4AF to these files.
                          POSITION pos1, pos2, pos3;
                          CPartFile       *cur_file;
                          CPartFile       *old_file;
                          CUpDownClient   *cur_source;
                          theApp.downloadqueue->DisableAllA4AFAuto();
  
                      // Moonlight : Step 1 - Remove any invalid items.
                          for (pos1 = pos2 = selectedList.GetHeadPosition(); pos2 = pos1;) {
                              cur_file = selectedList.GetNext(pos1);
                              if (!cur_file->IsPartFile() && (cur_file->GetStatus(true) == PS_READY || cur_file->GetStatus(true) == PS_EMPTY))
                                  selectedList.RemoveAt(pos2);
                          }
  
                      // Moonlight : Step 2 - Assemble a list of re-assignable clients, re-assign them only once.
                          CTypedPtrList<CPtrList, CUpDownClient*> sourceList;
  
                          for (pos1 = selectedList.GetHeadPosition(); pos1;) {
                              cur_file = selectedList.GetNext(pos1);
                              for (pos2 = cur_file->A4AFsrclist.GetHeadPosition(); pos2;) {
                                  cur_source = cur_file->A4AFsrclist.GetNext(pos2);
                                  if( cur_source->GetDownloadState() != DS_DOWNLOADING
                                      && ((cur_source->reqfile && !cur_source->reqfile->IsA4AFAuto())
                                          || cur_source->GetDownloadState() == DS_NONEEDEDPARTS)
                                      && !cur_source->IsSwapSuspended(file)
                                      && !sourceList.Find(cur_source))
                                          sourceList.AddTail(cur_source);
                              }
                          }
  
                      // Moonlight: Step 3 - Go through the client list, seeking possible swaps, rotating the selection to redistribute more evenly.
                          for (pos1 = sourceList.GetHeadPosition(); pos1;) {
                              cur_source = sourceList.GetNext(pos1);
                              for (pos2 = pos3 = selectedList.GetHeadPosition(); pos3 = pos2;) {
                                  cur_file = selectedList.GetNext(pos2);
                                  if (cur_file->A4AFsrclist.Find(cur_source)) {
                                      old_file = cur_source->reqfile;
                                      cur_source->DontSwapTo(cur_source->reqfile);
                                      if (cur_source->SwapToAnotherFile(true, false, false, cur_file) && old_file)
                                          cur_source->DontSwapTo(old_file);
                                      selectedList.RemoveAt(pos3);
                                      selectedList.AddTail(cur_file);
                                      break;
                                  }
                              }
                          }
                          SetRedraw(true);
                          this->UpdateItem(file);                     
                          break;
                      }
  
                  case MP_ALL_A4AF_TO_OTHER: {
                          SetRedraw(false);
                          if (!selectedCount) break;
                  // Moonlight: Multiple Selection A4AF to Any Other.
                          POSITION pos1, pos2;
                          CPartFile   *cur_file;
                          theApp.downloadqueue->DisableAllA4AFAuto();
                  
                      // Moonlight : Step 1 - Remove any invalid objects and set the "Any Other" flag on affected partfiles.
                          for (pos1 = pos2 = selectedList.GetHeadPosition(); pos2 = pos1;) {
                              cur_file = selectedList.GetNext(pos1);
                              if (cur_file->IsPartFile() && (cur_file->GetStatus(true) == PS_READY || cur_file->GetStatus(true) == PS_EMPTY))
                                  cur_file->SetA4AFAnyOther(true);
                              else
                                  selectedList.RemoveAt(pos2);
                          }
  
                      // Moonlight : Step 2 - Repeat the re-assignment for each file.
                          for (pos1 = selectedList.GetHeadPosition(); pos1;) {
                              cur_file = selectedList.GetNext(pos1);
                              for (int sl=0;sl<SOURCESSLOTS;sl++)
                              {
                                  if (!cur_file->srclists[sl].IsEmpty())
                                  {
                                      POSITION pos1, pos2;
                                      for(pos1 = cur_file->srclists[sl].GetHeadPosition(); (pos2 = pos1) != NULL;)
                                      {
                                          cur_file->srclists[sl].GetNext(pos1);
                                          cur_file->srclists[sl].GetAt(pos2)->SwapToAnotherFile(false, false, false, NULL);
                                      }
                                  }
                              }
                          }
  
                      // Moonlight : Step 3 - Reset the A4AF Any Other flags.
                          for (pos1 = selectedList.GetHeadPosition(); pos2 = pos1;) {
                              cur_file = (CPartFile*) selectedList.GetNext(pos1);
                              cur_file->SetA4AFAnyOther(false);
                          }
                          SetRedraw(true);
                          break;
                      }
  [...]
  }
  

• DownloadClient.cpp

  bool CUpDownClient::SwapToAnotherFile(bool bIgnoreNoNeeded, bool ignoreSuspensions, bool bRemoveCompletely, CPartFile* toFile){
  [...]
      if (!m_OtherRequests_list.IsEmpty()){
          usedList = &m_OtherRequests_list;
          for (POSITION pos = m_OtherRequests_list.GetHeadPosition();pos != 0;m_OtherRequests_list.GetNext(pos)){
              cur_file = m_OtherRequests_list.GetAt(pos);
              if (cur_file != reqfile && theApp.downloadqueue->IsPartFile(cur_file) && !cur_file->IsStopped() 
                  && (cur_file->GetStatus(false) == PS_READY || cur_file->GetStatus(false) == PS_EMPTY) &&
                  !cur_file->GetA4AFAnyOther())   // Moonlight: Multiple Selection A4AF to Any Other file - do not re-assign to an any other.
              {
  [...]
      if (!SwapTo && bIgnoreNoNeeded){
          usedList = &m_OtherNoNeeded_list;
          for (POSITION pos = m_OtherNoNeeded_list.GetHeadPosition();pos != 0;m_OtherNoNeeded_list.GetNext(pos)){
              cur_file = m_OtherNoNeeded_list.GetAt(pos);
              if (cur_file != reqfile && theApp.downloadqueue->IsPartFile(cur_file) && !cur_file->IsStopped() 
                  && (cur_file->GetStatus(false) == PS_READY || cur_file->GetStatus(false) == PS_EMPTY) &&    
                  !cur_file->GetA4AFAnyOther())   // Moonlight: Multiple Selection A4AF to Any Other file - do not re-assign to an any other.
              {
  [...]
  

Hits since December 5, 2003:

---