Moonlight's Save Upload Queue Wait Time (MSUQWT, last updated: 2004-02-04)

Lastest updates:

• 2004-01-11: Started re-reading the code and doing some minor touch-ups.
  • Fixed a few copy-paste errors and removed some dead/erroneous links in the documentation.
  • Removed one or two redundant calls.
  • Changed the priority modifiers - mods that make SUQWT optional should probably use an 'if' to select which priority set to use... standard eMule priorities when SUQWT is off, my modified priorities or something like them when SUQWT is enabled.
• 2004-01-17: Implemented wait time credit depreciation on failed transfer sessions to prevent 'broken' clients from hogging the queue top with repeated failed transfer sessions - on session failure, save only 90% of the wait time.
• 2004-02-04: Changed the depreciation to be proportionnal to the amount of data transferred in the session. (This also includes an extra 10% "penalty" to make abuse highly unlikely.)
  • Note: I will not have time to test the changes myself for a while since I have to prepare for mid-term exams season which begins on Friday. Until then, do not be surprised if there happens to be one or two obvious mistakes.
  • Other note: The source zip has not been updated in quite a while so the January changes are not in it. I will probably not do another zip until the next official/stable (0.30f / 0.31a / 0.4?a) eMule release is out.
  • Changed functions:
    • CUploadQueue::RemoveFromUploadQueue - Changed for proportional time credit reduction.
    • CClientCredits::SaveUploadQueueWaitTime - Changed to use a 0-100 integer instead of a float.
• 2004-02-29:
  • Simplified the time correction in CUploadQueue::RemoveFromUploadQueue.
  • Removed redundant and technically self-cancelling time correction in CUpDownClient::GetWaitStartTime.

Ever tried to get some rare files using eMule only to see sources quit on you after waiting for days to get from #6000 to #500? Well, with SUQWT, this is not going to be as much of a problem anymore.

How does SUQWT help?

1. Client A is sharing some rare files with a queue of 5000 and an average uptime of 20-30 hours and an average downtime of 10-20h.
2. When joining the queue without SLS, you always end up around 5000 unless you get a lucky shot and it takes 24h to get to 4000, at which point the source disappears.
3. Next time the source comes back, instead of having an initial wait time of 0h, you are at 24h and will start at around 4000. By the time the source disappears again, you might manage to reach 3000. Without SUQWT, you would again go back to ~5000 the next time you find the source.
4. Step #3 repeats until you get a send slot and have a successful upload session.
5. After the session has ended, the wait time are reset and the cycle restarts anew for the next session.

Basically, SUQWT allows one to carry wait time from previous dead-end upload queue waiting sessions over to the next upload queue sessions until one manages to get through the upload queue, somewhat like holding a spot in a waiting line by using your gym bag or other article - if you're not there, the bag stays there and people move around it. (in an ideal civilized world anyway)

The point behind SUQWT is that many sources that have rare files often have very long queues and go offline every now and then. Each time these sources disappear, your queue ranking you had to wait 2-3 days to get from #1000 to #10 is lost and you have to start waiting all over again. These excessively long source queues make the sources worth little more than dead sources since you have little to no chance of ever getting an upload slot through normal means and downloads take forever if there are few sources or the sources have so many favoured regulars in their client queues that clients with no credit are eternally stuck near the upload queue's end.

Warning:

Because saving the wait time requires a new CreditStruct, the clients.met file's format will not be backwards-compatible so you should make a backup of the clients.met file before using the SUQWT-modified client since the next official eMule may not integrate this enhancement or not integrate it the exact same way. For this reason, SUQWT now also saves a 30c-format file in clients.met.30c.bak. Just replace clients.met with clients.met.30c.bak to restore 30c credits format.

Deprecated:

CREDITFILE_VERSION_30_SUQWTv1 and CreditStruct_30c_SUQWTv1 should be replaced by v2 ASAP since choosing the next number up for a mod was not a good idea for obvious reasons - gets in the devs' way. v2 uses an 'out of the way' version ID to avoid possible clashes.

I recommend using 0x80 as the base version for unofficial CreditStruct implementations so the stock client could check for the 8th bit to detect unofficial formats.

Warning:

All full files and code fragments are taken from Moonlight's Mod (Upcoming build: 0.42g-0.22g, ????) and may contain changes unrelated to the ones enumerated here.

Implementation

Implementing this requires surprisingly few changes (ommitting comments and structure/code duplications, the total being about 50 lines) in the following files:

• Opcodes.h:
• ClientCredits.h:
• ClientCredits.cpp:
• UploadQueue.cpp:
• UploadClient.cpp

For complete source files for Moonlight's Save Upload Queue Wait Time (MSUQWT, last updated: 2004-02-04) and more, download the newest Moonlight's Mod (Upcoming build: 0.42g-0.22g, ????) zipped changed sources.

Opcodes.h:

• Add the new credit file version numbers.

  #define CREDITFILE_VERSION_30_DYN       0x81    // Moonlight: Dynamic Transportable CreditStruct.
  #define CREDITFILE_VERSION_30_SUQWTv2   0x80    // Moonlight: SUQWT CreditStruct v2.
  #define CREDITFILE_VERSION_30_SUQWTv1   0x13    // Moonlight: SUQWT CreditStruct v1.
  #define CREDITFILE_VERSION_30           0x12
  #define CREDITFILE_VERSION_29           0x11
  #define CREDITFILE_VERSION              CREDITFILE_VERSION_30_SUQWTv2   // Define the current version number.
  

ClientCredits.h:

• Modify CreditStruct to add the Wait-Time variables.
• Add functions to Save and Clear the wait times.
• Add a client expiry check function.

  // Moonlight: SUQWT: Add the wait time data to the structure.
  #pragma pack(1)
  struct CreditStruct_30c{    // Moonlight: Formerly only "CreditStruct"
  [...]
  };
  #pragma pack()
  
  typedef CreditStruct_30c    CreditStruct;   // Moonlight: Standard name for the credit structure.
  
  // Moonlight: SUQWT: Add functions to save and clear wait times.
  class CClientCredits: public CLoggable
  {
  [...]
  public:
      bool    IsActive(uint32 dwExpire);                      // Moonlight: SUQWT, new function to determine if the record has expired.
      void    SaveUploadQueueWaitTime(int iKeepPct = 100);    // Moonlight: SUQWT
      void    ClearUploadQueueWaitTime();                     // Moonlight: SUQWT
  [...]
  };
  

ClientCredits.cpp:

• Change CClientCreditsList::LoadList() to import eMule 0.30c CreditStruct.
• Change CClientCredits::SetSecWaitStartTime(uint32 dwForIP) to use the time credit.
• Implement the Save/Clear functions.
• Implement the client expiry check function.

  // Moonlight: SUQWT - Conditions to determine an active record.
  // Returns true if the client has been seen recently
  bool CClientCredits::IsActive(uint32 dwExpired) {
      return (GetUploadedTotal() || GetDownloadedTotal() || m_pCredits->nSecuredWaitTime || m_pCredits->nSecuredWaitTime) &&
              (m_pCredits->nLastSeen >= dwExpired);
  }
  
  // Moonlight: SUQWT: Change the file import 0.30c and SUQWTv1 format.
  void CClientCreditsList::LoadList()
  {
  [...]
      // Moonlight: SUQWT - Import CreditStruct from 0.30c and SUQWTv1
          if (version != CREDITFILE_VERSION_30_SUQWTv1 && version != CREDITFILE_VERSION_30_SUQWTv2 &&
              version != CREDITFILE_VERSION_30 && version != CREDITFILE_VERSION_29){
          AddLogLine(false, GetResString(IDS_ERR_CREDITFILEOLD));
          file.Close();
          return;
      }
  [...]
      for (uint32 i = 0; i < count; i++){
          newcstruct = new CreditStruct;
          memset(newcstruct, 0, sizeof(CreditStruct));
          
          // --> Moonlight: SUQWT - import 0.30c and 30c-SUQWTv1 structures.
              if (version == CREDITFILE_VERSION)
                  file.Read(newcstruct, sizeof(CreditStruct_30c_SUQWTv2));
              else if (version == CREDITFILE_VERSION_30_SUQWTv1) {
                  file.Read(((uint8*)newcstruct) + 8, sizeof(CreditStruct_30c_SUQWTv1) - 8);
                  file.Read(((uint8*)newcstruct), 8);
              }
              else if (version == CREDITFILE_VERSION_30)
                  file.Read(((uint8*)newcstruct) + 8, sizeof(CreditStruct_30c));
              else
                  file.Read(((uint8*)newcstruct) + 8, sizeof(CreditStruct_29a));
          // <-- Moonlight: SUQWT
          
          if (newcstruct->nLastSeen < dwExpired){
              cDeleted++;
              delete newcstruct;
              continue;
          }
  
          CClientCredits* newcredits = new CClientCredits(newcstruct);
          m_mapClients.SetAt(CCKey(newcredits->GetKey()), newcredits);
      }
  [...]
  };
  
  // Moonlight: SUQWT - Save the wait times before saving the list, also save 0.30c-style credits.
  void CClientCreditsList::SaveList()
  {
  [...]   
      CFile fileBack; // Moonlight: SUQWT - Also open a file to save original 30c format.
      if (!fileBack.Open(name+".30c.bak", CFile::modeWrite|CFile::modeCreate|CFile::typeBinary, &fexp)){
          CString strError(GetResString(IDS_ERR_FAILED_CREDITSAVE));
          TCHAR szError[MAX_CFEXP_ERRORMSG];
          if (fexp.GetErrorMessage(szError, ELEMENT_COUNT(szError))){
              strError += _T(" - ");
              strError += szError;
          }
          AddLogLine(true, _T("%s"), strError);
          return;
      }
  
      fileBack.Seek(5, CFile::begin);
      uint32 count = m_mapClients.GetCount();
      BYTE* pBuffer = new BYTE[count*sizeof(CreditStruct)];
      CClientCredits* cur_credit;
      CCKey tempkey(0);
      POSITION pos = m_mapClients.GetStartPosition();
      const uint32 dwExpired = time(NULL) - 12960000; // today - 150 day
      count = 0;
      while (pos)
      {
          m_mapClients.GetNextAssoc(pos, tempkey, cur_credit);
          if (cur_credit->IsActive(dwExpired))    // Moonlight: SUQWT - Call the expiry check function.
          {
              cur_credit->SaveUploadQueueWaitTime();  // Moonlight: SUQWT
              memcpy(pBuffer+(count*sizeof(CreditStruct)), cur_credit->GetDataStruct(), sizeof(CreditStruct));
              fileBack.Write(((uint8*)cur_credit->GetDataStruct()) + 8, sizeof(CreditStruct_30c));    // Moonlight: SUQWT - Save 0.30c CreditStruct
              count++; 
          }
      }
  
      uint8 version = CREDITFILE_VERSION_30_SUQWTv2;
      fileBack.SeekToBegin();
      fileBack.Write(&version, 1);
      fileBack.Write(&count, 4);
      fileBack.Flush();
      fileBack.Close();
  [...]
  }
  
  // Moonlight: SUQWT - Save the wait times. fKeep is the fraction of the time credit to keep - used to prevent clients from hogging the queue top with repeated failed requests.
  void CClientCredits::SaveUploadQueueWaitTime(int iKeepPct) {
      if (m_dwUnSecureWaitTime) m_pCredits->nUnSecuredWaitTime = ((GetTickCount() - m_dwUnSecureWaitTime) / 100) * iKeepPct;
      if (m_dwSecureWaitTime) m_pCredits->nSecuredWaitTime = ((GetTickCount() - m_dwSecureWaitTime) / 100) * iKeepPct;
      ClearWaitStartTime();
  }
  
  // Moonlight: SUQWT - Clear the wait times.
  void CClientCredits::ClearUploadQueueWaitTime() {
      m_pCredits->nUnSecuredWaitTime = 0;
      m_pCredits->nSecuredWaitTime = 0;
      // Doing SaveUploadQueueWaitTime(0) should be reduced to something equivalent during compile.
  }
  
  // Moonlight: SUQWT: Adjust to take previous wait time into account.
  void CClientCredits::SetSecWaitStartTime(uint32 dwForIP){
      m_dwUnSecureWaitTime = ::GetTickCount() - m_pCredits->nUnSecuredWaitTime - 1;   // Moonlight: SUQWT
      m_dwSecureWaitTime = ::GetTickCount() - m_pCredits->nSecuredWaitTime - 1;       // Moonlight: SUQWT
      m_dwWaitTimeIP = dwForIP;
  }
  

UploadQueue.cpp:

• Change CUploadQueue::RemoveFromUploadQueue to reset or save wait time depending on transfer session transfer volume.
• Change CUploadQueue::RemoveFromWaitingQueue to save queue wait time before removing from queue.

  // Moonlight: SUQWT: Reset wait time on session success, save it on failure.
  bool CUploadQueue::RemoveFromUploadQueue(CUpDownClient* client, bool updatewindow){
  [...]
          if (client->GetSessionUp() > 0) {
              int keeppct = ((100 * client->GetSessionUp()) >> 23) - 15;
              if (keeppct < 0)    keeppct = 0;
              client->Credits()->SaveUploadQueueWaitTime(keeppct);
              ++successfullupcount;
              totaluploadtime += client->GetUpStartTimeDelay()/1000;
          } else {
              ++failedupcount;
              client->Credits()->SaveUploadQueueWaitTime(90);
          }
  [...]
  }
  
  // Moonlight: SUQWT: Save queue wait time and clear wait start times before removing from queue.
  void CUploadQueue::RemoveFromWaitingQueue(POSITION pos, bool updatewindow){ 
      CUpDownClient* todelete = waitinglist.GetAt(pos);
      waitinglist.RemoveAt(pos);
      if (updatewindow)
          theApp.emuledlg->transferwnd.queuelistctrl.RemoveClient(todelete);
      todelete->Credits()->SaveUploadQueueWaitTime(); // Moonlight: SUQWT
      todelete->SetUploadState(US_NONE);
  }
  

UploadClient.cpp

• Clear queue wait time to avoid crediting wait time while banned.
• [2004-01-12] Change queue priorities to increase the spread.

  uint32 CUpDownClient::GetScore(bool sysvalue, bool isdownloading, bool onlybasevalue){
  [...]
      if(GetUploadFileID() != NULL){ 
          if(theApp.sharedfiles->GetFileByID(GetUploadFileID()) != NULL){
              switch(theApp.sharedfiles->GetFileByID(GetUploadFileID())->GetUpPriority()){ 
              // --> Moonlight: SUQWT - Changed the priority distribution for a wider spread.
                  case PR_VERYHIGH:
                      filepriority = 27;  // 18, 50% boost    <-- SUQWT - original values commented.
                      break;
                  case PR_HIGH: 
                      filepriority = 12;  // 9, 33% boost
                      break; 
                  case PR_LOW: 
                      filepriority = 5;   // 6, 17% reduction
                      break; 
                  case PR_VERYLOW:
                      filepriority = 2;   // 2, no change
                      break;
                  case PR_NORMAL: 
                      default: 
                      filepriority = 8;   // 7, 14% boost
                  break; 
              } 
          // <-- Moonlight: SUQWT
          } 
      } 
  [...]
      if (onlybasevalue)
          fBaseValue = 100;
      else if (!isdownloading)
          fBaseValue = (float)(::GetTickCount()-GetWaitStartTime())/1000;
      else{
          // we dont want one client to download forever
          // the first 15 min downloadtime counts as 15 min waitingtime and you get a 15 min bonus while you are in the first 15 min :)
          // (to avoid 20 sec downloads) after this the score won't raise anymore 
          fBaseValue = (float)(m_dwUploadTime-GetWaitStartTime());
          // Moonlight: SUQWT - I'm exploiting negative overflows to adjust wait start times. Overflows should not be an issue as long
          // as queue turnover rate is faster than 49 days.
          // ASSERT ( m_dwUploadTime-GetWaitStartTime() >= 0 ); //oct 28, 02: changed this from "> 0" to ">= 0"
          fBaseValue += (float)((int)(::GetTickCount() - m_dwUploadTime) > 900000)? 900000:1800000;
          fBaseValue /= 1000;
      }
  [...]
  }
  
  // Moonlight: SUQWT - Reset the wait time on ban, do not give time credit for banned clients queue time!
  void CUpDownClient::Ban(){
      theApp.clientlist->AddTrackClient(this);
      ClearWaitStartTime();   // Moonlight: SUQWT
  [...]
  }
  
  // Moonlight: SUQWT - Compare linear time instead of time indexes to avoid overflow-induced false positives.
  uint32 CUpDownClient::GetWaitStartTime(){
      if (credits == NULL){
          ASSERT ( false );
          return 0;
      }
      uint32 dwResult = credits->GetSecureWaitStartTime(GetIP());
      // Moonlight: SUQWT - To avoid erroneous check due to overflow, cast the difference to int.
      if (((int)(m_dwUploadTime - dwResult) < 0) && IsDownloading()){
          //this happens only if two clients with invalid securehash are in the queue - if at all
          dwResult = m_dwUploadTime-1;
  
          DEBUG_ONLY(AddDebugLogLine(false,"Warning: CUpDownClient::GetWaitStartTime() waittime Collision (%s)",GetUserName()));
      }
      return dwResult;
  }
  

  Hits since December 5, 2003:
  
  

---