Reduce memory usage to 15% of its current level.

[Steve Coffee]

This is a little long, but I think it can reduce memory usage to about 15% of it's current level, so bear with me.  (I was encouraged by Jeroen's implementation of my last memory optimization idea, so blame him.  )

High memory usage has been a recurring theme for JKDefrag and MyDefrag.  I believe that the numbers are something like 600 bytes per file for JKDefrag and 300 bytes per file for MyDefrag.  This is just the initial cost of scanning the MFT.  Sorting options use even more.  I had a flash of insight on this today that I wanted to share before I forgot about it.

The in-memory structures are used to support two very different tasks:

  Category 1: Some of the data (names, dates, attributes, full paths, etc.) is needed to support determining a target position for a file: a zone and sometimes a sorted position within the zone.  Once a target position has been determined for a file, this information is no longer needed.  This can all be disposed of before file placement begins.  For files in unsorted zones, this can be discarded almost immediately.

  Category 2: Data needed to support the file placement process.  I am not certain of the exact data structure, but let's assume we just have one record per file segment.  The required data would be something like:
    file id (8 bytes)
    starting internal file position (8 bytes)
    starting lcn (8 bytes)
    length (8 bytes)
    target zone (2 byte)
    target sorted position within zone (4 bytes)
    pointer to next segment (8 bytes)
This category 2 data comes to approximately 46 bytes per segment, or about 15% of the current memory requirement.

I propose that it work this way (warning, psuedocode ahead):

Code:

preload the volume booleans
for each MFT item
  if we can determine the zone without using a full path then
    store the target zone
  end if
  if (this item is a directory) or (the target zone is either undetermined or requires sorting) then
      create a category 1 data structure for this item
  end if
next MFT item
for each MFT item with a undetermined zone
  calculate the zone using full path information from the category 1 data and store it in the category 2 data
  if (this record is not a directory) and (the target zone is unsorted)
    dispose of the category 1 data for this item
  end if
next MFT item
for each sorted zone, perform the sort and store the target position in the category 2 structure
dispose of all the category 1 data structures
perform the file placement using the category 2 data.

[Steve Coffee]

Let me address some likely criticism, just to cover my bases:

- This is a big task to fix something that is not technically broken (although MyDefrag is unusable on many file servers due to memory constraints).

- This is an optimization issue and Jeroen is focused on real problems in the beta.  (I Agree.  This can wait.)

- This would prevent MyDefrag from being able to show file names when you mouse over the volume map.  (So only dispose of the category 1 data when LowMemoryUsage() is turned on.)

- Sometimes the file placement engine requires the file name.  (I think that in this relatively rare case, the file name should be looked up on demand, rather than storing all the file names just in case they are needed.)

[cg]

Very interesting idea!

[jeroen]

Thanks for sharing your idea, I appreciate it! It's an interesting idea to only load necessary data from the MFT, and to throw away data after it's no longer needed. However, there are some problems, as Steve Coffee points out. Implementing this is far more complex than the directory-filename idea, and that's not what I am looking for right now. Another problem is that the program can no longer display filenames (and other data) on mouse-over, and would not be able to calculate statistics at the end. There is a myriad of other complications that I don't have time for to go into right now. Nevertheless, it's an interesting idea and I'm putting it in my hat to think about.

[Steve Coffee]

One interesting benefit of this approach is that you know the target location of all the files before you move any of them.  This could eliminate some unnecessary file moves.  When a file is in the wrong zone and needs to be moved out, the current algorithm moves the file to a temporary location (perhaps more than once) while processing other zones and then correctly places the file only when it gets to that file's zone.  This change would allow the application to move the file directly into the proper zone (most of the time).

It might also make it simpler to do a back-to-front defrag pass or otherwise handle packing files to the end of the drive.