[Script] Robust System Disk script (good access performance with less shuffling)

[Bill]

System Disk Robust:

Code:

Title('System Disk Robust')

Description("Minimizes data shuffling while keeping related files close together.
Provides good bootup, application startup, and small-file access performance.

Requires volume free-space to be at least 10% volume size.
The space used by small files should not fluctuate largely between script runs.

Both variables 'skipNumFiles' and 'skipBlockSize' must be
set to ZONE220N and ZONE222N respectively on this script's first run.
Doing so will properly order the zones and compact all files belonging to each zone.")


/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("MyDefrag.log","LogHeader")

/* Process System volume. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("MyDefrag.log","LogBefore")


  setVariable(dy,  86400)
  setVariable(wk,  604800)
  setVariable(mnt, 2628000)

  setVariable(recently, 4*dy)  // recently modified age <= 'recently'
  setVariable(rarely,   3*mnt) // rarely modified age > 'rarely'

  setVariable(small, 4MB)  // size of small files <= 'small'
  setVariable(large, 64MB) // size of large files > 'large'

  setVariable(skipNumFiles,  16)      // must set to ZONE220N on script's first run, then reset to original value
  setVariable(skipBlockSize, 4*small) // must set to ZONE222N on script's first run, then reset to original value

  setVariable(tolerance, 0.05*volumeSize) // normalized slackspace for zones to avoid re-positioning zones.


  /* Zone 1 : small and medium Windows boot files. */
  FileSelect
    size(0,large) and importListFromBootOptimize() and directoryPath("?:\Windows")
  FileActions
    sortByImportSequence(ascending skipBlock(skipNumFiles,skipBlockSize*2))
    addGap(roundUp(zoneEnd,small*2), doNotVacate)
  FileEnd

  /* Zone 2 : small program-hinted files. */
  FileSelect
    size(0,small) and importListFromProgramHints("*.pf")
  FileActions
    sortByImportSequence(ascending skipBlock(skipNumFiles,skipBlockSize))
  FileEnd

  /* Zone 3: scratch area : small and medium files modified recently. */
  FileSelect
    ((size(0,small) and (lastChange(recently ago,) or creationDate(recently ago,) or
        (lastAccess(recently ago,) and lastAccessEnabled(no))))
    or (size(small,large) and lastChange(recently ago,) and creationDate(,recently ago)))
    and not ((directory(yes)) or selectNtfsSystemFiles(yes))
  FileActions
    moveUpToZone()
    defragment()
    fastFill(withShuffling)
    addGap(roundUp(zoneBegin,tolerance), doNotVacate)
  FileEnd

  /* Zone 4: MFT and some other special NTFS files. */
  FileSelect
    selectNtfsSystemFiles(yes) and (size(0,large) or fileName("$MFT"))
  FileActions
    placeNtfsSystemFiles(ascending, maximum(128MB - mftSize, roundUp(mftSize*0.02,1MB)))
    addGap(roundUp(zoneEnd,small))
  FileEnd

  /* Zone 5: Directories. */
  FileSelect
    directory(yes)
  FileActions
    sortByName(ascending)
    addGap(roundUp(zoneEnd,small))
  FileEnd

  /* Zone 6: small files modified occasionally. */
  FileSelect
    size(0,small) and (lastChange(rarely ago,) or creationDate(rarely ago,) or
        (lastAccess(rarely ago,) and lastAccessEnabled(no)))
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles,skipBlockSize))
    addGap(roundUp(zoneEnd,tolerance*0.5), doNotVacate)
  FileEnd

  /* Zone 7: small files modified rarely. */
  FileSelect
    size(0,small)
  FileActions
    sortByName(ascending skipBlock(skipNumFiles,skipBlockSize))
    addGap(roundUp(zoneEnd,tolerance*0.25), doNotVacate)
  FileEnd

  /* Zone 8: medium files not modified recently. */
  FileSelect
    size(small,large) and lastChange(,recently ago) and creationDate(,recently ago)
  FileActions
    sortByNewestDate(ascending skipBlock(maximum(skipNumFiles,large/small), maximum(skipBlockSize,large)))
  FileEnd

  /* Zone 9: scratch area : medium and large files modified recently. */
  FileSelect
    size(small,large) or (size(large,0) and lastChange(recently ago,) and creationDate(,recently ago))
    or fileName("$UsnJrnl*")
  FileActions
    moveUpToZone()
    defragment(fast)
    fastFill()
    addGap(roundUp(zoneBegin,tolerance*2), doNotVacate)
  FileEnd

  /* Zone 10: spacehogs : less important files that take up a lot of space. */
  FileSelect
    all
  FileActions
    sortByNewestDate(ascending) // or moveToEndOfDisk() or defragment(fast) fastFill()
  FileEnd


  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("MyDefrag.log","LogFooter")

System Disk Robust v2:

Code:

Title('System Disk Robust v2') // Updated: June 6, 2011

Description("Minimizes data shuffling while keeping related files close together.
Provides good bootup, application startup, and small-file access performance.

Requires volume free-space to be at least 10% volume size.
The space used by small files should not fluctuate largely between script runs.
This script should be run weekly.

Both variables 'skipNumFiles' and 'skipBlockSize' must be
set to ZONE220N and ZONE222N respectively on this script's first run,
as well as on a monthly basis.
Doing so will properly order the zones and compact all files belonging to each zone.")


/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("MyDefrag.log","LogHeader")

/* Process System volume. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("MyDefrag.log","LogBefore")


  setVariable(dy,  86400)
  setVariable(wk,  604800)
  setVariable(mnt, 2628000)

  setVariable(recently, 3*dy) // recently modified age <= 'recently'
  setVariable(rarely,   5*wk) // rarely modified age > 'rarely'

  setVariable(small, 2MB)  // size of small files <= 'small'
  setVariable(large, 64MB) // size of large files > 'large'

  setVariable(skipNumFiles,  16)   // must set to ZONE220N on script's first run, then reset to original value
  setVariable(skipBlockSize, 16MB) // must set to ZONE222N on script's first run, then reset to original value

  setVariable(tolerance, 0.05*volumeSize) // normalized slackspace for zones to avoid re-positioning zones


  /* Zone 1 : small and medium Windows boot files. */
  FileSelect
    size(0,large) and importListFromBootOptimize() and directoryPath("?:\Windows")
  FileActions
    sortByImportSequence(ascending skipBlock(skipNumFiles*2,skipBlockSize*2))
    addGap(roundUp(zoneEnd,4MB))
  FileEnd

  /* Zone 2 : small program-hinted files. */
  FileSelect
    size(0,small) and importListFromProgramHints("*.pf")
  FileActions
    sortByImportSequence(ascending)
    addGap(roundUp(zoneEnd,tolerance*0.15))
  FileEnd

  /* Zone 3: MFT and some other special NTFS files. */
  FileSelect
    selectNtfsSystemFiles(yes) and (size(0,large) or fileName("$MFT"))
  FileActions
    placeNtfsSystemFiles(ascending, roundUp(mftSize+4MB,32MB)-mftSize)
    addGap(roundUp(zoneEnd,1MB))
  FileEnd

  /* Zone 4: directories. */
  FileSelect
    directory(yes)
  FileActions
    sortByName(ascending)
    addGap(roundUp(zoneEnd,2MB))
  FileEnd

  /* Zone 5: small files modified occasionally or recently. */
  FileSelect
    size(0,small) and (lastChange(rarely ago,) or creationDate(rarely ago,) or
        (lastAccess(rarely ago,) and lastAccessEnabled(no)))
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles*2,skipBlockSize))
    addGap(roundUp(zoneEnd,tolerance*0.25), doNotVacate)
  FileEnd

  /* Zone 6: small files modified rarely. */
  FileSelect
    size(0,small)
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles*2,skipBlockSize))
    addGap(roundUp(zoneEnd,tolerance*0.1))
  FileEnd

  /* Zone 7: scratch area : medium files modified recently. */
  FileSelect
    size(small,large) and lastChange(recently ago,) and creationDate(,recently ago)
  FileActions
    moveUpToZone()
    defragment()
    fastFill(withShuffling)
    addGap(roundUp(zoneBegin,tolerance*0.5), doNotVacate)
  FileEnd

  /* Zone 8: medium files not modified recently. */
  FileSelect
    size(small,large)
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles, maximum(skipBlockSize,large)))
    addGap(roundUp(zoneEnd,tolerance*0.2), doNotVacate)
  FileEnd

  /* Zone 9: scratch area : large files modified recently. */
  FileSelect
    size(large,0) and lastChange(recently ago,) and creationDate(,recently ago)
    or fileName("$UsnJrnl*")
  FileActions
    moveUpToZone()
    defragment(fast)
    fastFill()
    addGap(roundUp(zoneBegin,tolerance*2), doNotVacate)
  FileEnd

  /* Zone 10: spacehogs : less important files that take up a lot of space. */
  FileSelect
    all
  FileActions
    sortByNewestDate(ascending) // or moveToEndOfDisk() or defragment(fast) fastFill()
  FileEnd


  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("MyDefrag.log","LogFooter")

System Disk Robust Simplified:

Code:

Title('System Disk Robust Simplified') // Updated: June 6, 2011

Description("Minimizes data shuffling while keeping related files close together.
Provides good bootup, application startup, and small-file access performance.

Requires volume free-space to be at least 10% volume size.
This script should be run weekly.

Both variables 'skipNumFiles' and 'skipBlockSize' must be
set to ZONE220N and ZONE222N respectively on this script's first run.
Doing so will properly order the zones and compact all files belonging to each zone.")


/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("MyDefrag.log","LogHeader")

/* Process System volume. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("MyDefrag.log","LogBefore")


  setVariable(small, 2MB)  // size of small files <= 'small'
  setVariable(large, 64MB) // size of large files > 'large'

  setVariable(skipNumFiles,  16)   // must set to ZONE220N on script's first run, then reset to original value
  setVariable(skipBlockSize, 16MB) // must set to ZONE222N on script's first run, then reset to original value

  setVariable(tolerance, 0.05*volumeSize) // normalized slackspace for zones to avoid re-positioning zones


  /* Zone 1 : small and medium Windows boot files. */
  FileSelect
    size(0,large) and importListFromBootOptimize() and directoryPath("?:\Windows")
  FileActions
    sortByImportSequence(ascending skipBlock(skipNumFiles*2,skipBlockSize*2))
    addGap(roundUp(zoneEnd,4MB))
  FileEnd

  /* Zone 2 : small program-hinted files. */
  FileSelect
    size(0,small) and importListFromProgramHints("*.pf")
  FileActions
    sortByImportSequence(ascending)
    addGap(roundUp(zoneEnd,tolerance*0.14))
  FileEnd

  /* Zone 3: small files. */
  FileSelect
    size(0,small) and not ((directory(yes)) or selectNtfsSystemFiles(yes))
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles*2,skipBlockSize))
    addGap(roundUp(zoneEnd,tolerance*0.3), doNotVacate)
  FileEnd

  /* Zone 4: MFT and some other special NTFS files. */
  FileSelect
    selectNtfsSystemFiles(yes) and (size(0,large) or fileName("$MFT"))
  FileActions
    placeNtfsSystemFiles(ascending, roundUp(mftSize+4MB,32MB)-mftSize)
    addGap(roundUp(zoneEnd,1MB))
  FileEnd

  /* Zone 5: directories. */
  FileSelect
    directory(yes)
  FileActions
    sortByName(ascending)
    addGap(roundUp(zoneEnd,2MB))
  FileEnd

  /* Zone 6: medium files. */
  FileSelect
    size(small,large)
  FileActions
    sortByNewestDate(ascending skipBlock(skipNumFiles, maximum(skipBlockSize,large)))
    addGap(roundUp(zoneEnd,tolerance*0.2), doNotVacate)
  FileEnd

  /* Zone 7: scratch area : large files modified recently. */
  FileSelect
    (lastChange(2 days ago,) and creationDate(,2 days ago))
    or fileName("$UsnJrnl*")
  FileActions
    moveUpToZone()
    defragment(fast)
    fastFill()
    addGap(roundUp(zoneBegin,tolerance*2), doNotVacate)
  FileEnd

  /* Zone 8: spacehogs : less important files that take up a lot of space. */
  FileSelect
    all
  FileActions
    sortByNewestDate(ascending) // or moveToEndOfDisk() or defragment(fast) fastFill()
  FileEnd


  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("MyDefrag.log","LogFooter")

Use only one of the above scripts, because zone placement is different for each script.

System Disk Robust v2 partitions the data by both file modified date and file size, whereas System Disk Robust Simplified partitions the data only by file size; however, Simplified requires less available free space than v2 and zones should be more stable.

Monthly and Weekly scripts for System Disk Robust v2 and Simplified are attached to this post. There is no need to change variables in these attached scripts.

Last update: June 6, 2011

[Darlis]

First of all: This is not a complete script. Title and Description are missing, plus the VolumeSelection block. You only have the volume actions. Before posting a script you should let MyDefrag do a syntax check first.

You say that the script has a good startup performance but you only use ImportListFromBootOptimize() with files that haven't been accessed within a week. Usually, boot files are accessed daily. I guess that most of these files will end up in zone 8.

I don't think that the placing of the MFT and directories is optimal. These should be placed where "the action" is, since directories are often read and the MFT often read/changed. You've placed these files by rarely accessed (and changed) files. I would rather put them near the "daily scratch area",

[Bill]

Thanks for your constructive criticism, I appreciate it.

I've updated the first post with a complete script with some changes that address your concerns.

Good startup performance means the script reduces disk-head thrashing/seeking during startup by placing small BootOptimize files at the beginning of the volume in the order given by Layout.ini.
Also, programs loaded after the bootup/startup phase are usually loaded by superfetch, which is outside the intended scope of this script.
Finally, ProgramHints and BootOptimize lists change file entries frequently. By adding and removing files within those zones, those files would also be removed and added by other zones, thus making those zones unstable and causing a lot of data shuffling.

The Access timestamp is not changed when a file is read by an application such as a virus scanner (at least it shouldn't change if the application is well behaved).

I have now placed the MFT between "older than monthly" and "monthly to daily". Doing so places the MFT within the bulk (by file count, not file size) of files.

Edit:
It seems the SkipBlock option can leave sorted blocks scattered across the volume. I hope the SkipBlock option will be updated in a future version of MyDefrag to bring files from outside the zone into the zone by maybe using FastFill() (making sure to only select files outside the ZoneBegin and ZoneEnd boundaries) after the SortBy*.

The Robust System Disk script above seems to be robust enough to minimize such scattering without explicitly using FastFill() after the SortBy*.  When files are created or modified, they are placed in the 'daily scratch area' that shuffles and compacts them using fastFill(withShuffling), which breaks the order of some files. Then, if the files aren't modified daily, they move up to SortByNewestDate for which most of the files outside that zone should be out-of-order. Then, if the files aren't modified monthly, they move up to sortByName, for which almost all of the files outside that zone should be out-of-order initially. If they are modified along the way then they go back to 'daily scratch area', which shuffles them again.

[Bill]

I've updated the script in the first post yet again (hopefully for the last time).
I've changed the order of some zones. Windows boot files and program hints are placed at the beginning of the volume, followed by a large scratch area, which is then followed by the MFT and directories, followed by remaining small files, then medium files, then large files.

Being able to configure MyDefrag using scripts is a brilliant idea!
I now have a robust System Disk script that gives me the perfect disk.

[poutnik]

You say that the script has a good startup performance but you only use ImportListFromBootOptimize() with files that haven't been accessed within a week. Usually, boot files are accessed daily. I guess that most of these files will end up in zone 8. 

If booted daily, than yes. Otherwise there is many files not accessed for few days, or even until next boot.

[Bill]

I forgot to exclude directories from the first scratch area; I've now updated the first post with the change.

On my Windows 7 machine, the Access timestamp seems to only change when a file is created or modified or moved across partitions or devices or an application explicitly changes the timestamp. The Access timestamp doesn't seem to change when a file is only read (i.e. accessed) by an application.

[poutnik]

On my Windows 7 machine, the Access timestamp seems to only change when a file is created or modified or moved across partitions or devices or an application explicitly changes the timestamp. The Access timestamp doesn't seem to change when a file is only read (i.e. accessed) by an application.

Are you sure ? Is not Last Access deactivated by default as in Vista, in opposite to XP where it is ON by default ?
In such case LA will probably follow other timestamps.

[Darlis]

Yes, last access is deactivated by default for Vista and 7. Afaik the key is not even present anymore, you'll have to recreate it to activate it.

[poutnik]

Yes, last access is deactivated by default for Vista and 7. Afaik the key is not even present anymore, you'll have to recreate it to activate it.

e.g. by  FSUTIL behavior set disablelastaccess 0 in privileged command line console.

[Bill]

"fsutil behavior query disablelastaccess" gives "DisableLastAccess = 1" on my machine.

On my machine, the Accessed timestamp is set to the Created or Modified timestamp whenever a file is created (copied, etc.) or modified resp. If I do a move operation to another partition or between partitions then the Accessed timestamp changes to the current time the file was moved.

[poutnik]

It is expected default behavior - last access timestamp is not used and if queried - it takes newer value from the other stamps.
See LastAccessEnabled() if not yet.

[Bill]

Well, I've now updated the script above to no longer consider lastAccess.

[poutnik]

Well, I've now updated the script above to no longer consider lastAccess.

Why ?
It would be more general approach, counting with both scenarios. when MyDefrag syntax offers ability to do that.
Some users prefer LA activated, some do not, regardless  on OS defaults.

[Bill]

lastAccess was only used in zone 6 (small files modified occasionally).
I could change the FileBoolean to:
size(0,small) and (lastChange(rarely ago,) or creationDate(rarely ago,) or (lastAccessEnabled(no) and lastAccess(rarely ago,)))

Files that are created or modified rarely can remain in zone 7 without ever being touched by MyDefrag because those files won't become fragmented or added or removed from the zone (i.e. no changes are caused to the zone), so zone 7 is a stable zone. (Zone 7 files could also be in Windows BootOptimize and ProgramHints zones. Windows BootOptimize seems relatively stable, but ProgramHints is unstable but the performance gain is probably worth more than the data shuffling trade-off.)

If I included (lastAccessEnabled(yes) and lastAccess(rarely ago,)) in zone 6 then those occasionally accessed files would be mixed in with zone 6 occasionally modified files. Zone 6 is less stable than zone 7, and so can cause unnecessary shuffling of files that are rarely fragmented. And I don't see any real performance benefit in mixing those zone 7 occasionally accessed files with zone 6 files given the expense of unnecessary shuffling.

(lastAccessEnabled(no) and lastAccess(rarely ago,)) is useful on my machine where for some weird reason, installation programs can extract files but leave the old archived creation and modified timestamps for that file, but the accessed timestamp is more recent (i.e. the time it was installed). An example is some of the files installed by Microsoft Office 2010.
Thoughts?

Edit: I've updated the script to consider disabled lastAccess in a few zones. Hopefully, this will be the script's final version.

[poutnik]

Do you prefer folders to be included in zone 1 + 2 and NTFS system files in zone 1+2+3 ( if match the boolean ) ?