MyDefrag could use another physical drive as virtual memory for sorting...

[BloodySword]

... to avoid running out of memory. The user could configure it in the corresponding script or settings.myd. MyDefrag could then use an SSD or SDXC card for example. It should not do it automatically, only if requested by user in the script or settings.myd. A file named MyDefrag.vmem could be created where some memory is stored for sorting or importing files from very large drives with huge number of files. After the script has finished, the file should be deleted by MyDefrag.

This approach is good especially for 32bit system wich does not have more than 3 GB memory installed. The OS needs around 1GB ram and the rest is not enough for drives with huge number of files.

This technology could be called Virtual Memory Extension -- VMX. If I would have money I would patent this internationally. ;o)

Example:

EnableVMX(yes) #Will enable this technology. Can be called from every script or in Settings.MyD
VMXDrive("I:\") #Tells MyDefrag to use this drive.

Another example:

VMXDrive("I:\") #Tells MyDefrag to use VMX on drive I:\.
VMXDrive(Disabled) #Tells MyDefrag to not use VMX at all.

A benchmark utility could test MyDefrag's access behavior to tell if the performance of this drive is good enough. Technically any good SSD or SDXC card are potentially candidates. Note that huge use of VMX on such drive will decrease the lifespan rapidly!!

The card reader for SDXC cards should be connected at least at PCI-Express or SATA or even at USB 3.0.

VMXDrive should check if this drive is another physical drive than the drive wich will be defragmented as soon as the script is interpreted. If not, MyDefrag should throw an error or continue without VMX.

Functionality:

Parts of the memory in sorting process wich is actually not used could be copied into this VMX-File to free memory for other files. Also the complete file list could be copied into this file, using the fast available memory for sorting techniques.

Using VMX right now, without waiting for a new MyDefrag version wich eventually supports VMX:

- Buy an SSD or an SDXC cardreader wich is connected at least with USB 3.0, SATA or PCIe.
- Insert a fast and good SDXC card and mount this thing into windows.
- Configure Windows to use this drive for the page file.
- Reboot
- Use MyDefrag :o)

Note: VMX won't be as fast as MyDefrag on 64-Bit Windows with enough RAM installed!!!

What do you think of this idea?

[Kasuha]

I think you're reinventing the wheel here a bit. Memory mapped files and external databases are known for some time already. More options include keeping these files on a ramdisk or creating a 'tree' of independent processes communicating with each other via IPC to keep the data.
But overall it's kinda hard to implement and, in case the data are held on disk, would also mean significant slowdown of any MyDefrag file sorting such as sorting zone files by filename or finding files to fit a gap as even the fastest drive is still three orders of magnitude slower than RAM.

[BloodySword]

Yes, but the drive should only hold old memory data that is used again later. This data would be read and written sequentially. But it would help on older systems with drives with many files such as drives with program source code and header files.

[jeroen]

Thanks for sharing your idea, I appreciate it. Temporary data (for example while sorting) does not take much memory, the vast bulk of the memory used by MyDefrag is pure file information. Parts of that information could be moved to diskmapped memory, and it is already on my wishlist to do something like that in the future. I also have some other ideas on the wishlist. Perhaps also moving temporary data to disk, but I am sure it will make MyDefrag extremely slow and I think it will reduce memory usage only by a little.

[BloodySword]

You should consider the option to use fast flash memory disks such as SSD or SDXC cards. With the settings command I advocated you could reduce MyDefrag slowness. When you move file information to a file on a flash disk drive, I think it won't slow down MyDefrag dramaticaly, only quite a bit. But since MyDefrag will walk through the script without breaking due to full RAM with out of memory error it is a BIG improvement. It will act like a soft slowdown of 60 or 80 on the disk wich is optimized. I intended this featuere as an option. MyDefrag should never use this technique when it is not activated with VMXDrive().

A big slowdown will only occur if MyDefrag moves big parts to pagefile.sys. If the drive is optimized where the pagefile resides in any partition, it will slowdown MyDefrag DRAMATICALLY. Defragmenting a disk with 4 GB data and 400 000 files will then take 2 days I think . This should be avoided. MyDefrag should only use a drive the user specified in Settings.MyD or in the corresponding script.