Limit File Enumeration to Path (instead of volume)

[vagabund]

I have a volume with several million files, where most of the files are hard links and, therefore, only consist of a directory entry.

My goal is to defrag all directories under a certain path. MyDefrag's superb scripting support can certainly handle this task and that's why I am quite convinced that MyDefrag is the ideal tool for it.
But: MyDefrag runs into the "Oeps! Memory full" exception even before the first million files got enumerated. MyDefrag uses almost 2 GB RAM in this case (of the machine's physical 4 GB) .

An enhancement to MyDefrag could help here:
Limit MyDefrags analyzation phase to a certain sub-path of a volume. So, instead of a top-level "VolumeSelect" a "PathSelect" could reduce the number of files seen in the enumeration phase.

This approach, of course, renders many optimizations like sorting rather useless but would allow to defrag volumes with many files.

What do you think: would my proposal be a valuable new feature? Or is there a workaround with the existing functionality?

[Darlis]

Since this "feature" would make almost any other feature of MyDefrag useless, it is not worth implementing imho (if that's even possible). There are a lot of other defraggers if you just want plain defragmentation.

Do you have a 32 bit OS? Then add the /3GB /USERVA=2990 switches to the boot.ini. This will allow MyDefrag to use up to 3GB instead of 2: http://support.microsoft.com/kb/316739/EN-US/

If you have 64 bit OS make sure you are running the 64 bit version of MyDefrag: start the MyDefrag script chooser and look at the process name in the Taskmanager. If there is "*32" appended to "MyDefrag.exe", you're running the 32 bit version.

[vagabund]

Thanks for your reply.

Just a short notice concerning the 32/64 bit issue:
I am running the 32 Bit version on a 64 Bit OS (because it's launched form the network where I still have many 32 Bit machines). I gave the 64 bit version a try, and as expected its memory consumption is almost identical (about 2 kiBytes per file). The 64 Bit version probably won't tell me "oeps!" but still require more than the machine's memory.

I never had a look at Windows' defrag API. But I'm quite confident, that the memory (and time) consuming part of MyDefrag's analysation phase is to get the info which file uses a cluster-- not which cluster is actually containing data or empty.
So I'm asking for a way to limit the scope of this enumeration process. So that not every file on the volume must be pressed through this expensive phase. I still hope my favorite defragger will improve it's handling of such huge file counts

[Darlis]

I found Jeroens post about memory consumption:

The amount of memory that MyDefrag uses depends on things like the number of files and directories on the disk, the number of fragments, the length of directory and filenames, whether or not NTFS compression is used on the disk, which MyDefrag script you are using, and more. A rule of thumb is that MyDefrag uses about 500 bytes of memory per file on disk.

You seem to have "special" files if you need 2KB per file. Apart from running the 64 bit version, there is nothing you can do right now. You have to wait for Jeroen, what he thinks about a "light" version.

[jeroen]

But I'm quite confident, that the memory (and time) consuming part of MyDefrag's analysation phase is to get the info which file uses a cluster-- not which cluster is actually containing data or empty.

Thanks for sharing your idea, I appreciate it. You are correct, the analysis gathers information about which file is occupying which cluster. This is essential information, without it MyDefrag would not be able to perform many of it's functions, especially disk optimization. Reducing MyDefrag's memory usage has been discussed many times before, and many ideas have been offered. Limiting the analysis to only certain folders is one of the possibilities, but there are better solutions which would not hamper the capabilities.

[poutnik]

Do you have a 32 bit OS? Then add the /3GB /USERVA=2990 switches to the boot.ini. This will allow MyDefrag to use up to 3GB instead of 2: http://support.microsoft.com/kb/316739/EN-US/

Correct me if I am wrong, but this is limitation of the whole OS.
Is not there another limitation 2 GB for process ?
( or more exactly 2 GB for user mode and 2 GB for kernel mode )

[Darlis]

Correct me if I am wrong, but this is limitation of the whole OS.

It limits the amount of memory available for the kernel to about 1GB, yes. But how often does it need more than that?

[poutnik]

It limits the amount of memory available for the kernel to about 1GB, yes. But how often does it need more than that?

Hardly ever.