File in very deeply nested folder results in "Memory full" error

[RichC]

The screen shot says that MyDefrag uses 2GB virtual memory. If you're using XP Pro x86 you can give MyDefrag up to 3GB by performing the 4-Gigabyte Tuning.

Yep, but I doubt it would help.

The amount of memory consumed while processing that one file/folder is over 535 MB.

[Kasuha]

I think it's pointless to dispute whether it's a bug or not on a program with last release over two years ago (and known bugs promised to "be fixed in next release" almost all that time). The question is whether Jeroen intends to investigate why is MyDefrag so memory-inefficient in such situation or if all you can do about it is to find a workaround (such as exclude the folder or remove it from the disk) or change defragger to some which does not have such problem.

[BloodySword]

Or add more RAM and switch to 64 bit Windows.

[Kasuha]

Or add more RAM and switch to 64 bit Windows.

Or buy more RAM, buy 64-bit Windows, and watch it run out of memory in a few times longer time. Of course, never underestimate an expert.

My wild guess on source of the problem (without knowing or having confirmed any details, so likely wrong or at least oversimplified):

Step 1: MyDefrag sends the MS API command to move a file fragment.

Step 2: MS API fails to move the fragment because it did not succeed updating directory record due to its too long pathname.

Step 3: MyDefrag detects it as if the destination clusters are occupied by previously unknown file so it finds next suitable spot and tries again from Step 1 - ad infinitum. Probably some memory structures are created along with it, permanent or temporary, which gradually fill up the memory.

If it's the reason then of course it may depend on what file system and OS is used, MS defragmentation API is known to have different properties on different systems. Or the reason may be completely different, we would never know without investigating it properly and the only person who can do it is Jeroen who stated his position clearly:

The problem is caused by not enough memory in your computer, not by a bug in MyDefrag. And MyDefrag has no problem with paths longer than MAX_PATH.

[Darlis]

Or add more RAM and switch to 64 bit Windows.

More like: buy a 200GB SSD and put your page file on it (if that's even enough).

I've made a test with 35 nested "TestDirectory"s and a test file and watched MyDefrag eat 12GB (virtual) RAM 
I've stopped it at that point, don't know how much more RAM it would have needed for that operation to finish.

[BloodySword]

What? If it's so, then how does MyDefrag handle the WinSxS-Directory on Windows 7 and Vista? Here we also have huge nesting.

[Darlis]

I couldn't find more than 3 nested directories (Win7). 

Under C:\Windows no directory is nested more than 12 levels (the deepest being and SysWOW64 and System32). That's not much.

[jeroen]

My wild guess on source of the problem
[.....]
Step 3: MyDefrag detects it as if the destination clusters are occupied by previously unknown file so it finds next suitable spot and tries again from Step 1 - ad infinitum.

If the Microsoft defragmentation API returns with an error then MyDefrag will immediately mark the file as unmovable and not try to move it again. Please note that marking a file as unmovable is done in memory inside MyDefrag itself, not in Windows.

[jeroen]

I've made a test with 35 nested "TestDirectory"s and a test file

How did you do that? I'd llike to repeat that test but can't get deeper than 17 levels of "TestDirectory". Then Windows 7 refuses to create deeper folders and complains "The file name(s) would be too long for the destination folder. You can shorten the file name and try again, or try a location that has a shorter path."

[BloodySword]

I couldn't find more than 3 nested directories (Win7). 

Under C:\Windows no directory is nested more than 12 levels (the deepest being and SysWOW64 and System32). That's not much.

12 levels is about 33% of the 35 levels. I think the memory consumption should be somewhat 25% of that what memory was uses with 35 levels. But it isn't.

[Darlis]

How did you do that? I'd llike to repeat that test but can't get deeper than 17 levels of "TestDirectory".

I've used some Java to do it:

Code:

final String name = "TestDirectory";
String path = "D:";

for (int i = 0; i < 35; i++) {
path += "\\" + name;
}
new File(path).mkdirs();

RichC has a C(++) version of that.

I've found out too that I couldn't just delete what what I've done via Windows Explorer. I couldn't get down to the test file either. So I had to use Java again to get rid of that construct.

12 levels is about 33% of the 35 levels. I think the memory consumption should be somewhat 25% of that what memory was uses with 35 levels. But it isn't.

Maybe the required amount of memory rises exponentially not linearly?

[CSGalloway]

Would the orginal poster be better off running another MyDefrag script or even Windows Defrag b3efore running the one he orginally posted about? 

[RichC]

Would the orginal poster be better off running another MyDefrag script or even Windows Defrag b3efore running the one he orginally posted about? 

The analysis report didn't work either. 

Defraggler doesn't have any problems with the folder.   

[boco]

The screen shot says that MyDefrag uses 2GB virtual memory. If you're using XP Pro x86 you can give MyDefrag up to 3GB by performing the 4-Gigabyte Tuning.

To use the increased virtual memore, the application must be compiled with a special compiler flag, as the article states. Do you know if MyDefrag uses this flag?

[Darlis]

Yes, MyDefrag is compiled with that option: http://www.mydefrag.com/forum/index.php?topic=3108.msg18697#msg18697