File in very deeply nested folder results in "Memory full" error

[RichC]

MyD tries to move a very deeply nested file for ~15 minutes and then eventually displays a "memory full" error. 

Does MyD support excluding specific directories from being processed?

Head and tail of debuglog.....

Version: MyDefrag v4.3.1
Date: 2012/06/07 09:37:32
Windows version: v5.1 build 2600 S
Commandline: "C:\Program Files\MyDefrag v4.3.1\MyDefrag.exe"
Working Directory: C:\Program Files\MyDefrag v4.3.1
Userid: Rich
InstallPath = C:\Program Files\MyDefrag v4.3.1\
09:37:32 Debug = 175
09:37:32 BatteryPower = ask
09:37:32 DiskmapFlip = ON
09:37:32 FileMoveChunkSize = 1073741824
09:37:32 OtherInstances = Ask
09:37:32 Ignoring SetScreenSaver(OFF) because screensaver is not activated.
09:37:32 Ignoring SetScreenPowerSaver(OFF) because screen power saver is not activated.
09:37:32 Process priority is now NORMAL.
09:37:32 Slowdown = 100.000000

...


09:43:15         I want to move 31 clusters at LCN=833862 of item 'C:\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestFile.txt:Stream1:$DATA' (Inode=76340).
09:43:15           Moving 'C:\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestDirectory\TestFile.txt:Stream1:$DATA' (Inode=76340)
Memory full in function 'CreateItemPaths' in source '.\Items.cpp' on line 147.10:03:42             Stopping....

[jonib]

Does MyD support excluding specific directories from being processed?

ExcludeFiles

jonib

[RichC]

Does MyD support excluding specific directories from being processed?

ExcludeFiles

jonib

Exclusion doesn't work on that folder either...MyD is hung performing "Looking for C:\TestDirectory\*".     

...and then eventually reports "Memory full..." 

[Darlis]

Try adding

Code:

ExcludeFiles(DirectoryPath("C:\TestDirectory"))

to the Settings.MyD file.

[RichC]

Try adding

Code:

ExcludeFiles(DirectoryPath("C:\TestDirectory"))

to the Settings.MyD file.

Nope.  Same result.   

[jeroen]

The memory full is caused by MyDefrag having to expand the full path of each and every level of your file. This can take a lot of memory. The "DirectoryPath" boolean is one of the commands that does the expansion. Instead you could try:

Code:

ExcludeFiles(FileName("TestDirectory"))

because "FileName" does not need to expand the paths. But you may run into the same memory full problem elsewhere, there are lot's of circumstances that require MyDefrag to expand the paths.

[RichC]

Ugh.  Its a bug.  Albeit an edge case, buts its still a bug.   

[Kasuha]

That file does not seem to be nested deep enough to deserve a memory overflow error. Is that directory structure linear or is it a tree-like structure? Is there just that one file or are there more of them?

[RichC]

That file does not seem to be nested deep enough to deserve a memory overflow error. Is that directory structure linear or is it a tree-like structure? Is there just that one file or are there more of them?

Linear directory structure.  One file with one alternate data stream.

The total file name length is longer than MAX_PATH.  

[RichC]

Back in the day, I dabbled a bit with NTFS alternate data streams.  I think that directory tree was created with this code.

Code:

inline bool CreateTestDirectory(LPCWSTR name)
{
    return !!CreateDirectoryW(name, 0);
}


bool CreateNestedTestDirectory(LPCWSTR root, LPCWSTR name, DWORD depth, LPWSTR completePath, DWORD completePathSize)
{
    bool success = true;
    lstrcpyW(completePath, L"\\\\?\\");
    lstrcatW(completePath, root);
    for (DWORD d = 0; success && d < depth; d++)
    {
        lstrcatW(completePath, L"\\");
        lstrcatW(completePath, name);
        _ASSERT(lstrlenW(completePath) < completePathSize);
        success = CreateTestDirectory(completePath);
        if (!success) {
            DWORD e = GetLastError();
            if (e == ERROR_ALREADY_EXISTS) {
                success = true;
                continue;
            }
        }

    }
    return success;
}

int main(int argc, char* argv[])
{
    const DWORD CompletePathSize = ((sizeof(L"TestDirectory")/sizeof(WCHAR))*25)+MAX_PATH;
    WCHAR completePath[CompletePathSize] = {0};
    bool b = CreateNestedTestDirectory(L"c:", L"TestDirectory", 25, completePath, CompletePathSize);
}

[jeroen]

Ugh.  Its a bug.  Albeit an edge case, buts its still a bug.   

I respectfully disagree. The problem is caused by not enough memory in your computer, not by a bug in MyDefrag. And MyDefrag has no problem with paths longer than MAX_PATH.

[RichC]

Ugh.  Its a bug.  Albeit an edge case, buts its still a bug.   

I respectfully disagree. The problem is caused by not enough memory in your computer, not by a bug in MyDefrag. And MyDefrag has no problem with paths longer than MAX_PATH.

How much memory do I need?  The process was chugging along until it hit the problematic folder, at which point I took a snapshot of the memory consumption.



And then an hour or so later after it ran out of memory:

[jeroen]

How much memory do I need?

I am sorry, but I do not have an absolute answer for you. MyDefrag memory usage depends on several factors, such as which script you are running and the number of files and folders on your harddisk.

[RichC]

   So 535 MB (and counting) for that folder is expected behavior? 

[Darlis]

The screen shot says that MyDefrag uses 2GB virtual memory. If you're using XP Pro x86 you can give MyDefrag up to 3GB by performing the 4-Gigabyte Tuning.