1.5 Gb Memory usage and 230MB logfile

[jeroen]

I was never able to complete a single defrag using any of the MyDefrag Betas.

Have you also tried FastOptimize? You only mention SlowOptimize. FastOptimize uses less memory than SlowOptimize.

Each time with less than 300,000 files, the memory ballooned to almost 2 Gigabytes used by MyDefrag.

The MyDefrag View->Statistics window will show some memory statistics. Could you please try again and tell me what you see happening in those statistics, and at which point in the script? The memory that is shown by the task manager is interesting, but I need more detail.

In particular it has always had a problem trying to move the C:\WINDOWS\CSC files which are offline files.

Could you please add the following line to your "C:\Program Files\MyDefrag v4.0b11\Scripts\Settings.MyD" script and test it? I have to admit that I have never tested MyDefrag with offline files, I do not have machines set up for that.

Code:

ExcludeFiles(Offline(yes))

This folder is also encrypted with EFS

The experiments that I did myself show no problems with encryption, but I will test some more.

[cg]

Have you also tried FastOptimize? You only mention SlowOptimize. FastOptimize uses less memory than SlowOptimize.

I think I did try it once but I'm not sure.  Honestly I'm doing these SlowOptimize tests to help you find the elusive memory issue that several people seem to run into.  It ran successfully yesterday with roughly 900 megs of RAM which was good, but there's no way that only 20,000 more files would bump the RAM up to 2 GBs.  The good thing is, I was able to pinpoint it to a single event change - which was removing offline files.

The MyDefrag View->Statistics window will show some memory statistics. Could you please try again and tell me what you see happening in those statistics, and at which point in the script? The memory that is shown by the task manager is interesting, but I need more detail.

Yeah - I think I did that earlier in this post, but I can do it again.  One thing you might consider adding is a simple Log event in your MyDefrag.debuglog that writes the Memory Statistics to the log every time the program uses up an additional 1 Meg.  The best option would be adding an additional commandline argument and script option.  Like -m 1 would mean dump the Memory Stats every 1 Megabyte increment in total memory usage.  With the date stamp, it'd make it easier to pinpoint the locations where the memory jumps so you could compare it to exactly what was happening in the debuglog at that moment.  The nice thing about that being a separate command line argument is that you don't need to use it unless you want it.  For example I might want to log the memory stats but NOT do the -d 6 level of debugging.

Could you please add the following line to your "C:\Program Files\MyDefrag v4.0b11\Scripts\Settings.MyD" script and test it? I have to admit that I have never tested MyDefrag with offline files, I do not have machines set up for that.

Code:

ExcludeFiles(Offline(yes))

Sure - I'm currently re-syncing my Offline files, so when that's done, I'll try a SlowOptimize both with and without that setting.

The experiments that I did myself show no problems with encryption, but I will test some more.

Maybe its the combination of Offline and EFS?  Not sure... I didn't see any specific encryption issues either, but thought I'd bring it up.

[jeroen]

Honestly I'm doing these SlowOptimize tests to help you find the elusive memory issue that several people seem to run into.

Thank you very much for your help, I appreciate it. The memory bug is where the program crashes when switching to the next volume. The problem you are describing is different, huge memory usage. I'm very happy that you have found that turning off the offline files solves your problem, perhaps I can do something about it.

One thing you might consider adding is a simple Log event in your MyDefrag.debuglog that writes the Memory Statistics to the log every time the program uses up an additional 1 Meg.

Thanks for the suggestion. I have put it in my hat and will think about it.

[jeroen]

I have just released MyDefrag v4.0b12 with lot's of fixes. Amongst other things it should now exit with an error message when memory is full. If you have a swapfile then the swapfile will still be used, though. Could you test it for me please and let me know how it works for you? Thanks, I appreciate it.

[Henk]

(This was at first a response to this thread, but it's more appropriate here)

When a program uses such an excessive amount of memory that it simply crashes (either by itself or through Windows), then either it's got a memory leak, or it's got a massive inefficiency in its basic functionality. A program that's as relatively simple as a defragmenting program (not saying it actually is simple, it just looks that way and its base functionality is composed of moving bits of hard drive around) really shouldn't be using 3 GB of memory.

I too thought it was a memory leak when my slow defrag hung itself on memory usage, but after some searching on these forums, guess it's not.

Jeroen said in this thread that the program 'has' to store the full path to all files on the hard drive, in order to 'sort' them. This isn't - imho - required, as a filesystem is always hierarchical - files X, Y and Z in folder A can be sorted according to the single names of X, Y and Z, as their folder A is known.

Second, it shouldn't be required to load all filenames into memory (or whatever) to sort them - instead, chop the filenames up into manageable bits, then merge them later on. Another option is to generate a key or index of the filenames, a number that equals to their natural order, and sort on that instead of on the full filename. See also 'memory efficient sorting' on Google and/or Wikipedia.

Of course, apart from that - why would anyone want to defragment their hard drive by sorting all files according to their name? What performance or efficiency benefit does that have? A proper defragmenting algorithm should place related files in close proximity to each other, and frequently accessed files at the fastest part of the hard drive, or the easiest accessible. Sorting alphabetically will place files in the same folder next to each other on the hard drive (as x/y/file would follow x/y/eile), but I'm sure there's more efficient methods. I'm no expert on defragmentation though.

I will however state that an app that uses up thousands of megabytes of memory is simply unacceptable.

On the SQLite discussion, I'd say that's a bit overkill - one would fire queries at a relational database, when all that's stored is some filenames. You could just chop the filename list up into bits and put them in plain textfiles, one line per filename. Much simpler, much faster.

And besides that, you could consider a caching library (at least, those are available for Java). That one will take over the memory management, and will store an X amount of data in memory (the most frequently accessed), and the rest it will store on the hard drive if memory consumption exceeds a certain maximum value. I'd recommend that over using SQLite, unless you guys can point out what advantages a full-on relational database system has over plain textfiles for storing file paths, or some other purpose it can be used for.

Not to mention that SQLite would use the hard drive heavily, so the disc's structure would change and fragment while it's defragmenting. Bit counter-productive, I'd say.

Anyways, to summarize: Memory usage for this version of MyDefrag and/or the slow optimize script is excessive and unnecessary, and should be fixed as soon as possible.

[jeroen]

When a program uses such an excessive amount of memory that it simply crashes (either by itself or through Windows), then either it's got a memory leak, or it's got a massive inefficiency in its basic functionality.

With all respect, but that is very stupid and shortsighted. Why do you think there are so many supercomputers in the world? There are many very simple problems in mathematics that require vast amounts of computer memory and/or CPU to be solved. Applications that need a lot of memory not always have "a memory leak, or a massive inefficiency in its basic functionality".

why would anyone want to defragment their hard drive by sorting all files according to their name?

If you don't want that then use the FastOptimize script, not the SlowOptimize script, and stop complaining.

A proper defragmenting algorithm should place related files in close proximity to each other

That's exactly what sorting by full path does.

and frequently accessed files at the fastest part of the hard drive, or the easiest accessible.

Windows does not keep "frequently accessed" statistics, the information is simply not available.

Anyways, to summarize: Memory usage for this version of MyDefrag and/or the slow optimize script is excessive and unnecessary, and should be fixed as soon as possible.

I am aware of the problem, and am looking at it and trying to find ways to use less memory, but it's not as easy as simply dropping a library into the program.

[colebantam]

Hello there,

I just run into this problem too. Using Slow Optimize, after a while MyDefrag used 5GB (its a 64Bit System with 8GB Ram) and mem-consumption was rising fast. Have aborted the defrag then, since I didn't believe it would successfully finish the job. Have read the last posts here, and I'm wondering why the sorting in MyDefrag is such a problem, when it was working perfectly in JKDefrag? Have started the same job as before in JKDefrag 3.36 (switch: -a 7), and it uses just 261MB of RAM (not increasing over time) and the speed is generally very much faster. I've used VISTA's Perfmon which displayed something like 80.000.000 B/Min (MyDefrag 4.1) compared to 990.000.000 B/Min (JKDefrag). Even while watching the XXDefrag Window, the difference in speed is easily noticeable...

Claus

[jeroen]

I'm wondering why the sorting in MyDefrag is such a problem, when it was working perfectly in JKDefrag?

Ah, a question. And a very good question. I like that a lot better than the previous poster, who flatly stated that there is a bug. MyDefrag needs less memory than JkDefrag and should always stay below that. Could you please uncomment the "Debug(175)" line in your "C:\Program Files\MyDefrag v4.1\Scripts\Settings.MyD" file and post the first and last few hundred lines or so of the "C:\Program Files\MyDefrag v4.1\MyDefrag.debuglog" file here on the forum? Please watch both the View->Statistics window and the debuglog while the memory is increasing, I need to know if the problem is caused by a single point in the program or if the memory usage increases by multiple points.

[colebantam]

Hi Jeroen,

I've sent the report you asked for via email to you. But I've received a warning:

"550 Warning: your email has been delivered, but has been marked as spam and will probably not be read"

So, please check you SPAM-Folder ;-)

Claus

[jeroen]

I've sent the report you asked for via email to you.

Thanks, I have received it. The logfile looks perfectly normal to me. I see in your email that you have stopped the program when it reached 300Mb. I'm afraid this logfile does not demonstrate the problem..... It is normal for the memory usage to slowly go up while the program is running, because it constructs the full path of every file as it is being processed. Without the LowMemoryUsage the full paths are kept in memory.

I think I will change the default for the LowMemoryUsage setting so the program will always use less memory. I am getting a bit tired about all the complaints....

[colebantam]

Good morning...

Yes, I haven't seen any suspect in the logfile too... But nevertheless, there are 2 main problems with MyDefrag or the SlowOptimize script:

1.) The program aborts jobs due to too heavy ram usage (this was reported earlier in this post)
2.) The speed is much slower compared to JKDefrag

I don't worry for the 300MB MEM, I wouldn't worry if its 3GB Mem. I DO worry when a process eats 5GB of Mem, and rises 1MB per second, and the job is at something like 1% completed.

But, I think my data partition is too big for a "Full Sort" anyway, because I left the computer sorting with JKDefrag over night, and if I interpolate the necessary time, it would take a week or so to complete :-( So, I will stop using Slow Optimize on my Data Partition. The Windows Partitions did complete with MyDefrag and SlowOptimize (haven't looked at RAM that time) in feasible time.

Anyways, great thanks for JKDefrag and MyDefrag

Claus

[jeroen]

1.) The program aborts jobs due to too heavy ram usage

Have you tried changing the LowMemoryUsage setting?

2.) The speed is much slower compared to JKDefrag

I don't know what exactly you are comparing. Comparable actions in MyDefrag are significantly faster than in JkDefrag.

[colebantam]

Hi again,

With "LowMemoryUsage=yes", I was able to successfully SlowOptimize my data partition :-) MEM Usage was below 300MB, and it took "only" five hours.

The "slow-speed issue" was most likely a measurement error (too little timespan for watching).  Sorry for that :-(

I understand, that in theory "LowMemoryUsage=no" should be faster, but I think in real world (at least for SlowOptimize) "LowMemoryUsage=yes" is the better choice for the default setting. So I agree with Jeroen, that delivering the next version with this setting will reduce headache (for Jeroen AND the users ;-).

Thanks again for that wonderful defraggers

Claus

[zaleksf]

Hello,

I'm a new user. I'm having difficulty getting MyDefrag v4.1.1 to complete a defrag optimization. I have a relatively new Dell Inspiron 1420 w/ dual-core Intel processor, 4Gb ram, 320Gb hard drive with 180Gb partitioned for Vista. The hard drive is somewhat over half-full. After reading the help, web, etc., I used the standard slow-optimization script using the LowerMemoryUsage(yes) switch, running Vista in Safe Mode. The defrag/optimization failed at the beginning of the zone 4 of 5 stage; memory usage spiked from 1.3Gb to 2.0(+)Gb and the defrag operation ended in an out-of-memory failure. I'm including two log files and an image of the app(s) shortly after the mem failure. the (zipped) debuglog file is too large to post it here in its entirety; please let me know if you would like to see it as well and I'm parse it for sending. Hope this helps.

Also, following this defrag the useable space on my drive was reduced from 85Gb free to 71Gb free. I'm attempting to run Fast Update now, with LowerMemoryUsage(yes) switch, in an effort to clean things up.

Regards,
SZ 

[jeroen]

I'm including two log files and an image of the app(s) shortly after the mem failure.

Thanks for the logfile and the screenshot, but I'm afraid they are only mildly interesting. What I need is the "c:\Program Files\MyDefrag v4.1.1\MyDefrag.debuglog", just the first and last few hundred lines or so, enough to demonstrate the problem.

Also, following this defrag the useable space on my drive was reduced from 85Gb free to 71Gb free.

For more information see Why do I have less diskspace after running MyDefrag?