1.5 Gb Memory usage and 230MB logfile

[rlarno]

Started MyDefrag SlowOptimize.MyD on my very old disk on my very old desktop machine a few days ago. It's an AMD Athlon XP 2400, with (only) 512 MB of Ram. Two disks, 1 (old) maxtor 80Gb and a (more recently bought, but otherwise also old) WD 80Gb disk. I bought the new one because the old one was starting to cause some problems. And I keep this machine alive for some development experiments and some games. So I recently rebuild the machine using the new disk, installed a clean streamlined Windows XP Sp3 on it.

The result of running MyDefrag SlowOptimize was that as I returned to the machine yesterday, the process was consuming 1.5 GB of memory and there was a Balloon tooltip stating that it could not read some sector (sorry, did not take a screenshot of the balloon). I then closed the UI, but I noticed that the process was still working in the background. I left it running for several minutes, when it eventually released all the memory.

I'm sure this is a rare case, MyDefrag still rocks, but wanted to let you know. Maybe you are interested in investigating. The old disk (D:) contains some unreadable sectors, which I'm testing running chkdsk (fix errors and scan for bad clusters) and now I'm running HD Tune, Error Scan, on it. (edit: 1.2% damaged blocks )

I already opened the LogFile using TextPad, which seems to be able to handle it. And it contains this in the summary:
These items are fragmented:
Fragments   Bytes          Clusters     Name
----------- -------------- -----------  -----
          8    308,019,200      75,196  D:\$MFT
          2 81,948,430,336          39  D:\$BadClus:$Bad:$DATA
...

obviously that $BadClus has me worried...

The last HeapBytes line is as follows
HeapBytes=1,027,147,217, HeapItems=5,176,863, Volumes=1,520, Items=24,676,736, FileNames=20,699,560, Fragments=5,607,840

It is a huge logfile, so I did not go through all of it trying to figure out why it is soo huge. Is it just logging all the files on the disk? Is it doing stuff repeatedly for some reason? And why then is MyDefrag consuming so much memory? All perhaps interesting questions you might want to look into? I will admit that this disk (D:) has seen Windows XP, all of the updates and servicepacks, lots of other installs and uninstalls of various programs and games. I even have 2 windows installs on the disk. And all I ever might have run on it is Windows Xps' build-in defragmenter, which is probably why MyDefrag has had to rearrange all the files on the disk.

But maybe that will be the case for a lot of people when they will first start to use MyDefrag?

If you are interested in investigating this, I can send you the logfile via e-mail, zipped its only 20MB, however for privacy reasons I do not wish to upload it here publicly. (Not that there is anything special on the disk, I'm just a bit paranoid  )

I'll be cleaning up the disk, removing all the windows install files, which I should have done beforehand anyway and then I'll run MyDefrag on it again.

[poutnik]

I would advice to use stable versions on problematic disks,
using testing ones on good disks only.

Otherwise HW problems and SW bugs can interfere.

[jeroen]

Thanks for testing and for the feedback, I appreciate it! I'm assuming you used v4.0b1, because v4.0b2 was released only yesterday. I know about the memory problem and am looking into it, but haven't found the problem yet. The beta is set for debugging and that is why it is generating such a big logfile. Thanks for offering to send me the logfile, but could you first try the new beta? The logfile of the old beta is of little use to me.

[rlarno]

I tried the new beta, using Fast Update, and that just ran perfectly.

I'm also cleaning up the disk, not sure if I will continue to use it in the future.

[cg]

I just had this same problem on my work machine using v4.0b2.   I ran the SlowOptimize script - walked away to a meeting.  When I got back it took forever to get logged into my screensaver.  Once I got in, the computer was barely responsive.  I checked Task Manager and MyDefrag was using 1,700,000 Kb of memory - but almost no CPU usage.

Suggestion in your debug logging - you could periodically check the app's memory usage and log it to see if you can pinpoint where the memory spike happens.  It might help you narrow the issue to a specific part of code or statement execution.

[cg]

Just as a test, I tried another script (MoveToEndOfDisk) and within 2 minutes it was using almost 700,000K and rising.

I know with .NET applications I write, I sometimes have to force .NET to clean up the memory for my application using the garbage collection.  Is something similarly possible here (to force windows to let go of memory)?   Just in the time it took me to type this up, I'm at 850,000K and still rising.  I can't think of a reason it'd be using that much memory.

[jeroen]

I can't think of a reason it'd be using that much memory.

The program needs a lot of memory for the filenames. See the LowMemoryUsage setting to reduce the memory usage somewhat. Apart from that there is a memory leak somewhere that I am trying to track down.

[cg]

I'll try the LowMemoryUsage flag to see what that does to the RAM usage.  I'll post a reply with that later.

As far as space, I can understand the need if its storing all the files in memory, but I think you're right that there has to be some memory leak somewhere.  Just some quick calculations:

1.7 Gigs = 1740.8 Megs = 1782579.2 KB = 1825361100.8 Bytes

So if I had 300,000 files ( I really have around 260,000 - but 300,000 is easier math), at 1.7 Gigs of memory used it'd be:

1825361100.8 bytes / 300,000 files = 6084 bytes per file.   I'm sure you're storing more than the path (size, date stamps, etc), but for most files that should be under 300 bytes per file.   Even if you used 600 bytes per file you'd drop the memory requirements by a factor of 10 to around 170 megs.

I suppose finding the memory leak will probably resolve the issue but maybe to reduce RAM, could the analysis process store the "ToDo" list in a file that is read during the move/defragmentation process?   That way you do take a hit up front on RAM while it analyzes the files based on the script, but once it begins the work, it'd be reading what it needs to do from a file rather than storing everything in memory.  My guess is that might be more work than you want to do - and if you did something like that, then why not use a standalone DB like SQL Lite for storing everything.   

Just throwing out ideas.... Love MyDefrag's power - but the memory hit right now makes it unusable on my laptop.  Hope you can find the memory leak soon!

[jeroen]

maybe to reduce RAM, could the analysis process store the "ToDo" list in a file that is read during the move/defragmentation process?

No, the program needs the full list of all files in memory at all times, for example to find the files that must be vacated.

why not use a standalone DB like SQL Lite for storing everything.

I've considered it, and maybe I will try it as an experiment in the future, but I'm pretty sure it would make the program a lot slower.

[cg]

No, the program needs the full list of all files in memory at all times, for example to find the files that must be vacated.

Oh so your saying as files are moved around you need to know where they were moved to because that could affect a later operation?   Makes sense - I see why just a read only list wouldn't work - you need to update on every move operation.

I've considered it, and maybe I will try it as an experiment in the future, but I'm pretty sure it would make the program a lot slower.

You might be surprised.  I've used SQLite before and its surprisingly quick.  The fact that you'd be reducing your memory footprint to almost nothing would make a huge difference - at least on any machine that has a decent set of files.  I was watching it earlier and the page file was being slammed when MyDefrag took up that much RAM- so it was writing to the page file and reading from it anyway.  If you have no or little RAM usage, a database will be far faster and efficient than a page file I'd think.  Especially if you created tables to reduce the total size.  For example a folders table with an index.  A files table wouldn't contain any path information - just linked number from the folders table. etc.

Wish there was a way we could help you with that type of test!

[jeroen]

The fact that you'd be reducing your memory footprint to almost nothing would make a huge difference

Yes, reducing the memory footprint is a big incentive to look into a disk-based database. I've never used SQLlite (I always use MySQL) but I'm sure you are right and that it's very fast. However, that's not what I was referring to. At the moment MyDefrag only has to walk a B-tree in memory to find something. There is no way that a SQL database can be faster than that.

[cg]

You're absolutely right in that using a file based database will never come close to the performance of in memory operations.  My point was more that as of right now because of the high memory usage,  its reading and writing to the page file which eliminates RAM speed gains since its actually using the disk as RAM.

All that said - when you pinpoint the memory leak and close it - this will all be a moot point.  Because once the memory footprint is small - why even consider disk based system as you pointed out.

I like MySQL a lot as well, but the SQLite is nice in that you don't need any installed or configured database system.  That makes it nice for operations like you might use here where you'd use it for simple selects and inserts.

Anyway - I'm loving how MyDefrag is coming!   Once the memory leak and the freezing problem I reported (http://www.kessels.com/forum/index.php?topic=1651.0) are resolved, MyDefrag will be the best defrag program around hands down.  The ability to use a scripting language to customize operations makes this so flexible!  Great job!

[rlarno]

Jeroen, I'm sure once you have found the memoryleak, MyDefrag will perform adequate again.

I doubt any kind of database would be a good option for MyDefrag, as any good database engine will attempt to cache as much of its datapages in to the main memory. Where MyDefrag and the DB engine will then start to contend for that memory. Possibly (most likely) causing even more memory-strain, leading to more paging, which leads to performance degradation.
On top of that, it will cause MyDefrag to carry a dependency to an external library, where MyDefrag at the moment is nicely contained within a single .exe

I'm pretty confident that your structures used to store the file-data are nicely tuned for the job, and nothing will beat a B-tree for performance (as you probably have done your research to determine the B-tree is the optimal algorithm).

Perhaps there is a possibility to reduce the amount of memory needed for files that are already processed? e.g if a file has been moved into a 'zone', and all processing for that zone is finished, only a link to the zone could be enough, with the zone having the starting LCN and its cluster count (length), just to know that that zone is 'off-limits' for placing files you are currently processing.- I'm just wildly speculating here - no clue how MyDefrag performs its magic.

I know with .NET applications I write, I sometimes have to force .NET to clean up the memory for my application using the garbage collection.

btw: I hope you really know why you need to do GC.Collect, incorrect garbage collection can have very bad effects on the scalability and performance of your code. In real-time apps, there are cases where you need to have more control. General Windows apps rarely need it, server apps (e.g. ASP.Net) should not call GC.Collect.

[jeroen]

Perhaps there is a possibility to reduce the amount of memory needed for files that are already processed?

Well, at first glance I don't think it will help much, because the maximum memory usage will remain the same, just before processing the first zone.

[cg]

btw: I hope you really know why you need to do GC.Collect, incorrect garbage collection can have very bad effects on the scalability and performance of your code. In real-time apps, there are cases where you need to have more control. General Windows apps rarely need it, server apps (e.g. ASP.Net) should not call GC.Collect.

Yeah I'd never use it for a general windows application.  As an example of one that I did use that call, I built an application that reads any data file into a SQL Server.  Because some of the files are over 100 million records, I made the application aware of the environment around it.  So as its running, it determines how much of the computer's resources its using and changes based on that information.  So for example this program would be able to run on a machine that had 512 Megs of RAM and it'd run on a machine with 12 Gigs of RAM - but when on the 12 Gigs of RAM machine, it would change its behavior to use more RAM to improve performance.   An easy way to think about it is the number of records you send to the server on each pass.  A machine with 12 Gigs can load many more records into memory before flushing them to SQL Server.  However if you do that and you don't clear the memory before you grab the next large set of records, you'll end up using more RAM than you need.  So in that example I called the GC.Collect to force a reduction prior to starting the next large load.   

So again - for normal usage I'd never do that.