Beta 4.2.0 uses all my memory then crashes

[AndyB]

Hello Jeroen,

first thing: mydefrag is really awesome!!
Thanks a lot ...

About the memory leak:
the beta definetly looses virtual memory along the way.

If you trace the process with process explorer, you can see how the virtual memory usage increases over time.
In the debug log, there is a line after each zone, stating memory cleanup. I don't see any memory freed in process explorer.

I attached my script, the debug log, a screenshot of taskmgr at time of pause, an earlier screenshot of taskmgr of the same drive of another try.

Hopefully this provides some help in finding the leak ;-)

If you need additional infos / tests let me know.

Cheers,
AndyB

[dhry]

MyDefrag really needs the memory.

Keeping it real: Perfectdisk 10's engine is defragmenting one of my drives, with 411,000 files using ~32Gb space right now, and Process Explorer shows it using 17.4Mb (peak 56Mb) working memory, 73.5Mb virtual memory locked, and barely any CPU power at all. I realise your software is beta, but JK v3 used nowhere near the memory that v4.2.0b is allocating. This may not be a memory leak in the classic sense, but I'd bet money on the fact that there is substantial memory allocation going on without proper deallocation after certain events are completed. Not saying that the memory is gone permanently, but that the program is taking bites and failing to chew or swallow while focused on an individual drive. 1Gb RAM or more required to perform a basic defragmentation pass on what people are claiming are drives with fairly standard amounts and types of data on them is unrealistic in the extreme.

DRS

[jeroen]

the beta definetly looses virtual memory along the way.

Thanks for posting the logfiles and the snapshots, I appreciate it. I am about to release a new version and I perhaps it will fix your problem. Your script looks ok to me, all I can say is that SortByName can use a lot of memory as compared to FastFill, for example.

I've often wondered about Process Explorer and what the numbers exactly mean that it is showing. It is most definitely NOT the amount of memory that is in use by MyDefrag, which I have verified many times with internal memory counters, debuggers, debug memory libraries, and other techniques. Process explorer can show several counters, and I find them all very confusing.

[bluefish]

Hi,
I've used JkDefrag in the past and thought I'd give MyDefrag a chance.
The issue described here happens to me on the latest version (4.2.1) while running the Optimize Monthly script (haven't tried any of the other scripts).
I've attached the debuglog and a screenshot to this post.

I'm running Windows XP SP3 on a Athlon 64, I have 2GB Ram and have the pagefile set at 2GB as well.

[jeroen]

Thanks for posting the logfile, I appreciate it. I'm sorry to see that the problem still exists in the new version. A quick look at your logfile has not turned up anything obvious, except perhaps that your disk is very full. It's a big pity that I cannot repeat this problem for myself, it makes it very difficult to fix this problem.

[kenpoole]

4.2.1, optimize monthly - ran out of memory - was watching it thru run - seemed to only use about 1/2GB when thru with zone 4, in zone 5 started climbing, was in a steady climb at end - while defraging c:\System Volume Information\_restore{.....}\RP165\snapshot\Repository....

[agnag]

NTFS compressed files have huge amounts of extends, but MyDefrag should be able to handle them without running out of memory. I have to repeat: At the moment I do not yet have enough information, I don't know what is happening inside MyDefrag. Can somebody please make a Debug(431) logfile?

My highly fragmented Virtual machine HDD is NTFS compressed since it saves 6GB (10GB Data -> 4 GB on HDD)
Fragments: ~ 45.000 (manually aborted with 29k Fragments)
Ram Usage: 750MB

I still have the problem with V. 4.2.1
Part of Logfile:

20:42:19             I want to move 1 clusters at offset=668004 of item 'G:\.VirtualBox\HardDisks\WinXP.vdi' (Inode=1050).
20:42:19               Moving 'G:\.VirtualBox\HardDisks\WinXP.vdi' (Inode=1050)
20:42:19                 Moving 1 clusters at offset=668004 to Lcn=59780.
20:42:19                 API is moving 1 clusters at offset=828752 to Lcn=59780.
20:42:19                 Getting fragment list for: G:\.VirtualBox\HardDisks\WinXP.vdi
20:42:19                   Extent: Lcn=30676, Vcn=0, NextVcn=9
20:42:19                   Extent (virtual): Vcn=9, NextVcn=16
20:42:19                   Extent: Lcn=30685, Vcn=16, NextVcn=19
20:42:19                   Extent (virtual): Vcn=19, NextVcn=32
20:42:19                   Extent: Lcn=30688, Vcn=32, NextVcn=35

<~ 80.000 similar lines>

20:42:34                   Extent (virtual): Vcn=1799823, NextVcn=1799824
20:42:34                   Extent: Lcn=3780032, Vcn=1799824, NextVcn=1799839
20:42:34                   Extent (virtual): Vcn=1799839, NextVcn=1799840
20:42:34                   Extent: Lcn=3782759, Vcn=1799840, NextVcn=1799914
20:42:34                   Extent: Lcn=3855416, Vcn=1799914, NextVcn=1799920
20:42:34                   Extent: Lcn=3785414, Vcn=1799920, NextVcn=1799984
20:42:34                   Extent: Lcn=4521775, Vcn=1799984, NextVcn=1800000
20:42:34                   Extent: Lcn=4521728, Vcn=1800000, NextVcn=1800047
20:42:34                   Extent (virtual): Vcn=1800047, NextVcn=1800048
20:42:34                   Extent: Lcn=562572, Vcn=1800048, NextVcn=1800060
20:42:34                   Extent (virtual): Vcn=1800060, NextVcn=1800064
20:42:34                   Extent: Lcn=3213084, Vcn=1800064, NextVcn=1800075
20:42:34                   Extent (virtual): Vcn=1800075, NextVcn=1800080
20:42:34                   Extent: Lcn=3643008, Vcn=1800080, NextVcn=1800092
20:42:34                   Extent (virtual): Vcn=1800092, NextVcn=1800096
20:42:34                   Extent: Lcn=3757126, Vcn=1800096, NextVcn=1800105
20:42:34                   Extent (virtual): Vcn=1800105, NextVcn=1800112
20:42:34                   Extent: Lcn=3810809, Vcn=1800112, NextVcn=1800122
20:42:34                   Extent (virtual): Vcn=1800122, NextVcn=1800128
20:42:34                   Extent: Lcn=1099696, Vcn=1800128, NextVcn=1800132
20:42:34                   Extent (virtual): Vcn=1800132, NextVcn=2621456
20:42:34                 Testing fragmentation from 668004 to 668005 (1 clusters).
20:42:34                   No, not fragmented
20:42:34             I want to move 12 clusters at offset=893869 of item 'G:\.VirtualBox\HardDisks\WinXP.vdi' (Inode=1050).
20:42:34               Moving 'G:\.VirtualBox\HardDisks\WinXP.vdi' (Inode=1050)
20:42:34                 Moving 12 clusters at offset=893869 to Lcn=59781.
20:42:34                 API is moving 12 clusters at offset=1093392 to Lcn=59781.
20:42:34                 Getting fragment list for: G:\.VirtualBox\HardDisks\WinXP.vdi
20:42:34                   Extent: Lcn=30676, Vcn=0, NextVcn=9
20:42:34                   Extent (virtual): Vcn=9, NextVcn=16
20:42:34                   Extent: Lcn=30685, Vcn=16, NextVcn=19
20:42:34                   Extent (virtual): Vcn=19, NextVcn=32
20:42:34                   Extent: Lcn=30688, Vcn=32, NextVcn=35
20:42:34                   Extent (virtual): Vcn=35, NextVcn=48
20:42:34                   Extent: Lcn=30691, Vcn=48, NextVcn=49
20:42:34                   Extent (virtual): Vcn=49, NextVcn=64

<~ 70k similar lines; manually aborted>

20:42:49                   Extent (virtual): Vcn=1800132, NextVcn=2621456
20:42:49                 Testing fragmentation from 893869 to 893881 (12 clusters).
20:42:49                   No, not fragmented
20:42:49           Preemptive vacate of items not belonging to this zone, from LCN=0 to LCN=2441.
20:42:49             Vacating movable items between LCN=0 and LCN=2441.
20:42:49               Finished vacating.
20:42:49           Looking for the first gap after Lcn=3.
20:42:49             Gap found: Lcn=3, Size=47
20:42:49           Moving 'G:\$LogFile' (Inode=2)
20:42:49       Finished PlaceNtfsSystemFiles.
20:42:49     ZoneBegin is now 17359.
20:42:49     Updating list of unmovable files at: E:\Programme\MyDefrag\MyDefrag.dat
20:42:49       Loading file into memory: E:\Programme\MyDefrag\MyDefrag.dat
20:42:49       Finished updating list of unmovable items.
20:42:49     Merging wrap-around fragments.
20:42:49       Found 0 items with wrap-around fragments.
20:42:49       Finished Scanning for items with wrap-around fragments.
20:42:49     Finished FileActions.
20:42:49   Finished VolumeActions.
20:42:49 Finished VolumeSelect.
20:42:49 Finished with disk G:
20:42:49 Finished executing the script.
20:42:49 Resetting ScreenSaverTimeout to 600
20:42:49 Stopped.

at the end the RAM load was ~ 660 MB

[dhry]

Jeroen, I myself did not see the problem in v4.2.1. I just ran it on the same drive that 4.2.0 crashed out on and this time it ran fast with the same profile and finished with a final working memory usage of 996k (less than a meg - very nice!) and showing a peak working memory usage of 138Mb. It left a giant chunk of space between a couple of zones, not sure why but it might just be a config file tweak.

From the last couple of comments it sounds like the problem hasn't been ironed out completely, but whatever you changed was definitely on the right track. Thank you!

DRS

[jeroen]

4.2.1, optimize monthly - ran out of memory - was watching it thru run - seemed to only use about 1/2GB when thru with zone 4, in zone 5 started climbing, was in a steady climb at end - while defraging c:\System Volume Information\_restore{.....}\RP165\snapshot\Repository....

Please make & post a debug logfile.

[jeroen]

I still have the problem with V. 4.2.1

Can you please be more specific? What problem? The program in your case did not crash. Climbing memory by itself is normal and is not a "problem". The amount of memory MyDefrag needs depends on things like the number of files, the number of fragments, the length of filenames, and more.

[agnag]

it didn't crash, but I aborted it because the memory increased while defragmenting this file and the crash is just a matter of time.

Now I've waited till MyDefrag crashed because the system had no more free Ram.


Logfile: just like the previously posted one, but it stopped with during the "Extend" lines.
Runned Script: Optimize monthly

After defragmenting the large file with Defraggler (now 1 Fragment) I've tried it again with MyDefrag with the same result (high memory usage leading to a crash)

After decompressing the file (NTFS Compression) MyDefrag needed 'only' 550MB RAM to defragment the files.

[jeroen]

After decompressing the file (NTFS Compression) MyDefrag needed 'only' 550MB RAM to defragment the files.

Are you saying that you had a 10 gigabyte NTFS compressed file that was causing the crashes, and that the problem disappeared when you turned off the NTFS compression?

[agnag]

yes, this seemed to be the problem

10GB compressed down to 3,9 GB

[dhry]

Houston, we still have a problem. What I woke up to this morning:


Below shows ProcessExplorer's final stats for app. 1Mb working ram, 935Mb peak working set size, 2Gb virtual memory locked. And next to the CPU usage graph, 5 hours 19 mins of runtime on a 45Gb drive with 14Gb free.

[jeroen]

yes, this seemed to be the problem
10GB compressed down to 3,9 GB

NTFS compressed files take a lot of memory. Perhaps in the future I can find a way to reduce the memory requirement, but at the moment this is just how MyDefrag works. It is not a bug.