When the XP defragger starts to run, it reports "Compacting files...".  What it's actually doing is putting all the files needed to boot the machine and the files needed to start your recently used applications together in order.  It puts them wherever it finds a space large enough to hold them.  It is not compressing the files.

Certainly the MFT is positioned when the volume is formatted.

But since Windows 2000, isn't the MFT reserved space determined by the NTFS disk driver (ntfs.sys) when the volume is mounted?  Hence you have that registry key that can tell it to use up to 50% of the drive for the MFT reserved space (but unfortunately, no less than 12%).

Normally with volumes formatted as NTFS, the reserved space doesn't move much.  But if you have a FAT32 drive, and convert it to NTFS, the reserved space can jump around from occasionally.  Invariably it is not immediately adjacent to the MFT because other files are in the way, so expansion of the MFT causes fragmentation as the new extents are in the MFT reserved zone.  Then, if you defrag the MFT (e.g. Diskeeper, Perfectdisk) the MFT reserved space goes somewhere else again. 

At least, in my experience!

I haven't seen it loop since I updated to 3.28 either.

[cmd]

even without START /wait, this shouldn't happen, so long as you are definately using jkdefragcmd.exe

Using the gui version (jkdefrag.exe) will lead to the script continuing before jkdefrag finishes.

What about this "workaround"?


--1--
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Dfrg\BootOptimizeFunction
Enable=N

And... REST!  The above step is the only one necessary to stop Windows from arranging the boot files every 3 days.

Windows will still re-build the layout.ini file every 3 days, which would mean that if JKDefrag would read that file it could place the boot files more logically than Windows does.

--2--
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
EnablePrefetcher=0

The above would stop windows from Prefetching the boot-up sequence and application start sequences, slowing down the system.  Prefetching is not the same as arranging the boot files.  Windows profiles the launch of an application (or the boot of the system) to learn which files are accessed and in which order.  It uses that data next time to pre-fetch these files into the system cache all at once.  This is faster than letting the app grab a little bit of file 1, then a bit of file 2 and then back to file 1 to load another part, etc. which is what tends to happen without prefetching.

--3--
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\OptimalLayout
EnableAutoLayout=0
Type: REG_DWORD

Or, the above key works too.  No point in setting both though.  I have a feeling that this one only stops Windows' arranging the files every 3 days, but if you manually run Windows Disk Defragmenter it will still arrange them (Compacting files stage) before doing the rest of the defrag.  It might also stop Windows re-building the layout.ini file

I think the BootOptimizeFunction one acts on Defrag itself, so that defrag will still be launched every 3 days but won't do anything (quits immediately) and if you manually defrag, it won't arrange the files.

--4--
Delete all files in C:\WINDOWS\Prefetch

Again, slows down your next boot and the next launch of each application until windows can profile them again.  Of course, if you're intent on disabling prefetching then you might as well do this, but otherwise don't bother.

--5--
Reboot

--6--
JkDefrag.exe -a 9 c:

-a 9 is 'Analyze and sort files by last access (newest first)'

I stumbled upon this page, its similar to Donn Edwards work on defragmentation reviews.

This guys gone into a lot of detail, and has some interesting information on the algorithms with some of the defraggers.

Specificly Jeroen, you might find it interesting (btw, your listed), that is, if you havn't seen it yourself already.

http://mindprod.com/jgloss/defragger.html

< Rant on >
IMNSHO, the above page's "reviews" are a crock.  Whole page seems geared towards "I love Norton" review at the end.  Thinks NTFS meta-data is the page file and the hibernate file in one review.  In another he thinks it's ADS.  Actually it's indexes (such as folders), bad cluster file, bit-map file, MFT, MFT mirror, quota, etc. plus any attribute of a file object that doesn't fit within the MFT record.)

Thinks that the defrag API is clunky, even though it achieves safe on-line processing of files on live disks.  But Norton is the best because it is fast and has pretty colours, never mind that it requires you to shut down every program that might write to the disk.

And Craig Jensen is a Scientologist, big wow.  I don't approve of their practices but this has no baring on the quality of the product.

Plus of course, JKDefrag is barely worth mentioning since it doesn't make the disk look the way he wanted it.
< Rant off >

Before writing anymore drivel, maybe that guy should read a bit.  Maybe he could start here: http://ntfs.com/

If the entries were already 0 or missing it means the performance counters were NOT disabled which is fine.

One last try then, look here:

Code:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib

and see if there is a  "Disable Performance Counters" key there?

If that is present and set to 1 it would disable the whole performance monitoring system of windows.

No need to create it if it is not there!

That's weird, mine certainly takes into account disk and CPU usage.  Basically if I'm connected to the net via my wireless and my personal firewall is logging to the disk, it will never go to standby.

It could be that the performance counters have been disabled - check under:

Code:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfNet\Performance
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfOS\Performance
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance

and see if there is a DWORD value:

Code:

Disable Performance Counters=1

Try changing 1 to 0 to re-enable the counters - it may allow windows to take disk, network, processor and "OS" into account when deciding if and when to shut down.

Or he could download a trial version of Diskeeper (you need at least the Pro version) or Perfectdisk and use the boot-time defrag which should reduce the fragmentation of the MFT.

Microsoft documented the prefetcher and boot optimisation features here:
http://www.microsoft.com/whdc/system/sysperf/benchmark.mspx

but it was clearly aimed at the techy reader, not end users since this feature is supposed to be transparent to the user and just speed things up.


As for the commercial defraggers, Diskeeper checks the registry entry that denotes where the boot files are currently located and doesn't move them unless they have become fragmented, thereby preserving the order of the files which helps speed boot-up and application start times.

Perfectdisk goes further and puts the files at the logical beginning of the volume in the order specified in the layout.ini file.  This could yield an even faster boot-up than Windows' positioning of the files "in the first place large enough to hold them".  Unfortunately Perfectdisk's other file pacement strategies are geared towards speeding up future defrags, not speeding file access.

Both companies have documented this in their help files and downloadable manuals.

I often noticed on the C: drive (where windows XP is installed) that even after a good defrag -a3 , though old files were well moved close to the beginning of the disk, that new files are still created by Windows XP far away from the spacehogs near the end of the disk.

Also when windows 'optimizes' the disc (every 3 days or so when idle) it tends to move a small group of files to about two thirds of the way up the disc.

You can emulate that behaviour and see for yourself if that is what you are seeing by running the ProcessIdleTasks command:

Start>Run>Type - rundll32.exe advapi32.dll,ProcessIdleTasks

Takes about ten minutes and has always moved a group of files about two thirds of the way up the disc on all my computers. You can stop Windows doing this by using Microsofts TweakUI (LINK) and unticking the 'Optimise hard disk when idle' option under the general section.
I've disabled it as it messes with JKD's optimal file placement, but I run it manually once a month or so as it needs to be run occasionally to get the best result from XP's prefetch system AFAIK.

If TweakUI uses the following keY:

Code:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Dfrg\BootOptimizeFunction
Enable=N

...then you shouldn't have to worry about running it manually.  The task scheduler will still run the "Idle tasks" including updating performance counters (whatever that actually does!) and building the layout.ini file, it just won't move any files.

new files are still created by Windows XP far away from the spacehogs near the end of the disk.

Yes, it can happen. The free zone's are not always used by Windows. I don't know why, I don't know how or why Windows decides where to place a new file.

One of the design goals of NTFS was to reduce fragmentation.  So the NTFS.sys driver attempts to guess how big a file is and place it in the most suitable gap.  If you're copying drive to drive, it knows exactly how big the file is and can work quite well to place files into gaps in one piece.

But if you're downloading files (e.g. Explorer cache files) it has to guess how big they'll be.  It does this by looking at how much data is waiting in the system cache to be written and how fast that data is coming in.  It does not do a very good job and tends to under-estimate the size a file will be!

In DOS/Win9x MS went for the "First-fit" approach - the first un-occupied cluster was used, even if the file was known to need 50 clusters, it could potentially be fragmented into 50 fragments.

In WindowsNT/2000/XP they went for "Best-fit" which doesn't always work properly.

Apparently Linux in their EXT2 and EXT3 file systems they go for "Worst-fit" where each new file is placed at the beginning of the largest free space gap.  Not sure what happens if 2 files are being written  simultaneously though...  But linux affectionados claim that EXT NEVER fragments and NEVER needs defragmenting unless you want to resize partitions.  Somehow I think they exhagerate...!

Seconded!

I ran jkdefrag on a system where the pagefile had not been moved from where Windows setup put it - right at the beginning of the volume.

Jkdefrag set zone 1 and most of zone 2 to be where the pagefile was.  When it ran out of space in it's zone 2, it left many of the regular files where they happened to be and moved onto zone 3.

So at the end you saw the page file, then some regular (zone 2) files then a mixture of regular files and space hogs (zone 3) files.

Problem with this is that JKDefrag calculates the positions of the zones from just the size of the volume and doesn't know whether a file is un-movable until it tries to move it.

I have the same problem using JkDefrag v3.25

Here's where it reads the MFT entry for $attrdef:

Code:

13:57:54 Processing Inode 4...
13:57:54   Attribute 0: $STANDARD_INFORMATION
13:57:54   Attribute 2: $FILE_NAME
13:57:54     Long filename = '$AttrDef'
13:57:54   Attribute 3: $SECURITY_DESCRIPTOR
13:57:54   Attribute 4: $DATA
13:57:54     Creating new stream: ':$DATA'
13:57:54     Extent: Lcn=604657, Vcn=0, NextVcn=1

And here's where it encounters $attrdef for the first time, during the Zone 2 sort:

Code:

 Moving 10 clusters from 552311 to 617738.
14:08:36 Getting cluster bitmap: C:\WINNT\SYSTEM32\Wbodbc.wll
14:08:36 Extent: Lcn=617738, Vcn=0, NextVcn=10
14:08:36 Vacated 7209 clusters (until 65573) from LCN=552311
14:08:36 C:\$AttrDef
  Moving 1 cluster from 604657 to 617748.
14:08:36 Getting cluster bitmap: C:\$AttrDef
14:08:36 Extent: Lcn=617748, Vcn=0, NextVcn=1
14:08:36 Vacated 59555 clusters (until 65573) from LCN=604657
14:08:36 C:\WINDOWS\system32\wshext.dll
  Moving 16 clusters from 610675 to 617749.

And here's where it tries to move $AttrDef and gets stuck during Zone 3 sort:

Code:

14:08:39 Zone 3: Sort
14:08:39 Starting to sort zone 3.
14:08:39 Gap found: LCN=604659, Size=13089
14:08:39 C:\$AttrDef
  Moving 1 cluster from 617748 to 604659.
14:08:39 Getting cluster bitmap: C:\$AttrDef
14:08:39 Extent: Lcn=604659, Vcn=0, NextVcn=1
14:08:39 Gap found: LCN=604660, Size=28531
14:08:39 Vacating 4624506 clusters starting at LCN=610675
14:08:39 Vacated 0 clusters (until 4624506) from LCN=610675
14:08:39 Vacated 22535 clusters (until 4624506) from LCN=633210
14:08:39 Vacated 22546 clusters (until 4624506) from LCN=633221
14:08:39 Vacated 22558 clusters (until 4624506) from LCN=633233
14:08:39 Vacated 22570 clusters (until 4624506) from LCN=633245
14:08:39 Vacated 22585 clusters (until 4624506) from LCN=633260
14:08:39 Vacated 22613 clusters (until 4624506) from LCN=633288
14:08:39 Vacated 22618 clusters (until 4624506) from LCN=633293
14:08:39 Vacated 22846 clusters (until 4624506) from LCN=633521
14:08:39 Vacated 23105 clusters (until 4624506) from LCN=633780
14:08:39 Vacated 23106 clusters (until 4624506) from LCN=633781
14:08:39 Vacated 23146 clusters (until 4624506) from LCN=633821
14:08:39 Vacated 23178 clusters (until 4624506) from LCN=633853

...etc.

Is there a clue in the fact that it already encountered and moved $attrdef once before, but can't the second time?  Did it forget to note where it put it?

Full log is available if required.

This applies to the normal defrag, right?

Diskeeper also has a command line boot optimization feature, which moves the boot files to a better location than defrag, if I remember correctly. Maybe old Pro version or something like that.

I think they removed this option in Diskeeper 2007 but in previous versions you could run:

Diskeeper C: /b

just like you could run

defrag C: /b

I don't recall that they claimed to move them to a better location than Windows though.  In my experience it just moved them to the next available area capable of holding them, like Windows.