$UsnJrnl (NTFS Change journal) huge and won't defrag on Domain Controller

[andublin]

All my domain controllers are showing extremely large and fragmented  $UsnJrnl (NTFS Change journal) .
JKDefrag won't move or defrag it.
May not be a bug, but What's causing it to grow? How to defragment, and shrink, without disrupting FRS and whatever else depends on it? Thanks.
sample analysis from jkdefrag shows 968 fragments,  569MB.
These items could not be moved:
Fragments       Bytes  Clusters Name
968   596758616    131216 C:\$Extend\$UsnJrnl:$J:$DATA
1    33128448      8088 C:\$MFT
1        4096         1 C:\$MFTMirr
1        4144         2 C:\.::$SECURITY_DESCRIPTOR
1      654904       160 C:\$Bitmap
--------- ----------- --------- -----
972   630550208    139467 Total
These items are still fragmented:
Fragments       Bytes  Clusters Name
968   596758616    131216 C:\$Extend\$UsnJrnl:$J:$DATA
--------- ----------- --------- -----
968   596758616    131216 Total
The 25 largest items on disk:
Fragments       Bytes  Clusters Name
1   792723456    193536 C:\pagefile.sys
968   596758616    131216 C:\$Extend\$UsnJrnl:$J:$DATA
1   134217728     32768 C:\WINDOWS\system32\config\SecEvent.Evt
1   134216884     32768 C:\WINDOWS\repair\Backup\ServiceState\EventLogs\SecEvent.Evt
1    67108864     16384 C:\$LogFile
1    62930944     15364 C:\WINDOWS\NTDS\ntds.dit
1    58344934     14245 C:\WINDOWS\Driver Cache\i386\driver.cab

[jeroen]

What's causing it to grow?

See the FsUtil manpage for more information.

How to defragment, and shrink, without disrupting FRS and whatever else depends on it?

See the JkDefrag homepage (or the FsUtil manpage) for instructions on how to delete this file. This will not disrupt anything, it's a cache (of sorts) that speeds certain actions up, and deleting it will return those actions to their regular speed.

[andublin]

Thanks Jeroen.
I do understand the purpose of the change log; (used on domain controllers by the File Replication Service, in particular), so I'm reluctant to just delete it; and that any change in any file in drive C is triggering a new record there. Has anyone got real knowledge about how dependent the FRS is on the Change Journal?

The growth/fragmentation was only noticed AFTER jkdefrag had been run a few times, so I wondered if jkdefrag file moves were actually causing it to grow? And does Windows defrag not cause it to grow, or it just wasn't noticed before?
[edit] I've checked a domain controller in another organisation which have never had jkdefrag; same situation, massive change journal, massive fragmentation of it; Windows defrag just doesn't report it. Probably the same on every Domain Controller everywhere . . .

Thanks again, great software.

Alan.

[lh]

The usn journal is just so you can track any sort of changes to the files.  For something like replication it would be very useful.  You could consult this file to see what to move instead of trolling the whole disk to see if anything changed.  It does not track the actual data changes just marks down that 'something happened to this file'.  It is also useful for things like a virus scanner.  It tracks all sorts of changes such as permissions, file size changes, data changes, metadata changes, etc.  It is just a change log minus the actual changes.

What you have discovered is the flaw with the usn journal and NTFS.   You can not defrag it as the ioctrl call does not let you (the one used by both the windows defragmenter and jkdefrag).  I suspect because the file system has it locked.  NTFS in many cases will also not create a slowly growing file (such as the event log or in this case the usnjournal) in a contiguous manner.  So you end up with a file that is growing and growing and you can not defrag it.  The only way I have discovered so far to clear up this fragmeted file is to delete it.

[andublin]

If deleted, what impact does that have? is it auto-recreated, or does file replication stop working, or merely slow down and increase i/o?

Thanks.

[lh]

Depends on what is using it.  If you are not using the feature from the service that turns it on, it should have 0 impact.  It holds onto a TON of stuff.  If you look at it using the fsutil you will see things in there from probably months ago.

Not quite sure what effect it would have on a domain controller acting as a active directory would do though.  I do not think they use that to replicate the changes thru an active directory tree.  But I could be wrong.  If you have 1 domain controller it is probably safe to remove.  But if it is in a cluster you probably want to offline it and do it.  The offline should force any syncing that needs to happen. 

I would setup a set of test virtual machines to try it first before pushing it onto your production machines.

[jeroen]

If deleted, what impact does that have? is it auto-recreated, or does file replication stop working, or merely slow down and increase i/o?

The $UsnJrnl is auto-recreated and you can safely delete it, everything will keep working. A few Microsoft applications use it to speed up some very specific tasks, somebody posted a list some time ago. I forget which tasks exactly but I remember I am not using any of them on my servers. For me the $UsnJrnl is a total waste of disk space.

[robert]

The $UsnJrnl is auto-recreated and you can safely delete it, everything will keep working.

This can be a really BAD idea on a domain controller. If you delete this file it KILLS the File Replication Service that keeps multiple DCs in sync. After doing this you have to manually reconnect and reinitialise the NTFRS  from a DC that is still working.  If you do this on all your DCs (eg: both) the only supported way to get back to a working system is a complete domain reinstall (or backups).

So while it is normally safe to delete this file there is a ball buster waiting for you.

[BloodySword]

That's the reason why I hate this f***n journal!