Defragment(Inplace)

[BloodySword]

Hi,

don't know if this was already suggested before but I have an Idea to defragment files very fast:

Let's assume we have a 5 GB file wich is fragmented into 4 pieces: A, B, C, D.

The layout is as follows:


DDDDDDDDDDDDDDAAAAAAAAAAAAAAACCCCCCCCCCCCCCCCBBBBBBBBBBBBBBBBBB

Instead of searching free space to move the file in the right order to, we shuld consider to kmove only the fragments till the file is put together.

The first step would be to move fragment B and C at the end of itself:

DDDDDDDDDDDDDDAAAAAAAAAAAAAAA__________________________________CCCCCCCCCCCCCCCCBBBBBBBBBBBBBBBBBB

Then move B to A:

DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBB________________CCCCCCCCCCCCCCCC

Move C to B:

DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCC

Move D to the end of the file:

AAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCDDDDDDDDDDDDDD

Defrag of the big file completed.

This would make the defragmentation MUCH faster in most cases.

[Kasuha]

Assuming the relative size of segments is proportional to number of letters you used, in your example you actually moved more data than would be moved if the file was placed to a new location in single operation.

I also disagree that it would make defragmentation much faster in most cases, in my opinion the speed gain would be quite small on average at a price of a very complicated algorithm that would be needed to handle all cases that may happen (typically with other files and unmovable chunks of permanent or temporary data lying in the way).

Apart of that, there sure are cases when defragmenting in place would mean less data moves than defragmenting by moving to a new location and it has been suggested already.

[BloodySword]

If possible MyDefrag should use this technique to defrag very big files, if this approach will move less data.
This should be taken into account, it is very annoying the defragment routine takes so long.

[Kasuha]

You can use defragment(chunksize) for very large files. It is not useful for general defragmentation but it is useful for growing files like logs or disk images.
In general case, a large growing file is surrounded by other files because of either sorting or filling the zone. Any extra data then mean at least the same amount of data of other files must be moved aside. Defragmenting just the extra data usually does not mean significant performance penalty for the file and prevents moving original large parts of the file.

[Tracey2031]

 Please, someone please correct me if I'm wrong (because I would really like to know), but I don't think that is possible.
From what I have read from Microsoft.com and experienced with Windows Disk Defragmenter, it appears that Windows defraggers will not move a file unless there is sufficient free space (fragmented or not) to accept the whole file (for integrity reasons).
I once had a 15GB drive with (2) fragmented 5GB files that no Windows Defragger on the market would defragment.

In other words, the defraggers will not move part of a file then update the (FAT/NTFS $MFT).
Every "move" moves the whole file and updates the (FAT/NTFS $MFT) as applicable.
This keeps the possibility of file and (FAT/NTFS $MFT) corruption to a minimum.
If the application is interrupted before the move is complete (and I have done that on junk data just to see what happens), the original data is still in the original location and the (FAT/NTFS $MFT) is correct (not corrupted).
Then, the place for error is to interrupt the updating of the (FAT/NTFS $MFT) which is a small percentage of the total time.
If you moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT for each file then you multiply the possibility of corruption with each updateMFT.

If I remember correctly, data integrity (and security) was a goal of MS-NTFS.
Just FYI, Tracey
I spent a lot of {fun} time using the DOS versions of Norton Disk Doctor and Disk Editor fixing FAT errors to recover data during the pre-NTFS era.

[BloodySword]

You can use defragment(chunksize) for very large files. It is not useful for general defragmentation but it is useful for growing files like logs or disk images.
In general case, a large growing file is surrounded by other files because of either sorting or filling the zone. Any extra data then mean at least the same amount of data of other files must be moved aside. Defragmenting just the extra data usually does not mean significant performance penalty for the file and prevents moving original large parts of the file.

It does not work like expected. Furthermore it does not change anything... :o(

Please, someone please correct me if I'm wrong (because I would really like to know), but I don't think that is possible.
From what I have read from Microsoft.com and experienced with Windows Disk Defragmenter, it appears that Windows defraggers will not move a file unless there is sufficient free space (fragmented or not) to accept the whole file (for integrity reasons).
I once had a 15GB drive with (2) fragmented 5GB files that no Windows Defragger on the market would defragment.

In other words, the defraggers will not move part of a file then update the (FAT/NTFS $MFT).
Every "move" moves the whole file and updates the (FAT/NTFS $MFT) as applicable.
This keeps the possibility of file and (FAT/NTFS $MFT) corruption to a minimum.
If the application is interrupted before the move is complete (and I have done that on junk data just to see what happens), the original data is still in the original location and the (FAT/NTFS $MFT) is correct (not corrupted).
Then, the place for error is to interrupt the updating of the (FAT/NTFS $MFT) which is a small percentage of the total time.
If you moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT for each file then you multiply the possibility of corruption with each updateMFT.

If I remember correctly, data integrity (and security) was a goal of MS-NTFS.
Just FYI, Tracey
I spent a lot of {fun} time using the DOS versions of Norton Disk Doctor and Disk Editor fixing FAT errors to recover data during the pre-NTFS era.

All I can say because I have seen it several times: MyDefrag can move a file fragment to another location, it can even move a part of a file that is NOT fragmented.

[Tracey2031]

Regardless of how a files is moved:
Is it true that the (FAT/NTFS-$MFT) is updated only once after the entire file is moved (yes or no)?
I assumed yes because I have halted/interrupted a large file move due to time constraints without loosing any data or corrupting the NTFS-$MFT

In other words, if I have a 4GB file and the most Windows will move at once is 1GB, then will Windows update the (FAT/NTFS-$MFT) 4 times or once after 4 (or more) iterations of moves?
I assumed once after 4 (or more) iterations of moves because of the reason above.

Please advise.
Thanks, Tracey

[Darlis]

The MFT will be updated (at least) after every move operation, and in case of a 4GB file, at least 4 times.
If you move such a file with MyDefrag and cancel the operation half way, the file will be in two fragments and the file and MFT are not corrupted.

You will have to use really drastic measures to get an NTFS file system into an inconsistent state. NTFS has also a journaling feature ($logfile) with which inconsistent states can be repaired.

Edit: http://en.wikipedia.org/wiki/Ntfs#NTFS_Log

[WHRoeder]

DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBB________________CCCCCCCCCCCCCCCC

Move C to B:

The problem is the API can not move all of C, You can only move that portion corresponding to free space. therefor you get

DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBB________________CCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBCCCCCCCCCCC________________CCCCC
DDDDDDDDDDDDDDAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCC________________
What would have been four subsections moved to one spot, is now potentially many more moves.

[Kasuha]

It does not work like expected. Furthermore it does not change anything... (

Yes, I understand it does not do what you'd like to do. It even may do what'd you expect it to do, or at least it did not do what I expected from it when I was trying to use it. But it works exactly as designed and as soon as you accept the fact that multi-gigabyte files don't need to be completely defragmented it reduces amount of data moves to optimize them significantly.

[jeroen]

it appears that Windows defraggers will not move a file unless there is sufficient free space (fragmented or not) to accept the whole file (for integrity reasons).

MyDefrag can do it. It will try to free up as much space as possible (by "vacating", moving files out of the way), then move as much of the fragmented file as possible, then vacate some, move some more, etc, until the file is contiguous.

In other words, the defraggers will not move part of a file then update the (FAT/NTFS $MFT).

MyDefrag can move parts of files. Also see the MyFragmenter utility that comes with MyDefrag, which can fragment files, which is only possible my moving part of a file.

If you moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT-moveFRAGMENT-updateMFT for each file then you multiply the possibility of corruption with each updateMFT.

Files cannot get corrupted when MyDefrag is interrupted. I don't know how Windows exactly does it, but it's simply not possible. For more information see How safe is MyDefrag?

[Vanex]

Hello,
 I like inplace idea, one more example:

standard idea:

Code:

DDDCCCBBBAAA
---------------DDDCCCBBBAAA = 4 moves
AAABBBCCCDDD                = 4 moves

inplace idea:

Code:

DDDCCCBBBAAA
---CCCBBBAAADDD            = 1 move
AAACCCBBB---DDD            = 1 move
AAACCCBBBDDD               = 1 move
AAA---BBBDDDCCC            = 1 move
AAABBB---DDDCCC            = 1 move
AAABBBCCCDDD               = 1 move

result:
standard: 8 moves, requires 4 free spaces
inplace: 6 moves, requires 1 free space

ok?

[BloodySword]

Files cannot get corrupted when MyDefrag is interrupted. I don't know how Windows exactly does it, but it's simply not possible. For more information see How safe is MyDefrag?

The answer is as easy as you think: The file fragments are not actually moved, but they are copied to another place. So the data of the fragment is on the old place, too and will not be overwritten till the MFT is updated with the new fragment location.

Even on a Power lost or on unexcepted Power off, when the MFT is not updated or even the fragment was not "moved" completely, the old position and old fragment data is still there. So there is nothing to worry. The MFT still points to the position before moving and it is as if the file fragment was never moved! That's no secret and no magic. And since NTFS is a journaling file system we know why there is no chance for corruption.

[poutnik]

Also since Vista NTFS operations in Windows are transactional,
i.e. operation is either full success either failure and roll back.