Intro Download and install Frequently Asked Questions Tips and tricks

Homepage







© J.C. Kessels 2009
MyDefrag Forum
October 23, 2014, 06:06:50 am *
Welcome, Guest. Please login or register.

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: [1] 2
  Print  
Author Topic: How to defragment FAT directories on Win32 without rebooting (basic CMD script)  (Read 12630 times)
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« on: August 02, 2009, 01:51:13 am »

It's true that the Microsoft's defragmentation API cannot move directories on FAT/FAT32 volumes, but this does not mean that directories cannot be moved. In fact, most FAT/FAT32 volumes are used for non-system disks used for storage, and not being able to move them means that they may be spread all over the disk and they will eventually cause large files to be fragmented as well. This is not a very cumfortable situation for devices used for long term preservation of files, because any fragmentation on a FAT volume exposes them to risks if those volumes are regularly used to add more backup files, if a bitmap or even a single directory gets corrupted (think about how CHKDSK works, to recover this situation, and it may happen that files are permanently damaged or overwritten by cross-linked clusters.)

That's why I have had an idea that effectively allows FAT directories to be consolidated on some free space (most probably in the middle of the volume if running this concept on Windows XP, 2003, Vista, 2008 or Seven):
- First run a normal fast defragmention
- consolidate the free space (use "Force Together"), even if this fragments some files
- run the following "movedirs.cmd" script to consolidate the directories
- rerun a normal fast defragmentation

The command file "MoveDirs.cmd" below (using cmd's EXTENSIONS and DELAYEDEXPANSION found in Vista) works on FAT32 volumes, not directly on FAT volumes without LFN support, because it requires long file names for naming intermediate working directories. DO NOT USE IT ON FAT-ONLY VOLUMES THAT DON'T SUPPORT THE LONG FILENAMES EXTENSION.


@echo off

verify NUL 2>NUL
setlocal ENABLEEXTENSIONS
if ERRORLEVEL 1 goto ExtensionsMissing

verify NUL 2>NUL
setlocal ENABLEDELAYEDEXPANSION
if ERRORLEVEL 1 goto ExtensionsMissing

call :MoveDirs %1
goto :EOF

:ExtensionsMissing
ECHO Cannot run this script (Extensions and DelayedExpansion are not supported in current CMD processor)
goto :EOF

:MoveDirs
setlocal
for /F "usebackq" %%D in (`DIR /B /A:D /OGNE %1`) do (
   set P=%~1\%%~D
   echo ===============================
   echo Processing !P!
   MKDIR 2>NUL "!P!._"
   FOR /F "usebackq" %%F in (`DIR /B /A /OGNE "!P!\."`) do MOVE 1>NUL 2>NUL "!P!\%%~nxF" "!P!._"
   RMDIR 2>NUL "!P!"
   MKDIR 2>NUL "!P!"
   FOR /F "usebackq" %%F in (`DIR /B /A /OGNE "!P!._\."`) do MOVE 1>NUL 2>NUL "!P!._\%%~nxF" "!P!"
   RMDIR 2>NUL "!P!._"
   call :MoveDirs "!P!"
)
:EOF
endlocal


Note: this script will not detect if subdirectories are spanning several distinct volumes, bcause it cannot detect when they are mount points (but such mount points, or "junctions", should not exist on FAT32 volumes (only on NTFS), so this should not be an issue).

If the system's Cmd.exe processor does not support the extensions, the same can be performed in another language that can create and remove directories, and move files from one directory to another in the same volume (without performing copies).

This script also reorders the contents of each directory, so that it will frst list the subdirectories before the other files (all of them being then ordered by name and extension): this speeds up the traversal of multiple directories when parsing long file paths, as it reduces the data that needs to be read from each traversed directory.


On volumes with tens of thousands of directories (and even more files to move from any directory to a work directory and back) it can take a considerable time (notably if running on the system disk of Windows).

It takes a single optional parameter, a single path of a parent directory to process (you can pass "." to process only directories within the current directory). If you run it without parameter, it will process all subdirectories of "\" on the current drive. IF you just give a drive letter like "D:" it will effectively process all files on that volume (pass "D:." if you want to process only directories on the current directory of drive D:)

It can work in NTFS volumes, but use it at your own risks (beware of mount points !). There are very minimum error checking in this script (but normally it should not loose any file). In case of failure or if you want to interrupt it, look for a possible single directory named "*._" and move any files or subdirectories in them back to the directory with the extra "._" extension. Then drop the now empty directory with the extra "._" extension.

Note that all directories that are read-write accessible will be recreated with today's date (so this may influence your backup). The script does not maintain the creation date. Anyway, I've been able to use it on a very large FAT32 volume created by an external NAS, and even on my Vista system partition (most directories were processed except a few ones whose contents are unreadable, even by administratrors with elevated privileges, or whose content are protected by Windows that does not allow some files to be moved). So you may experiment a few benign errors when a file cannot be moved, or when removing an old unmoved directory still contains some files or subdirectories that could not be moved, and when trying to recreate it with the same name when it was not first removed).

Finally, note that there may exist two variants in the way the recursion is done: above it uses the depth-first traversal order, so that the parent directories will be handled and ordered AFTER their children subdirectories.
You may want to use the parent-first traversal order, by moving the "call :MoveDirs..." line in the middle of the loop just at end of it, just before the ")" loop-closure line.

----

I've just found a work-around for the directories that were treated multiple times (this is a caveat that occurs only on FAT volumes, where the CMD.EXE processor has a bug when handling the higher-level "/FOR /D... in (*) DO " loop; I replaced it with a "FOR /F" loop).

This script also works on Windows 7.
« Last Edit: December 02, 2009, 06:18:15 am by verdy_p » Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #1 on: August 02, 2009, 08:12:45 am »

Thank you very much for contributing your script, I appreciate it. Personally I think it's best to convert all FAT disks to NTFS, but that is not always possible, especially for external equipment with a harddisk such as MP3 players. I also think your method takes a LOT of time and a lot of datamovement, just to defragment the directories. Personally I would not be willing to invest so much time, and would just leave the directories as they are.

I see in your script that you create a new directory into a temporary name, move all the stuff from the old directory to the new directory, delete the old directory, make it new again, move all the stuff back from the temporary directory, and then delete the temporary directory. Isn't that a very dangerous procedure? What happens if the temporary could not be created? I think all data will be lost? I am thinking about Windows permissions and userid's.

Perhaps it is possible to replace the first three steps by using the move command to rename the original directory into the temporary directory? It would save a lot of time and data movement, and would be a bit more safe, because if the rename doesn't succeed then you can abandon and nothing has happened.
Logged
usch
JkDefrag Senior
****
Posts: 35


View Profile
« Reply #2 on: August 03, 2009, 08:41:15 am »

Isn't that a very dangerous procedure? What happens if the temporary could not be created? I think all data will be lost? I am thinking about Windows permissions and userid's.
There are no permissions on FAT drives, so that should be no problem. If the first MOVE fails (for example because the disk is full), the RMDIR will fail in turn because the directory is not empty, then the MKDIR will fail because a directory with that name already exists, and the second MOVE will simply move the files back to the same directory where they came from. Nevertheless it is a bad idea to suppress all error messages ("2>NUL") IMHO.

But there is another issue that might cause problems. Moving things around like this breaks the mapping of long filenames to their short 8.3 aliases.

Suppose you install a program in a directory called "SomeProgram v4.0". That directory will probably have an 8.3 alias of "SOMEPR~1". Now you install two updates in directories called "SomeProgram v4.1" and  "SomeProgram v4.2". Consequently these directories will have an 8.3 alias of "SOMEPR~2" and "SOMEPR~3". Now you delete the first two directories, leaving only "SomeProgram v4.2" aka "SOMEPR~3" behind. If you remove and recreate this directory, windows will use the lowest possible number for the alias tail, so "SomeProgram v4.2" will now have an alias of "SOMEPR~1" or "SOMEPR~2". If something somewhere used the short path name to remember the program location, it won't be able to locate it anymore. Even worse, files and directories may swap their aliases after the script is run, so the old short path will now point to the wrong file without notice. Try a search for "~1" or "~2" in the registry with regedit, you will be surprised how many 8.3 path names there are, at least on XP (don't know about Vista).
Logged
pepoluan
JkDefrag Supporter
***
Posts: 19


View Profile
« Reply #3 on: August 13, 2009, 11:04:53 pm »

If you have enough free space in another drive, I recommend copying everything from the FAT/FAT32 drive into that other drive, QuickFormatting the FAT/FAT32 drive, and copy everything back in.

To make the copy process fast, I'd recommend also using copy-accelerators such as:
* FastCopy - http://www.ipmsg.org/tools/fastcopy.html.en
* Teracopy - http://www.codesector.com/teracopy.php (my personal fave)
* RichCopy - http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx?pr=blog
* SuperCopier - http://supercopier.sfxteam.org/
* Copy Handler - http://www.copyhandler.com/
Logged
poutnik
JkDefrag Hero
*****
Posts: 1112


View Profile
« Reply #4 on: August 16, 2009, 12:25:52 pm »

Does MOVE command on FAT perform real data movement ?
I ask, because on NTFS it does not, but cannot try on FAT.
Logged

It can be fast, good or easy. You can pick just 2 of them....
Treating Spacehog zone by the same effort as Boot zone is like cleaning a garden by the same effort as a living room.
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« Reply #5 on: August 27, 2009, 09:14:01 pm »

Does MOVE command on FAT perform real data movement ?
I ask, because on NTFS it does not, but cannot try on FAT.

No, and it is not supposed to do so. It just moves the reference of a file from one directory to another, without moving the data, that's why this script can be quite fast.

Yes it has caveats with some long filenames or directory names referenced by their short name, in the rare case where there are multiple long names that have the same short name prefix: in that case, the ~number suffix may have a lower value (in very rare cases in fact) after running this script.

Unfortunately, there's nothing in the Win32 File API or in the CMD builtin commands to fix the short file names that can be added to long files names on FAT32 or NTFS volumes.

But don't worry too much: the really long names used in the registry will almost always be for "Program Files" or similar directories, whose short name uses the ~1 suffix that this script will preserve (even if a ~2 suffix is used for the temporary directories used to recreate the original directories).

I've tested this script even after braking it in the middle: there will always be at most one temporary directory remaining that could contain files remaining in it. I've also tested this case: you can just restart the script, and it will automatically fix the files automatically in the original directory.

Some files will also refuse to be moved to another location, so they will stay in their original directory: the original directory will NOT be deleted (that's why I hide the errors by redirecting them to "2>NUL", they are just boring, and in fact not giving useful information; in addition, displaying too many messages just makes the script run MUCH slower).

So if some files can't be moved (because somme running app or service has some locks on them or maintain them open), they will stay in place, and the original directory will not be deleted (RMDIR here does not delete files present, and just reports an error, that this script can SAFELY ignore). Only the movable files or subdirectories (not locked and not open in Windows apps) will be moved (apriori safely) between the temporary directory and the new directory, for a very short period of time.

Note also that this script processes each directory, one at a time. If you break it in the middle (CLTR+C) you know where to look for to restore files in a single location, just with your Windows Explorer (and possibly recreating the new directory if you're just between the RMDIR and MKDIR commands, something that is very unlikely as these commands are quite fast compared to the other moves performed).

Note that I used an "._" extension for the temporary directories that is very unlikely to be used by your actual directories. The script can only run on FAT32 volumes, but MUST not be used on FAT only volumes that do not support the LFN extension (this generally happens with some USB devices storing files on flash storage, like cameras, but there's generally no need to defragment the directories, as there are in fact very few directories on such disks and their possible fragmentation is really not a problem)

This method can be used manually from within the Windows Explorer for a single directory: create a second directory with a similar name so that it will be visible just side by side with the existing directory to defragment, then move files from the source to the temp directory (by selecting themall and cutting them with CTRL+A CTRL+X, then displaying the target and pressing CTRL+V). If some files can't be moved, just keep them in their existing location. Now:
* if all files could successfully be moved, permanently delete the now empty source directory (if yuo use delete, empty your desktop bin)  and recreate it immediately. Then move back files from the temp directory and permanently drop the temp directory when it is now empty.
* if some files could not be moved, just move back the content of the temp directory to the original location and drop the temp directory: even in this case, yuo may notice that some allocated space as been freed up (this happens with directories that can be used to accumulate lots of log files, after purging some of them, notably the oldest ones: not all the directory space is reclaimes by windows as there will remain unused entries near the begining of the directory).

This can be used if you have run MyDefrag and it signals that there's still a few fragmented directories spread in the middle of the disk, and sometimes blocking a more complete defragmentation of other files stored in the same region of the disk.
« Last Edit: August 27, 2009, 09:51:23 pm by verdy_p » Logged
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« Reply #6 on: August 27, 2009, 09:30:49 pm »

Perhaps it is possible to replace the first three steps by using the move command to rename the original directory into the temporary directory? It would save a lot of time and data movement, and would be a bit more safe, because if the rename doesn't succeed then you can abandon and nothing has happened.

You're right, in theory, and it's true that it will mostly double the speed, but I've noted that it left more fragmentation on disk than recreating a new tempory directory (in its own location) and then delete it after use.

This can be something to try. Now, if this can give you an idea for implementing it in the C++ code of your application (but only for FAT or FAT32 volumes with the LFN extension enabled ! You need to check at the volume properties using the Win32 API to see if the FAT/FAT32 volume supports LFN).

It could be even faster, but does not happen as much often in fully defragemented directories, that this script will also presort (for faster display in the Browser) and compress in size (dropping the unnecessary free places in the middle of a directory, and that Windows does not free up if there are other directory entries stored further down).

Note that the FOR loop to scan a directory has a caveat: when some files within a directory is moved out of a directory, it can return the same file names multiple times (I've seen that this occurs when there are free directory entries in the middle of an existing FAT/FAT32 directory, but also on NTFS volumes for the same reason) This is, I think, a bug of the windows CMD.EXE processor. But it does not make the script unsafe to use because of that, but just longer to run than necessary.

I've seen also that this script can be used to compress a directory that has plenty of unused entries in them also on NTFS volumes.

But I cannot recommand this method on NTFS volumes, given that the newly recreated directory will not inherit the same security descriptors, creator attributes, and other attributes stored as named streams, and given that the script will behave very badly with NTFS reparse points (for mounted volumes or virtual directories managed by some DLL extension or running application or by some external communication protocol). It is not needed anyway, as MyDefrag can process NTFS volumes, even if it cannot compress their unused entries.
Logged
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« Reply #7 on: August 27, 2009, 10:04:12 pm »

If you have enough free space in another drive, I recommend copying everything from the FAT/FAT32 drive into that other drive, QuickFormatting the FAT/FAT32 drive, and copy everything back in.

To make the copy process fast, I'd recommend also using copy-accelerators such as:
* FastCopy - http://www.ipmsg.org/tools/fastcopy.html.en
* Teracopy - http://www.codesector.com/teracopy.php (my personal fave)
* RichCopy - http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx?pr=blog
* SuperCopier - http://supercopier.sfxteam.org/
* Copy Handler - http://www.copyhandler.com/


This is possible, but this method is much longer for large volumes, and is in fact much more dangerous (because you'll likely use the DELETE operation after moving files to another volume, before restoring them). Note also that these fast copies do not guarantee that the files will be unfragmented on the destination voulme after the transfer.

Notably, you'll very often see that the directories themselves ARE fragmented, because they are filled progressively after each file has been copied, instead of globally in one operation.) The same problem occurs when restoring files from a backup (including when restoring older versions of files from Windows Shadow Copies on NTFS volumes, or when using the Windows System Restoration tool on such volumes): the newly recreated directories will also get fragmented and spread all over the volume, and the MFT will also get refilled in a quite fragmented way with fragments spread all other the volume.

My script runs on the SAME volume, without needing any extra space. As a bonus, it will also sort your directories (on FAT volumes, because on NTFS, the directory entries are kept stored in ascending name order, independantly of the fast that these directories entries are designating subdirectories or files) so that their subdirectories will be listed first, sometimes speeding the access time a bit, when a directory has lots of subdirectories: these subdirectories will be scanned and found faster with less IO operations and using less memory.
Logged
Gurky42
JkDefrag Senior
****
Posts: 45


View Profile
« Reply #8 on: September 19, 2009, 04:40:17 pm »

Howdy all,

I'm not sure if a separate thread is needed since most of the info prior to my post is relevant.

--

One can also consolidate FAT32 directory structures, even on a FAT32 windows system drive (C:) when using Linux. Mind you, this is a catch-22 situation, because all files locked by the win OS won't usually be locked when a device is mounted under Linux with FAT32... For example: When I was done testing my bash script (over and over), my 1G 'pagefile.sys' file ended up having more then 20 fragments. I'd assume this can happen with other files windows puts a lock on. Anyway, this is what I did as an experiment:

** CAUTION: DO NOT TRY THIS AT HOME **

Unless you are brave, stupid, or have experience with both OS's... But most importantly, I may have forgotten to include certain steps I took.
 
1 - Use "MyDefrag 4.2.0 Beta --> MoveToEndOfDisk.MyD" to move as many files as possible towards the end of the disk C:.

2 - Rebooted using the newest version of PuppyLinux. Like I said, this was just an experiment!

3 - Made the choice to consolidate every directory under 'C:/Program Files' (Thousands of files and about 500+ directories). Mounted both my C: drive and a second FAT32 drive to hold a master copy of 'C:/Program Files'.

4 - Here's the fun part I can't explain. I created a bash script to copy only directory entries one after the other. Once done, copy over the files associated with those directories. During this process I may have used a ton of move/copy/replace/rename commands on my C: partition. Might explain why some of the windows system files got fragged.

5 - Once I was sure the bash script did what I wanted, I re-booted Windows XP with no problems. Re-ran the MYDefrag daily script(with pause), and low and behold, I saw a long red line marked as unmovable. When I checked this, it was the directories from C:/Program Files.

--

The end result was that a huge number of gaps on my FAT32 system disk was reduced by up to half. Of course, I have to wonder if doing something like this is worth the effort and risk.

Lastly, for me, converting to ntfs is not an option.
Logged

--

Nothing New...
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« Reply #9 on: December 02, 2009, 03:52:51 am »

If you have enough free space in another drive, I recommend copying everything from the FAT/FAT32 drive into that other drive, QuickFormatting the FAT/FAT32 drive, and copy everything back in.

Actually NO ! Copying everything back in, will NOT consolidate the directories, notably because they will be recreated one after the other, and they will fill up progressively, and will be allocated in the middle of of other files copied into those directories. You will even see that some directories are created on disk in the middle of a file copied into it or onto a nearby directory. The reason is that Windows will delay the actual allcoation and will not create all the directories at once.

This is easy to see:

If you have a USB flash key with at least 2.5GB, format it with FAT32 (it will be completely empty), then copy all files of the Windows DVD onto it, using Windows Explorer. You'll see that it has high level of fragmentation, notably for very large files like *.WIM files containing the mountable partition images that the Windows installer will use. And these files are spread all over the disk. You'll also see that Windows will have allocated the directories near the middle of the volume.

If you really want to make sure that the directories will be packed together, you have first to create the directory structures (using all existing files names but using files with empty contents, to make sure that each directory will be sorted and will have the correct size allocated as one fragment on disk.

My script solves the problem without having to boot on another OS, or using any other special tool than the default CMD.EXE command processor built into Windows.
Logged
verdy_p
JkDefrag Hero
*****
Posts: 62


View Profile
« Reply #10 on: December 02, 2009, 06:29:10 am »

But there is another issue that might cause problems. Moving things around like this breaks the mapping of long filenames to their short 8.3 aliases.
I replied in an earlier post long time ago. This is a non issue for almost cases. Short filenames automatically generated from long file names, and that may cause conflicts if they change, should not occur on most modern applications for Windows, where short filenames should not be used by installers.
There are still a few ("C:\Program Files" is frequently mapped to "C:\PROGRA~1") but the probability that there will be at least 3 directories whose short file name may be renumbered in the same parent directory is extremely small (this case mostly affects filenames, but typically not directory names).

I've even used this script on the system partition of Windows XP, Vista or Seven, and there was no such issue. There was NO directory actually renumbered. Software installations should NEVER depend on the short filenames of their directories, when they can safely use LFN's instead.

Note: the script ignores hidden directories (so the "System Volume Information" will not be processed) unless you specify one explicitly in the first parameter, between double quotes.
Logged
khagaroth
JkDefrag Junior
**
Posts: 6


View Profile
« Reply #11 on: December 12, 2009, 11:07:25 pm »

Theoretically, it should be possible to defragment directories on a FAT volume if mydefrag locks and dismounts the volume before starting the defragmentation.
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #12 on: December 13, 2009, 02:01:58 pm »

Theoretically, it should be possible to defragment directories on a FAT volume if mydefrag locks and dismounts the volume before starting the defragmentation.
No. That has been tested, and a lot more, but the Microsoft defragmentation API simply refuses to move directories on FAT volumes.
Logged
JFD62780
Newbie
*
Posts: 3


View Profile
« Reply #13 on: December 16, 2009, 10:50:59 am »

Okay, first time poster with a question:  Would it be possible to consolidate the FAT32 directories at the very beginning of the drive with your script, if I manage to move all the files themselves to the end of the drive?

And will the files themselves retain their modified timestamps?  Also I'm concerned about certain long filenames with certain characters, be they Unicode or something...

Long story short, I've split a terabyte-sized portable hard drive in two: about half a terabyte of FAT32 for the first half, and the rest is NTFS.  Basically it's a backup/media drive that happens to be PS3-compatible, sort of. (at least the first half is.)  I'm merely trying to optimize the FAT32 partition.

What I'm getting at is, if this script works, then this might be a lifesaver for me!

Oh, and Jeroen:  Can this idea be converted to a MyD script, for a potentially new feature for a future version if evolved enough?
« Last Edit: December 16, 2009, 10:53:33 am by JFD62780 » Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #14 on: December 16, 2009, 01:08:32 pm »

this might be a lifesaver for me!
MyDefrag can optimize all the data on FAT disks, except for the directories. It's not really a life-and-death situation.... There is a small performance penalty because the directories are not optimized but it's not all that serious. My advise is to forget about the directories, and not take any risks.

Quote
Can this idea be converted to a MyD script, for a potentially new feature for a future version if evolved enough?
Well, perhaps in the future I can add something like this to the program itself. At the moment MyDefrag is totally solid and very safe. Using a trick like this is potentially dangerous, and in my opinion the benefits do not outweigh the risks. So at the moment I am not considering it.
Logged
Pages: [1] 2
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.5 | SMF © 2006-2008, Simple Machines LLC Valid XHTML 1.0! Valid CSS!