Intro Download and install Frequently Asked Questions Tips and tricks

Homepage







© J.C. Kessels 2009
MyDefrag Forum
October 25, 2014, 04:27:12 am *
Welcome, Guest. Please login or register.

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: unmovealbe files and dirs  (Read 1566 times)
molecule
JkDefrag Junior
**
Posts: 6


View Profile
« on: July 19, 2011, 04:00:12 am »

hi,

I'm trying to gather some basics on unmovealbes.

As I understand it, MyDefrag (or some other defrag front end) gives the standard defrag API a command to move [a sector?|a path\file?] to [some sector], and the API return a refusal.

MyDefrag adds that [file/dir? | sector?] to an unmovable list.

Q1: What are the usual tags that cause the API to refuse?

Is there typical registry entry?  A lot of my unmovable dirs can be deleted with no complaint as to file in use, or other similar distraint.  The programs that own the files or dirs are not running.  The only thing that I can find that is possibly "tagging" them is a registry key under microsoft installer ...

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Folders]

Can a registry key be set so as to "tag" a file or dir so that the defrag API will return a refusal?

Q2: the MyDefrag.dat file appears to be a binary type.  What tool do I need to read it?  I'm using w2k-sp4, on a FAT32 primary.

Does the unmovable list give a full path to the unmovalbe dirs and folders?

Q3: Is the standard MS defrag API just a lite version of the full version API used by Diskeeper?  Would it make any difference to have MyDefrag use the full version API of Diskeeper?  Has anyone done a file compare?

Q4: Under 98se, Norton Speed Disk would move FAT32 directories, to beginning of disk, end of disk or whatever.  A directory is really just another file, the content of which is a simple structured table, a movable-FAT table, i.e. a simple structured database with file names, attributes, sectors, etc.  It's no different that any other database file.  If you can move a database file, you can move a directory -- admittedly you have to open the file's parent directory (one level up) and change the sector address in its FAT table for the subdirectory filename, to show the new defragged address for the subdir.  The parent dir would then refer any disk/file services API to the correct sector for the mini-FAT file of the subdir filename.

It's no different from moving a file.  when a file is moved, the parent dir needs to have its mini-FAT file db modified to show the new defragged sectors that locate the filename.  they are nearly identical operations.

thanks ...
Logged
jonib
JkDefrag Hero
*****
Posts: 822


View Profile
« Reply #1 on: July 19, 2011, 02:54:01 pm »

Q2: the MyDefrag.dat file appears to be a binary type.  What tool do I need to read it?
The MyDefrag.dat file is a simple textfile and can be opened with Notepad or any other texteditor.

Quote
Does the unmovable list give a full path to the unmovalbe dirs and folders?
Yes

Quote
Q4: Under 98se, Norton Speed Disk would move FAT32 directories, to beginning of disk, end of disk or whatever.  A directory is really just another file, the content of which is a simple structured table, a movable-FAT table, i.e. a simple structured database with file names, attributes, sectors, etc.  It's no different that any other database file.  If you can move a database file, you can move a directory -- admittedly you have to open the file's parent directory (one level up) and change the sector address in its FAT table for the subdirectory filename, to show the new defragged address for the subdir.  The parent dir would then refer any disk/file services API to the correct sector for the mini-FAT file of the subdir filename.

It's no different from moving a file.  when a file is moved, the parent dir needs to have its mini-FAT file db modified to show the new defragged sectors that locate the filename.  they are nearly identical operations.
Moving a dir is somewhat different to moving a single file within the FAT file system, a file only has one reference to it but a dir can be pointed to by other dirs and point to other dirs and files, So my guess is that Microsoft decided it's not safe to move dirs in a multitasking environment when using the FAT filesystem.

jonib
Logged

Darlis
JkDefrag Hero
*****
Posts: 1771


View Profile WWW
« Reply #2 on: July 19, 2011, 07:37:54 pm »

Q1: What are the usual tags that cause the API to refuse?
It can be anything from simply being locked by the program (or driver) that uses it up to a Defragmentation API limitation. Afaik "tagging" by a registry key is not an option.

Q3: Is the standard MS defrag API just a lite version of the full version API used by Diskeeper?  Would it make any difference to have MyDefrag use the full version API of Diskeeper?  Has anyone done a file compare?
I know that the Windows Defragmenter is a lite version of Diskeeper, but I'm not sure about the API (i doubt it). But if it was, what would you expect from the full version of it? Support for outdated file systems?
Logged

Need help creating a script? Try MyDefrag Script Creator.
ff_mfg
JkDefrag Hero
*****
Posts: 51


View Profile
« Reply #3 on: July 19, 2011, 07:46:11 pm »

Q1: What are the usual tags that cause the API to refuse?

Is there typical registry entry?  A lot of my unmovable dirs can be deleted with no complaint as to file in use, or other similar distraint.  The programs that own the files or dirs are not running.  The only thing that I can find that is possibly "tagging" them is a registry key under microsoft installer ...

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Folders]

Can a registry key be set so as to "tag" a file or dir so that the defrag API will return a refusal?
It has nothing to do with registry. There are various file locks in Windows. Certain types of these locks make usual files unmovable while they are locked.
Stopping the program doesn't always releases all used files. For example, some DLLs might still be locked by the system.

Q3: Is the standard MS defrag API just a lite version of the full version API used by Diskeeper?  Would it make any difference to have MyDefrag use the full version API of Diskeeper?  Has anyone done a file compare?
No no no. Built-in defrag (which was based on simplified Diskeeper, yes), commercial Diskeeper, almost all other defragmenters for Win2K/XP and up use Microsoft Defragmentation API.
As I understand, the API itself is pretty simple, it's just a bunch of control codes, mostly FSCTL_GET_VOLUME_BITMAP, FSCTL_GET_RETRIEVAL_POINTERS and FSCTL_MOVE_FILE (there's basic info at http://msdn.microsoft.com/en-us/library/aa363911(v=vs.85).aspx). What you can do with them - that's defragmenter software. And as we all know, it can be quite different.
Logged
molecule
JkDefrag Junior
**
Posts: 6


View Profile
« Reply #4 on: July 21, 2011, 06:46:21 am »

it just doesn't make any sense ...

there are many screenshots on this forum, showing hundreds of "red dots scattered across free space" ... just like mine!

what IT engineer could possibly write an installer (either from application side or from OS side) that would install a directory out at the farthest reaches of a partition, and then claim it as a sector fixitude?  If for example, when a competitor's program such as wordperfect or open office needs access to a small .ini file, what the OS installer does is put a parent dir at the beginning of a partition, the first-child subdir1 at the end, the second-child (grandchild) subdir2 back at the beginning, the third-child (greatgrandchild) subdir3 at the end, the fourth back at the beginning, the fifth at the end, and finally, a cluster address for access to the file at the beginning of the hdd.  that's six full head swings across the disk, with maximal seek times, just to access for a small ini ... that will make any competitor's software feel "mushy" and lagged, in addition to wearing out the head motors of the hdd.  That's exactly what some installer did to some dirs for Open Office.

it makes no difference that other files and services can call another file, or that the database in a parent subdir can point to the cluster in a child subdir.  other files can call a file, and can be called by a file. and they are movable ... in a PE environment, MyDefrag can even move the pagefile.  moving a file or a dir(file) means changing a cluster entry in the fat table of the parent dir.  no big deal.  files and subdirs are the same in that they are pointed to by a parent dir, if the pointer in the parent dir can be changed for one, it can be changed for the other.  all sector addressing still has to go in through the front door of the hdd to get to a file's address.  it's ridiculuous to even suggest that there can be multiple APIs that can have power of appointment over FAT file or cluster assignments, or to suggest that since files can call other files, and other files can call files, that therefore files can't be moved, and the cluster pointer in their parent files (dir fat table) cannot ever be changed.  A subdir is just another file, with file/dir names and cluster pointers.  if only one API that has power of appointment over cluster assignments, then EVERY access to a file or subdir-file must come in through the front door.  it's called FAT-walking.  the fact that files can be moved, while they call other files, and can be called by other files is NOT the issue.

The issue is, what service is responsible for keeping the list of unmovable files.  It's not regenerated each time an application or service starts up.  Totally secondary, is how did these critters get created in the first place, way out there in partition freespace, and to a lesser extent who wrote the disk services API so as to maximize back and forth head seeks when building a heirachy of subdirs?

My thought is, if microsoft IT engineers are so clueless about sector assignments in a simple FAT file system (they aren't -- they are maybe just little 'itlers, meaning IT'lers haha), so as to assign clusters for new files or dirs as they come up that are way out into the farthest reaches of freespace (instead of the next available freespace), with parent-child-grandchild dirs alternating close-in and way-out, thereby maximizing head seek times to access or run a file [for ... a competitor's program?], then how can MS be trusted to know what they are doing with the more complex and secretive NTFS.

everybody keeps saying the dat file (at c:\Program Files\MyDefrag v4.3.1\MyDefrag.dat) has the list of unmovables.

Mine doesn't.  Do I need to change one of the debug settings in the master script file, Settings.MyD.  One of the first things MyDefrag does when it starts a defrag is offer the Message ("Loading list of unmovable files").  If it is loading a list of unmovable files, can someone please point me to where is it getting the list?  It says it is loading the list, not regenerating it.  It's ridiculuous to suggest that everytime some tiny little service has to startup, the list has to be regenerated all over again.

It happens so quickly, it's obvious that MyDefrag is not frog-walking the entire fat system (which includes from hundreds to thousands of mini-FATS called subdir files that the file services API puts all over the place).  So, I suspect the list is already maintained by the system and is thus already available from some other service.  In other words, MyDefrag is not alone in perceiving these files as unmovable.  Services other than MyDefrag would also have need to know.

Is there a way to output that list ... it's not in my .dat file ... Can I write my own MyDefrag script to do it?  There are easily several hundred (several thousand?) of these little IT'ler fixitudes littered across my primary partition, top to bottom.  From my last run (from a PE environment, like Bart's PE) says "E#:#\#P#A#G#E#F#I#L#E#.#S#Y#S#"  So even in a PE environment, that was booted from CD, MyDefrag found the list.  The MyDefrag.log lists 25 largest files ... but no list of unmovables.

I'm trying to imagine how it happens, not just to me but to many many others as well?  how can 80% of a primary FAT partition be wide open and ununsed ... and some engineer writes an API that assigns next file/dir clusters that are way out there in freespace, and then locks them down.  If that is allowed, he gives a bad name to ALL engineers.  I'm trying to imagine the kind of person that engineer would have to be, to even think of doing that ... how does he even sleep at night?  No one can call that person an honest person if he uses special skills, that most of us do not have, to do such a devious and hurtful thing as slowing down programs and causing permature deaths to hdds.

I love MyDefrag!!  And I have tremendous respect, for its openness, its community, and its honesty!

The monthly script is the BEST deep defragger I have ever seen, and I've tried a few.  Keep up the good work!

Booted from a PE/LiveXP/BartsPE environment, MyDefrag will even move a pagefile (which Mark Russinovich's PageDefrag post-MS takeover frags, and puts out there at the last clusters on the hdd.) to the middle of a disk.  well done!!  If MyDefrag can figure out how to move a $pagefile$, I suspect the cluster pointers in other files can be tweaked as well.
« Last Edit: July 21, 2011, 02:22:06 pm by molecule » Logged
Darlis
JkDefrag Hero
*****
Posts: 1771


View Profile WWW
« Reply #5 on: July 21, 2011, 06:44:10 pm »

what IT engineer could possibly write an installer (either from application side or from OS side) that would install a directory out at the farthest reaches of a partition, and then claim it as a sector fixitude?
The people who write these installers (and 95% of all other programs) don't decide to which clusters the files are written to. They only tell Windows to write/copy file A to directory D. That's it. The rest is all up to Windows. Again: It's the Windows Defragmentation API the refuses to move directories on FAT volumes, not the programs or installers.

moving a file or a dir(file) means changing a cluster entry in the fat table of the parent dir.  no big deal.
The big deal in case of MyDefrag is that the Windows Defragmentation API refuses to move directories on FAT volumes and since Jeroen has always refused to implement operations that can't be done using said API, it's very unlikely that such a feature will find it's way into MyDefrag.

The issue is, what service is responsible for keeping the list of unmovable files.
None. There is no such thing. Files are getting locked by the disk driver, the Defragmentaion API or the programs themselves.

everybody keeps saying the dat file (at c:\Program Files\MyDefrag v4.3.1\MyDefrag.dat) has the list of unmovables. (...) So, I suspect the list is already maintained by the system and is thus already available from some other service.  In other words, MyDefrag is not alone in perceiving these files as unmovable.  Services other than MyDefrag would also have need to know.
The list maintained by MyDefrag and MyDefrag alone. There is no service that maintains such a file or where MyDefrag could get such a list. MyDefrag will add files to that list if it fails to move a file, i.e. it has to try to move a file before it can find out if it is unmovable or not. So, in order to generate such a file you have to let MyDefrag move every file on the volume.

Note that there is a know bug where MyDefrag would empty the list if you closed it before it finishes processing the volume.
MyDefrag does output this list in the MyDefrag.log file, but not when using the Analyze Only script.

Booted from a PE/LiveXP/BartsPE environment, MyDefrag will even move a pagefile (which Mark Russinovich's PageDefrag post-MS takeover frags, and puts out there at the last clusters on the hdd.) to the middle of a disk.  well done!!  If MyDefrag can figure out how to move a $pagefile$, I suspect the cluster pointers in other files can be tweaked as well.
MyDefrag moves the pagefile like any other file: With the Windows Defragmention API. In a PE environment, the page file on the offline Windows system partition is no longer write locked by that system. PageDefrag is only able to move that during boot-time, i.e. at a point where Windows is not using it. The directories on a FAT volume are (again) a whole different story.

I have a question too: Why do you keep using the FAT file system? NTFS is more secure and MyDefrag can move directories on it.
Logged

Need help creating a script? Try MyDefrag Script Creator.
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7233



View Profile WWW
« Reply #6 on: July 22, 2011, 07:08:12 am »

everybody keeps saying the dat file (at c:\Program Files\MyDefrag v4.3.1\MyDefrag.dat) has the list of unmovables.
Mine doesn't.
The MyDefrag.dat file is not a list of unmovable files. It is a list of files that could not be moved by MyDefrag on the last run. If MyDefrag did not try to move a particular (unmovable) file, for example because it was already defragmented, then the file will not be listed. Directories on a FAT disk are never listed, because MyDefrag knows they are always unmovable.

Quote
One of the first things MyDefrag does when it starts a defrag is offer the Message ("Loading list of unmovable files"). If it is loading a list of unmovable files, can someone please point me to where is it getting the list?
It's the MyDefrag.dat file. Each and every listed file is tested by trying to move a small portion of it, to allow for files that were temporarily unmovable on the previous run.

Quote
I suspect the list is already maintained by the system and is thus already available from some other service.
As far as I know there is no such list anywhere in the system. The only way to find out if a file is unmovable is by actually trying to move it. There are several reasons why a file can be (reported as) unmovable by the Microsoft Defragmentation API, not just because a file is locked by another program.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.5 | SMF © 2006-2008, Simple Machines LLC Valid XHTML 1.0! Valid CSS!