Change the way NTFS writes new files?

[Axel Mertes]

Hi all,

I found JKDefrag today, after having tested dozens of other apps deeply, freeware and expensive commercial defrag tools.

I do a lot of computer graphics and film and video production. Therefor I need to handle large amount of files, mostly single frame sequences (myPic0001.tga, myPic0002.tga, .. , myPic2387.tga and so on) as well as lots of large streaming video files (.avi, .mov. etc.), some of them in then 20+ GByte per file range. Some apps split those files into 4 GByte tiles, to stay compatible with FAT32. Also the partition are really huge, the largest currently an 6 TByte RAID 6 (actually 8 * 1 TByte drives) in one single partition using 4 KByte sectors (instead of 512 bytes, to overcome the usual 2 TByte barrier on partitions).

Anyhow, one of the biggest issues I need to fight is that when I capture fresh video, e.g. in uncompressed HD format, I am forced to have the harddisk

a) completely space defragged, to have a contiguous large chunk of free disk space for capture and at the same time no small gaps that could be filled - this way the capture will not be interrupted by jumping between free disk fragments

b) hopefully have that free chunk in a fast place on the disk.

Most defrag apps fail, as they do the space defrag with all files from the beginning of the disk, leaving the big gap at the end.

Then there is the issue of defragging such a large partition, which can take a few days if not done on a daily basis.

I use for instance the Diskeeper Undelete, to not delete the deleted files immediately. Instead I keep them hold in the background and clean up that recovery bin just before the nightly defrag, so that small fragments are just free'ed up before the space defrag process will close the gaps again. Next morning the drive should be clean with the least effort thinkable.

But sometimes it fails, just because the time required wasn't enough to clear the small gaps. Sometimes I end up filling the gaps with blank data to get the free area contiguous, but thats quite stupid.

The real issue is the way NTFS actually allocates the new blocks for the files.

I wish I could force NTFS to allocate fresh disk space ALWAYS from the biggest free contiguous disk space area.
Even better would be if I could control the behaviour by the extension of the file created!

Imagine you could manage to write e.g. AVI, MOV, TGA, TIFF, JPEG etc. to the largest free area, while .DOC, .TXT etc. will be forced to go to the small fragment blocks. That would perfectly reflect the real needs.

Is that thinkable for anybody here? 

I wish Mark Russinvich would be reading this 



Another issue is that I need to play back single frame sequences FAST. So at best they are in pure sequential order, each file being defragged and each file after each other in alphabetical order. As I use to render new files on multiple machines, e.g. using 2D or 3D software, they do not necessarily arrive in sequential order.

So defragging them to become a sequential ordered series of files seems to be possible in JK Defrag, right? 

I already consider sending out a defrag command line automatically after rendering, so that whenever a sequences has been created, it will be brought into a sequential order. As windows will probably fill the free zone from the beginning, I suggest having the option to sort the files to the end of the free zone, so that that zone eats up from two ends, while the start is being free'ed up about as much as is being used at the end.

I will see what JK Defrag can really do. Right now my biggest concern is the speed of the process. It feels quite slow with its analysis phases, compared to the O&O Defrag and Diskeeper and contig.exe. But maybe I am not doing fair justice now, not having tested it enough yet.

At least it sounds very scriptable and the announcement of v4 having custome zones and scripting interface sounds like the way to go.

JK, if you like, drop me an email, I have many more infos to share.

Cheers,
Axel

[Axel Mertes]

One more thing that I wish to see:

Queueing!

What about the ability to send commands to a defrag queue (with all know options as if you were running it as usual), but the queue gets worked out step by step.

So if I render 5 different sequences, it will first defrag sequence 1, then 2 and so on. Not all simultaneously, as if it would happen when just starting the dos command.

I guess writing a basic queue would be manageable for myself, but I think it makes sense as a common, general feature. And Jeroen seems to be fit in programming 

Cheers,
Axel

[jeroen]

b) hopefully have that free chunk in a fast place on the disk.

The easiest way to create a big free chunk at the beginning of the disk is with JkDefrag's "-f" option.

Then there is the issue of defragging such a large partition, which can take a few days if not done on a daily basis.

Moving a lot of data will take a lot of time, it's as simple as that. JkDefrag's "-a 2" and "-a 3" options will do the least possible data movement.

Is that thinkable for anybody here? 

As far as I know there is no way to influence how Windows decides where to place a file. You could consider repartitioning your harddisk to make a separate partition for your temporary files.

So defragging them to become a sequential ordered series of files seems to be possible in JK Defrag, right? 

Yes, JkDefrag can defragment (or optimize) only a set of files, or a folder, or whatever. For example "jkdefrag c:/temp/*".

I suggest having the option to sort the files to the end of the free zone

See the "-a 6" option.

[Axel Mertes]

Hi Jeroen,

b) hopefully have that free chunk in a fast place on the disk.

The easiest way to create a big free chunk at the beginning of the disk is with JkDefrag's "-f" option.

Thanks! I'll give that a try.

Then there is the issue of defragging such a large partition, which can take a few days if not done on a daily basis.

Moving a lot of data will take a lot of time, it's as simple as that. JkDefrag's "-a 2" and "-a 3" options will do the least possible data movement.

Is there any info on how the algorythm works then?

My point was not only actually moving the data taking a long time (which is expectable and understandable), but the analysis and sorting phase felt comparably slower than the professional apps I used. This is in no way meant badly, its just the experience I had. Maybe your sorting is more complex etc. I don't know.

In fact the applying of the "mask" took really a long time! Maybe the real issue is the screen refesh?!

It appears that you draw really small pixel previews per block. So - imagine a 6 TByte partition in one piece, with millions of files on it - the re-drawing of this view of the drive blocks takes a serious amount of time. Maybe it can be switched to update only every x seconds or the like. I don't know if you ever had the chance to deal with this size of partitions and amount of files in the past. So the slowness I experienced *could* be caused by the visual feedback, that eats the time here. And sorting this amount of files and blocks is another game too.

When I start O&O with stealth mode it starts moving files immediately. Space mode takes maybe like two or three minutes on the same big volume and starts moving files. That is the difference I am talking about. JK Defrag took like more than 15-30 minutes just for sorting files and applying masks...

Honestly, I really really like your software and will try to assist here to solve those last points I have currently less enthusiastic comments on.

I just have the comparision between various apps, mainly Diskeeper, O&O Defrag (which did an internal build just for me due to the 4KB sector size of my RAID subsystems - they couldn't handle it before, as they did not 100% rely on the API) and PerfectDisk, as well as contig.exe and Eyeon Fusion (which has a clever defrag build in, based on contig.exe source codes, but able to defrag a file sequence to stay "serially in line"). So each app has pro's and con's, but none of them has "all". So O&O can defrag alphabetically, but only entire disks... Think of this with 5 TByte used disk space and you'll quickly drop the idea... JK Defrag looks very promising in this context.

So again, JK Defrag has an amazing functionality, and I want to push it to the limits in terms of processing speed. This means mainly the sorting/analysis phase, as the rest relies on the API anyway.

One more point to add:

Many commercial defraggers detect which partitions come from which physical drive and allow to defrag one partition per physical drive at a time, but surely several independent partition on independent physical drives at once. This calls for the queue idea. As I wrote before, the queue would be nice, and given you work on scripting anyway for v4, the queue concept might help you out on generating multi-core support here. Just as a thought.

Is that thinkable for anybody here? 

As far as I know there is no way to influence how Windows decides where to place a file. You could consider repartitioning your harddisk to make a separate partition for your temporary files.

Well, that sounds more easy than it is, as many apps have very strict directory path structures etc. which can not always be freely modified. Further, multiple partition on a single RAID for instance allow for less fragmentation per partition, yes. But it will kill any performance of the RAIDs. When you have 2 or more partition on a single physical drive (or RAID), you force the heads to seek around. All your current optimizations regarding zoning etc. will be wasted immediately. Its a "no-go", "it's as simple as that"

So defragging them to become a sequential ordered series of files seems to be possible in JK Defrag, right? 

Yes, JkDefrag can defragment (or optimize) only a set of files, or a folder, or whatever. For example "jkdefrag c:/temp/*".

Questions:

I want to defragment a file sequence within a specific directory path, and there are multiple file sequences within the same directory, how can I control that ONE specific file sequence will be defragmented, so that afterwards the files are sequentially, alphabetically on the harddrive?

If I pass on several individual file names, it will treat each file name as a single new defrag job, right?

So I need to pass on some kind of wildcard pattern, like:

MyFirstSequence_?.tif

which would apply to MyFirstSequence_0000.tif ... MyFirstSequence_9999.tif then, right?

If there are other sequences like MySecondSequence_?.tga or MyThirdSequence_?.jpg within the same directory, they will not be affected at all, correctly?

Is there any information that tells me afterwards in how many sequential chunks the now defragmented sequence has been stored, (1 chunk = perfect defragmented, completely gap free and alphabetically sequential)?
How?

I suggest having the option to sort the files to the end of the free zone

See the "-a 6" option.

I'll look into that. Looking forward to your v4 version, which sounds like it could be exactly for what I was looking...

Looking forward to your valued comments!

Thanks in advance,
Axel

PS: In which area of Sri Lanka do you stay? I've been there in 2000 and travelled nearly all around the country (bypassing some of the Tiger controlled area, of course, unfortunately).

[jeroen]

Is there any info on how the algorythm works then?

For which "-a" mode? I have never written detailed documentation about the algorithms. You can look in the sources, if you want, they are heavily commented. Defragmentation will look for a gap above the file and then move the file into that gap. If there is no gap big enough to hold the entire file then it will take the biggest gaps available and move sections of the file into those gaps, but only if it reduces the number of fragments.

the analysis and sorting phase felt comparably slower than the professional apps I used.

Well, it's as fast as I have been able to make it. Remember that JkDefrag is freeware, I am not a big company and I do not have a whole department of programmers at my disposal. There's just little old me.... JkDefrag is just a small defragger filling the gap between the free Windows defragger and the commercial defraggers. It is not fair to expect/ask for the same performance/features that you are getting from commercial defraggers.

Maybe it can be switch to update only every x seconds or the like.

That's exactly what I am doing, only once every 300 milliseconds.

If I pass on several individual file names, it will treat each file name as a single new defrag job, right?

Yes, a single file name is a single job, including full analysis of the entire disk. If you specify a mask (for example "MyFirstSequence_????.tif") then all the matching files will be processed together as a single job, only 1 analysis. Other files will be treated as unmovable, they will not be touched. Please note that they will not be moved out of the way to make room for the selected files.

Is there any information that tells me afterwards in how many sequential chunks the now defragmented sequence has been stored, (1 chunk = perfect defragmented, completely gap free and alphabetically sequential)?

The only information available from JkDefrag is what you find in your JkDefrag.log file. You can control the amount of information with the "-d" commandline option. There is no reporting of the chunks that you describe, though.

PS: In which area of Sri Lanka do you stay?

In Ratmalana (Colombo).

[Axel Mertes]

Is there any info on how the algorythm works then?

For which "-a" mode? I have never written detailed documentation about the algorithms. You can look in the sources, if you want, they are heavily commented. Defragmentation will look for a gap above the file and then move the file into that gap. If there is no gap big enough to hold the entire file then it will take the biggest gaps available and move sections of the file into those gaps, but only if it reduces the number of fragments.

OK, thats what I hoped for. Makes perfect sense to me Thumbs up!
 

the analysis and sorting phase felt comparably slower than the professional apps I used.

Well, it's as fast as I have been able to make it. Remember that JkDefrag is freeware, I am not a big company and I do not have a whole department of programmers at my disposal. There's just little old me.... JkDefrag is just a small defragger filling the gap between the free Windows defragger and the commercial defraggers. It is not fair to expect/ask for the same performance/features that you are getting from commercial defraggers.

No offense here!

I am absolutely aware that you are working on a freeware solution, so I can not expect that price/performance ratio from a commercial product, because equation 0/performance is nothing, and performance/0 is illegal

The amazing point is: You react and support, while the "big guys" do not really care about us that much. The story behind O&O and me was, that I pointed out their software is unable to work with todays large scale server & RAID solutions. They didn't wanted that bean to spread, so they worked with me to solve the issue (which is now solved), but not much beyond that. I explained them the exact same things I explained here, but they do not even understand the point of "what sense makes it to defrag - context sensitive - only a single directory or file". Well, as I never got what sense it makes to sort a whold drive alphabetically... (which makes a lot of sense if done for a few files or directories...).

Anyhow, I will need to test JK Defrag a bit more and stress it out, then I have a better idea of its performance.

Maybe it can be switch to update only every x seconds or the like.

That's exactly what I am doing, only once every 300 milliseconds.

OK, any chance you make that amount of time "user adjustable" in a new build?
I really don't care about a 3 times per second update if it may be slow. I often feel that I can see it drawing the pixels, maybe because my partitions are so excessive?

If I pass on several individual file names, it will treat each file name as a single new defrag job, right?

Yes, a single file name is a single job, including full analysis of the entire disk. If you specify a mask (for example "MyFirstSequence_????.tif") then all the matching files will be processed together as a single job, only 1 analysis. Other files will be treated as unmovable, they will not be touched. Please note that they will not be moved out of the way to make room for the selected files.

Hey, thats perfectly as I hoped for. I can make a very intelligent file handling from this. See, if I defragment each individual file sequence into ONE large chunk of data, that get presevered by other defrag operations, it will be like if I had a few large AVI or MOV files instead of hundrets of thousands of single frames. The total overall fragmentation will be significantly reduced when this is performed regularly, especially after rendering of each sequence.

Is there any information that tells me afterwards in how many sequential chunks the now defragmented sequence has been stored, (1 chunk = perfect defragmented, completely gap free and alphabetically sequential)?

The only information available from JkDefrag is what you find in your JkDefrag.log file. You can control the amount of information with the "-d" commandline option. There is no reporting of the chunks that you describe, though.

OK, I'll have a look into that. Contig.exe printed out only the info how many fragments a file had before, and after its defragmentation. I would be glad if you put that on your list for an updated v4 then, because its a helpful information if you start doing iterative defragmentation.

PS: In which area of Sri Lanka do you stay?

In Ratmalana (Colombo).

I don't know this exact place, but we clearly have been in Colombo. At that time some serious bombings where just under go, so we had a lot of street stopping, military all over. Anyhow, its such a nice country and I wish I can go and have a visit in a couple of years. I am still in occasional contact with our guide who lives there with his family. Has the country recovered somewhat from the Tsunami? We had been in many places that got totally destroyed. I learned from a lot of projects to help out, but as usual, public reports get very calm after a while. The only infos I get are from the guide, but he is just in his place...

Thanks,
Axel

[Axel Mertes]

Hi Jeroen,

I have now made some tests.

I tried to defrag a file sequence of a few hundret TGA files, which are placed in a single directory on a large, mildly fragemented 1.5 TByte partition, which is about 90+% full. For that reason I copied the JKDefrag.exe into that directory, opened a DOS box, cd'ed to the directory containing the TGA sequence AND the JKDefrag.exe. Then I started JKDefrag.exe with the following command:

R:\
CD R:\TEMP\JKDefragTest
jkdefrag.exe -a 7 -d 6 *.tga

As a result it took from 19:12 until roughly 21:15 o'clock (so like 2.5 hours) for analysis and sorting phase 1, then it started moving files. All TGA files from the entire partition, oh no! So it didn't care about the fact it was installed and started from the sub path "R:\Temp\JKDefragTest\" at all, instead it filtered out ALL TGA files on a roughly 800,000 files partition... And there are quite a huge amount of them, most files I presume...

So I stopped it at 21:36 and restarted with the following commands:

R:\
CD R:\TEMP\JKDefragTest
jkdefrag.exe -a 7 -d 6 -u DisableDefaults R:\TEMP\JKDefragTest\*.tga

hopefully minimizing the files to this particular subdirectory.

It looks like the phase 1 will again take the same time. Now, at 22:17 it says 51% of phase 1.

What I do not understand is why you analyze the entire partition first, substract all "unwanted" files with the mask and then start with the remaining ones. OK, when the aim is to defrag mainly complete partitions, this makes sense, as you would sort out just a few unwanted files and then potentially defrag thousands of other files.

But in this case (and thats what I really like on the theoretical concept of JK Defrag) I want to defrag a specific series of files, within a specific subdirectory. So would it be possible to gather information just for the "wanted" files, and not first for all minus the unwanted files?

If you see a defrag tool analysing 2.5 hourse to defrag a few hundret frames then in a few minutes, I remain with the saying the analysis is a bit slow . As other apps do this really quick I wonder what can be done about speed here. Eyeon Fusion for example, based on contig.exe, does its "analysis" in nearly no time and defragments then a file sequence sequentially alphabetically ordered.

So my idea would be to gather the info on wanted files only, then on free disk blocks, and start the defragmentation.

Why do you need to analyse all files at all?

Or is your zone calculation the issue here, so that you want free disk blocks within a specific zone?
If that could be it, how can I switch it off?

Oh, you are sooo close, sooooo close to what I want

Looking forward to your feedback,
Axel

[Axel Mertes]

Some more questions:

When JK Defrag is aimed at defragmenting a sequence that is already perfectly defragmented, how quickly will it detect that?

When starting JK Defrag from DOS box, it opens up, runs, and afterwards stays with its window open. It will not close by itself, and it prevents starting another instance from JK Defrag until its manually closed. I would like to run it automatically (queued) via a batch file (.bat), so I need the app to close after having finished its defrag.

It appears that two instances of JKDefrag can not run at the same time, which makes defragmenting physical independent partitions (on different physical drives) impossible.

The screen refresh is extremely time consuming on very large partitions, I can see how it takes a few seconds, maybe 20 or 30, until the full drive map is drawn. Is that normal?
Can it be switched off completely?
Or every n seconds (not every 300ms)?

Axel

[SledgeHammer]

What I do not understand is why you analyze the entire partition first, substract all "unwanted" files with the mask and then start with the remaining ones. OK, when the aim is to defrag mainly complete partitions, this makes sense, as you would sort out just a few unwanted files and then potentially defrag thousands of other files.

If you can get your hands on an older version (e.g. 3.16), try it with analysis-only (commandline -a1 R:\TEMP\JKDefragTest\*.tga). It should be much faster.

When starting JK Defrag from DOS box, it opens up, runs, and afterwards stays with its window open. It will not close by itself, and it prevents starting another instance from JK Defrag until its manually closed. I would like to run it automatically (queued) via a batch file (.bat), so I need the app to close after having finished its defrag.

Add the commandline-parameter "-q"

It appears that two instances of JKDefrag can not run at the same time, which makes defragmenting physical independent partitions (on different physical drives) impossible.

JKdefrag.exe does not start if it detects an other process called jkdefrag.exe
Simple Solution: copy jkdefrag.exe to jkdefrag1.exe (for first disk) and jkdefrag2.exe (for second disk).

[jeroen]

OK, any chance you make that amount of time "user adjustable" in a new build?

Version 4 of JkDefrag will (ultimately) have several choices for the display, and lot's of settings everywhere. The commandline of version 3 is complex enough as it is, that's why version 4 will have a small scripting language. It makes it possible to add scores of settings and options.

I really don't care about a 3 times per second update if it may be slow. I often feel that I can see it drawing the pixels, maybe because my partitions are so excessive?

Perhaps another application is causing JkDefrag to completely refresh it's window very often in your case. Normal JkDefrag behaviour is to only draw the file it is working on, which is very fast. Perhaps you can get better speed if you run JkDefrag minimized or maximized. Or use the commandline version (JkDefragCmd.exe) instead of the Windows version.

if I defragment each individual file sequence into ONE large chunk of data

Have you considered organizing your files into virtual disks? There is lot's of free software around for that (see for example Wikipedia: Disk image emulator). You create a virtual disk and mount it on a folder, then move all the little files into that folder. JkDefrag can then sort the files inside the virtual disk (possible for most virtual disk drivers). Once sorted the files will stay sorted. On the real disk the virtual disk will appear as a single huge file, which JkDefrag can defragment just like any other file.

What I do not understand is why you analyze the entire partition first

It's a requirement of the analyzer, it must analyze the entire MFT before it can construct the full path name of every file. Early versions of JkDefrag analyzed only the selected files, like SledgeHammer says. For most users the analysis is quite fast, normal speed for 800.000 files is 3 minutes or so. Do you have a background virus scanner?

When JK Defrag is aimed at defragmenting a sequence that is already perfectly defragmented, how quickly will it detect that?

Unfragmented files are skipped and take zero time.

When starting JK Defrag from DOS box, it opens up, runs, and afterwards stays with its window open.

The commandline version of JkDefrag (JkdefragCmd.exe) will automatically exit. The Windows version will stay open, unless you specify the "-q" option.

It appears that two instances of JKDefrag can not run at the same time

Yes, JkDefrag has a build-in protection for that, to protect budget computers from overheating.