Script linearity

[BloodySword]

I think the way MyDefrag executes the scripts should be improved, they should be executed like batchscripts in Windows.

Each VolumeSelect is for one Volume. There should be the possibility to add more than one VolumeSelect...VolumeEnd blocks.
Each VolumeSelect...VolumeEnd block should be worked out, then the next one is parsed. Same with FileSelect inside of VolumeSelect.
Only of the first FileSelect is worked out, the next block will be parsed (and code between the FileSelects).

Also I suggest to make the definitions simpler. Instead of VolumeSelect, VolumeEnd and FileSelect, FileEnd I suggest a structure as follows:

Volume
  [VolumeBoolean]
Actions
  DeleteJorunal(WipeNewUnallocatedSpace(Fast))
  //New IDEA!! FreeSpace wiping methods could be implemented to enhance the privacy safety.
  //1PassZero or Fast, 2PassRandomThenZero, 3PassDoD, 7PassSchneider would be great for it.
  //Here the new free unallocated space should be wiped before analyzing the volume.
  OptimizeNTFSExtends() //Shrinks or deletes the NTFS Extent-Information.
  Zone
    Select
      [FileSelect/FileBoolean]
    EndSelect
  Actions
    [FileActions]
  EndZone
 
  Zone [..]
  EndZone

  WipeFreeDiskSpace(Fast)
  //Sometimes it is a good Idea to wipe free space after the drive was optimized.
EndVolume

Volume
 [...next volume]
EndVolume

//Now after all Volumes are done, the code that will follow is executed like "WhenFinished()".
//This function should be thrown away:

Exit()

//The user can use Pause(), Shutdown(), Hibernate(), ....

What do you think of this sctructure? MyDefrag will then parse block for block, not the whole script as once.
This could also save a bit of RAM, since additional information of already processed files can be removed. Only
a list of the files that were processed is used to avoid double processing (ripping files out of their zone in the first place).

[Darlis]

As far as I know, MyDefrag first reads the script completely (twice!) but then does a "batch processing" with the parsed script. And imho, parsing does not really need that much time. Also, what would you do if there is a syntax error in the last zone and MyDefrag (because of the batch processing) tells you this minutes later, instead right at the beginning?

  Zone
    Select
      [FileSelect/FileBoolean]
    EndSelect
  Actions

The Select pice is redundant. Between "Zone" and "Action" is only a FileBoolean. Consequently, you also have to do this:

  Volume
    Select
      [VolumeBoolean]
    EndSelect
  Actions

[BloodySword]

There is no problem to validate the whole script before running. But due to the fact, that a file can only processed once, there must be a list in memory wich files should not be selected again in a next zone. If ther's the last zone, this list can be removed from memory completly wich makes space for the sorting process. This is important especially with volumes that are holding billion of files to avoid out of memory errors.

- After sorting/filling/defragmenting the analysis list can be converted in a simple list with files that should not be selected again, even if the selector passes.
- The list with the file sizes, cluster beginning and ending should then be deleted if a zone is fully processed. This will free a LOT of memory.

MyDefrag should read one volume block at once and the commands after every zone are only executed when the zone has finished completely. This also applies to volume blocks. After each volume block, an application could be run for exsample. After the last zone, there are commands that are today included in WhenFinished(...).

Yes, the select block should be the same both in Volume and Zone blocks. But I think it is better and simpler without the select block. It explains itself that there must be the selector for the volume(s) in the volume block and files or directories in the zone block.

[Darlis]

But due to the fact, that a file can only processed once, there must be a list in memory wich files should not be selected again in a next zone. If ther's the last zone, this list can be removed from memory completly wich makes space for the sorting process.

As long as there is a diskmap, you need a list with all the files and their status. Otherwise you won't see the details of the file, if you move the cursor over them.
This might only work with a commandline-version of MyDefrag. (and if MyDefrag does not use the processed list anymore)

MyDefrag should read one volume block at once and the commands after every zone are only executed when the zone has finished completely.

Doesn't MyDefrag do that already?

After each volume block, an application could be run for exsample. After the last zone, there are commands that are today included in WhenFinished(...).

Wouldn't a RunProgramm() (or something else) just before VolumeEnd do that? And why would you need exit, shutdown and reboot in the middle of a script?

But I think it is better and simpler without the select block.

My words.

[BloodySword]

As long as there is a diskmap, you need a list with all the files and their status. Otherwise you won't see the details of the file, if you move the cursor over them.
This might only work with a commandline-version of MyDefrag. (and if MyDefrag does not use the processed list anymore)

Perhaps the detailed list is important for this. But if the script fails with memory error, MyDefrag could be executed without these details and it runs through.

As far as I know, MyDefrag does not have a linear script execution. If you write RunProgram at the end of the script, it is not executed when the script is finished.
WhenFinished() is intended to fix it, but I think this is only a dirty solution

Exit, shutdown and reboot is intended only for the end of the script. With a linear execution, there could also be if statements possible in the future.

[Kasuha]

As long as there is a diskmap, you need a list with all the files and their status. Otherwise you won't see the details of the file, if you move the cursor over them.

That's not true. Diskmap is only displaying a bitmap with differently colored clusters. That does not require list of all files, all you need for it is a set of bitmaps or interval trees. If you place cursor over some place in the map MyDefrag could determine which cluster are you pointing at and get the information directly from the disk.
In fact, I believe MyDefrag will need to change to this approach if Jeroen wants it to be able to optimize really large disks.

[Darlis]

If you place cursor over some place in the map MyDefrag could determine which cluster are you pointing at and get the information directly from the disk.

I thought about that too, but if the information has to be always fetched from the disk and the disk is currently stressed by MyDefrag, it could be slow.

[Kasuha]

If you place cursor over some place in the map MyDefrag could determine which cluster are you pointing at and get the information directly from the disk.

I thought about that too, but if the information has to be always fetched from the disk and the disk is currently stressed by MyDefrag, it could be slow.

Yes, but main purpose of MyDefrag is optimizing your disk, not providing you information about where is each file placed. If you really need to know what the file is, you can wait a bit or pause the optimization.

It's about priorities. It is nice and convenient to get this information fast, but if this is about choice if this function or ability to sort large disks with millions of files, I'm behind large disks.

[Darlis]

If MyDefrag offers a feature like the diskmap and shows information about the displayed files, I personally do not want a constantly delayed display, just so that optimization finishes a bit faster. It's like with games: If I want high detailed graphics, I have to live a stuttering gameplay, or buy a better graphiccard.

I would't like a half/half solution. It would be better, if there was a cmd-version of MyDefrag, like there was one with JKDefrag.

[Kasuha]

Well... personally I don't believe MyDefrag is going to change its strategy of storing information about each file on the disk in memory ... or at least not anytime soon. It'd require major rework, file information is not used ony for displaying purposes, I doubt Jeroen would implement it just to be able to show which file is under the cursor.
I also highly doubt Jeroen is going to develop two parallel version of optimizer, one file-based and one diskmap-based.
I only believe using diskmaps instead of file list would make it possible to optimize much larger disks with much more files - using much less memory. Such disks will come over time but new technologies will come too, starting probably with SSD disks with completely different optimization needs. Personally I believe current disks are going to follow the fate of CRT monitors in a few years.

[BloodySword]

(...) but main purpose of MyDefrag is optimizing your disk, not providing you information about where is each file placed. If you really need to know what the file is, you can wait a bit or pause the optimization.

It's about priorities. It is nice and convenient to get this information fast, but if this is about choice if this function or ability to sort large disks with millions of files, I'm behind large disks.

I can fully agree to that.

But my intention was to discuss about the linear way of script execution.

I didn't mention this before, but I like the pastel-colors MyDefrag uses. Almoust all other programs are using colors from the 16 color EGA palette or something.

[Kasuha]

Um... well... to be honest, I don't see anything really wrong regarding linearity of current MyDefrag scripts. Most settings and commands are executed in the place where they lie in the script, only some don't and these usually have some reason behind it. Also, nobody prevents you to use multiple VolumeSelects in a single script, in fact one of my scripts contains that (completely different processing for C: and for D:)

There sure are things that could be improved on scripting language.

ExcludeFiles lying outside VolumeSelect is beyond my logic, it should be one of VolumeActions. It's nothing more than selecting a bunch of files and marking them as processed, the only difference to FileSelect with empty FileActions block is that it doesn't move the current position pointer.

AddGap should be made part of FileSelect structure. There's no point in having multiple AddGap() commands within FileSelect anyway and it is always executed last. Size of the zone should be calculated including the gap and files should be filled into it using this gap space.

MakeGap() should be changed to Seek(). That's what it is used for anyway.

It should be possible to process one disk twice. For some task there is no solution than to create two scripts and run them one after another.

It should be possible to inspect volume state and decide which of alternative sub-scripts implementing different strategies for processing the disk will be used.

It should be possible to inspect state of zone files and decide which of alternative sub-scripts implementing different strategies for processing the zone will be used.

Plus I'm keeping track of multiple improvements individual optimization commands could use, but that's not really about scripting language.

[jeroen]

I think the way MyDefrag executes the scripts should be improved, they should be executed like batchscripts in Windows.

Thanks for sharing your ideas, I appreciate it. Perhaps in the far future (version 5 or something like that) I will change MyDefrag into an interpreter and make it possible to jump around in scripts, but at the moment it is not possible within the infrastructure of the current MyDefrag sources.

Each VolumeSelect is for one Volume. There should be the possibility to add more than one VolumeSelect...VolumeEnd blocks.

You can already do that. To make things easier for the programmer MyDefrag automatically processes a volume only once. It also does this for files, when a file has been placed in a zone it is automatically not selected for next zones.

Also I suggest to make the definitions simpler. Instead of VolumeSelect, VolumeEnd and FileSelect, FileEnd I suggest a structure as follows:

I have chosen the commandwords so they clearly show their relation to each other and to the overall block. Your proposal is not bad, but I feel the commandwords are less clear. For example, you are using the words "Actions" for both the volume and the file actions. I feel this will be confusing to the scriptwriter and can cause hard to find bugs. Another thing would of course be that if I were to change these commandwords now that scriptwriters all over the world would have to change all their scripts.

But due to the fact, that a file can only processed once, there must be a list in memory wich files should not be selected again in a next zone.

There are several reasons why the whole list of files must be kept in memory and changing the scripting will not make it possible to use less memory. One of the most important reasons (there are more) is that MyDefrag must be able to find the (name of the) file that uses a particular cluster on disk. This information is not available from Windows. When you know the name of a file you can ask Windows which clusters it occupies, but that is exactly the other way around.

If you write RunProgram at the end of the script, it is not executed when the script is finished.

Yes it is executed. Only not if you terminate a script yourself, or if it crashes. Windows .bat scripts work exactly the same.

[jeroen]

ExcludeFiles lying outside VolumeSelect is beyond my logic, it should be one of VolumeActions. It's nothing more than selecting a bunch of files and marking them as processed, the only difference to FileSelect with empty FileActions block is that it doesn't move the current position pointer.

Files that are selected with ExcludeFiles will never be moved. Files that are NOT selected by FileSelect can still be moved, they are assumed to be for next zones.

By having ExcludeFiles outside the VolumeSelect it has become possible to globally define unmovable files, on all disks.

AddGap should be made part of FileSelect structure. There's no point in having multiple AddGap() commands within FileSelect anyway and it is always executed last. Size of the zone should be calculated including the gap and files should be filled into it using this gap space.

AddGap does more than just allocating more space for the zone. And I feel it is not logical to specify the size of a zone in the FileSelect.

MakeGap() should be changed to Seek(). That's what it is used for anyway.

I agree that the MakeGap() commandword should be changed, it's function has changed and the word no longer reflects the function. I will consider "seek", but haven't decided yet.

It should be possible to process one disk twice. For some task there is no solution than to create two scripts and run them one after another.

Sorry, but processing disks only once is a feature and I don't want to change it. I don't know what tasks you are thinking of but it should be solved differently, so everything can be done in a single MyDefrag run.

It should be possible to inspect volume state and decide which of alternative sub-scripts implementing different strategies for processing the disk will be used.

Already possible. You can place multiple VolumeSelect-VolumeEnd blocks in a single script.

It should be possible to inspect state of zone files and decide which of alternative sub-scripts implementing different strategies for processing the zone will be used.

It's on my wishlist.

[Kasuha]

Files that are selected with ExcludeFiles will never be moved. Files that are NOT selected by FileSelect can still be moved, they are assumed to be for next zones.

ExcludeFiles("*.brb")

does effectively the same as

FileSelect
  FileName("*.brb")
FileActions
  AddGap(ZoneBegin)
FileEnd

The only difference is, first colors these files red, second green.

AddGap does more than just allocating more space for the zone.

Extends the zone and eventually vacates added space? Or do you mean it can be "abused" for seeking around like MakeGap is? As long as we already have MakeGap for jumping around, I believe AddGap could be restricted to extending the zone space only. And it's frustrating to see that some file did not get moved to its zone because it collided with a file right behind the zone which gets moved away the next second while vacating the behind-the-zone gap.

And I feel it is not logical to specify the size of a zone in the FileSelect.

But that's what you do. All defragmenting and filling commands and AddGap are already inside FileSelect.

Sorry, but processing disks only once is a feature and I don't want to change it. I don't know what tasks you are thinking of but it should be solved differently, so everything can be done in a single MyDefrag run.

640 kB should be enough for everyone?

Already possible. You can place multiple VolumeSelect-VolumeEnd blocks in a single script.

Not possible. I can run AnalyzeOnly script, see how bad does the disk look and decide whether I will use monthly, weekly, or daily optimization script. I think the script should be able to do that without need for the user to decide.