Scripting improvement/simplification

[TeDDy]

q1
Deleted from the "filename array" in memory not disk.
q2
yes and archives newly acessed, sorted in first pass are re-moved to the end, leaving gaps at the begining
q3
after setbeginofzone, all future placements can be done AFTER THIS cluster, never before it. Placing files (first rule)with setbeginofzone=x>0 will leave empty space at the begin of drive. No more files will be placed BEFORE x in next rules of script. That's the problem. Zone declaration for fileplacement from endofdrive to beginofdrive and file processing from begin to end, CONSIDERING declared selections.
When put files to begin, to KNOW what files must go to end, and if need to vacate them cause take place, not to spread them around BUT to place it AT THEIR PLACE(whorst case, 2 moves this way)

I can make a more detailed explanation if you wish that.

If you make me a detailed description of program internal behaviour from the start to the end(steps done, rules manipulation, order of actions) I can figure where to make some inversions in execution order or split actions and you can try to adjust them to see the results.

[TeDDy]

 Thank you to give me the same response on diferent posts.
 Perhaps I deserve this.
 I am talking with the wind, as I see.
 I see the scripting language is complex.
 I am not talking about programming language or number calculations.
 But it DOES STRING MANIPULATION by tell to program what to do.
 It passes declared zones to program, one by one and the program executes them one by one.
 Understanding just a zone at the time will move WITHOUT PURPOSE files who WILL FALL IN OTHER RULES instead to place them by this rules at their place DIRECTLY.
 It's time and drive consuming.
 It understands select files and setbeginofzone but instead of reading other zones too, it places files by this declaration.
 Last sector for this zone= first sector for the next zone.
 This way you can't declare zones in reverse order. Why? I think this is a BIG MISS.
 Is it so difficult to implement few keywords for zones(like z1..zn) so the user can declare them FROM n to 1 and the program AFTER reading ALL zones to process them from 1 to n ? This way, when DO z1 he knows what NOT DO because will be DONE in zi ?
 Before other jobs, MyDefrag analyses the drive, load filenames and their attributes. After that executes the first rule/zone.
 Why not read ALL the rules and do the calculations BEFORE moving files? This way selecting files define labels in the big array in tle memory. After the labeling, files are moved directli(or almost) to their places.
 I know, why to stress the CPU when we can stress the HDD?
 Why to make more calculations BEFORE when files will be placed as the user wish after more jumpings?
 I am sure you know why better than me.
 I try to make you understand a way to simplify the scripts by avoiding repeated selection of files because the program is reading and procesing SINGLE zone each time, IGNORING whats declared in the other zones.
Not giving to files PRECISE zone placement by reading all declared zones, will move files without purpose before they are selected by some rule.
This is frustating for the user who need to retype almost same NOT conditions(n-1 times for n zones) because the VERY COMPLEX LANGUAGE and COMPILER :
-CAN'T parse all declared zones in the script and sort founded files by this rules AT THE BEGINING of file movements.
-It works with zones in the declared order, ignoring what was done BEFORE and what need to be done after that
-CAN'T understand(missing syntax) mask for files to exclude in other zones.

If someone wish to optimize his drive in 8 phisical zones, no free space between them, he must:
z1 acess<10day AND NOT files for zones 4,5,6,7,8
z2 10day<acess<20day AND NOT files for zones 4,5,6,7,8
z3 20day<acess<30day AND NOT files for zones 4,5,6,7,8
z4 mask4 AND NOT files for zones 5,6,7,8
z5 mask5 AND NOT files for zones 6,7,8
z6 mask6 AND NOT files for zones 7,8
z7 mask7 AND NOT files for zone 8
z8 mask8

I am not stealing your work, I am not teaching you what to do. I try to let you know what I see that can be done to your program to be more simple and useful for the drives, OUR DRIVES too.
I hope I can use your program in the future to make the job. I am SURE in the future, after loong user feedback and testing you will make it SHAREWARE but before that, by testing your work and making PERTINENT suggestions, we can use it to make things better for our drives.

I am sure I am not wrong with what I say.
I am still use SpeedDisk for w9x to defrag my drives monthly.
I try to help people to get a proggy who does almost everything SpeedDisk does.
I was askyng for long time what SpeedDisk does, and I have investigate deep the results of his placement.
It moves files THE WAY I TELL YOU. Some one did it before you. And did it well !
Why reinvent the circle instead of using it?
I understand you NEVER use SpeedDisk. Try it, tweak it. It's simple and FAST. I can give it to you to test it if you have an computer old enought to work with windows 95,98,ME. You will see I am right.
THIS modus operandi want users to get from a defrag program.
I AM VERY SURE.
Please read about it, test it and AFTER THAT blaim me to be a fool.
Else, tell me IT CAN'T BE DONE instead of I DON'T WANT/KNOW to do it.
I am ready to tell you what and almost how SpeedDisk do the job. I've see the results. They are the best.
It reads drive, reads rules, sort placement in memory and AFTER THAT begin to move files. In rare cases, some files are moved TWICE(when they are switched, taking eachother's place). This is not an impediment for SpeedDisk to place files DIRECTLY in 8-9 zones on drive.
This file? need to be placed at end? place it. next file to the middle? place it, and so on.

I write this loong posts on your forum not to force you to do what I want, but what I think it's best for you and us, of course.
You realise(I hope) I do this for a more NON PROFIT cause than you maybe.
I want a defrag to do the best possible job.
I found someone who has started to build an opensource defrag and improve it.
Great, the guy has a forum too!
Lets test his work.
Ooh, it works...but. Okay, I see what's missing, let's tell him/them.
Sorry, program does not that, thank you for suggestion, maybe in the future, next version.
You can modify the sources if want dear Teddy.
Okay, but not moving unloked files?
Ahh, v4 will do it!
Okay. waiting and learning Visual Studio for that.
Sorry, no future releases fo 3.x and no sources with fixed unmovables for 4.x.
Okay again, since I can't achieve same number of moved files with 3.x sources and working with dead version is futile, I try to make light so YOU CAN DO IT.
But it seems, somehow I can't speak in a way to make you understand me.
I don't have your experience with high level programming language like VisualStudio.
I am just a FreePascal, making just console applications.
I try to learn Studio JUST to modify the latest 3.x sources and this way to let you see the point.
For you it is possible to try to implement what I tell you.
I am sure you will see the diference and the increase of quality and speed.
I am sure you can do it better/easier than me, even IF I HAD THE SOURCES.
But telling me it's not possible make me SAD.
I though the forum is for improvements so you improve your work(wich in the future will give you money) and in time we benefit of improved program.
If it's a tell me to do what I want/know to do forum, I've wasted my time for nothing.
Reboot to 98se and defrag my fat32 drives.

Please correct me if I say something WRONG.
If I make you angry, If my in vain advices are not welcome, please EXCLUDE me from your forum.
I'll understand I come to the wrong place.
You are in command.

But if NOT, try to see the benefits of my advices. I give them with ALL MY HEART.
From 12 years I work with low level hardware/software, with "hacker like" approach when observing behaviour of programs.
Don't be affraid, I am a NonProfit activist/lover, I am a perfectionist about 40 years old. Beside my payed job as constructions eingineer, ALL my activities and free time is dedicated to make a better world, without profit for me or others. I just help people how they can be better for themselves.

I will post this on the 2 topics where you give me the copy/paste answer.
You can delete it.
You can close all my topics.
You can exclude me from your forum.

If NOBODY like my advices, then ignore my post and please, DO NOT REPLY, NOBODY !!!
Then I stop to disturb YOU and let you go in other direction, even if I think I know where is the right direction.

Depending on your reactions/reply's, this can be my LAST POST, EVER, or a new beginning for defragmentation program.
UP
TO
YOU !!!

[poutnik]

I believe your posts are made by your the best invention and intention. ( I am 43 :-)   )

But, unfortunately, they are quite long, not formatted to easy read and time consuming to evaluate.
Try short post with idea formulated in big clarity, the chance would be bigger.

I think Jeroen has in these days full hands in debugging and user feedback about the beta version,
and th MyDefrag is his spare and enjoy time project.

It is not good time now to make suggestions for features or big project chaages.
Maybe later.

[TeDDy]

Short posts can't express the entire idea and placement explanations are long, for sure.
Short messages can be done, without misunderstandings or truncate the ideas on a chat session.
Not the case on forums.
I will try to be as shortter as I can, in the future if SOMEBODY NEEDS SOME WORD FROM ME !

[poutnik]

Short posts can't express the entire idea and placement explanations are long, for sure.
Short messages can be done, without misunderstandings or truncate the ideas on a chat session.
Not the case on forums.
I will try to be as shortter as I can, in the future if SOMEBODY NEEDS SOME WORD FROM ME !

Look, Teddy,

There are many ways to express complex ideas. I have read the post and idea is good at least to evaluate.

Why not to provide a short summury?
Why not to separate paragraphs with subideas?
Why not to use more senseful character bolding and formatting ?

Your text is hard to read, not because of content, but form.

Significant part of the post are your feelings not essential to idea understanding.
One sentence or short separate paragraph about them is enough.

Your post looks rather as chat record than as forum article.
There is no need to capital shouting - it shows something about you, not about the idea.

[jimbo]

Having read through Teddy's posts a couple of times, I'd like to say that there are some interesting thoughts in there, but also a couple of flawed assumptions.

Assumption 1) if a file matches the filter for zone x, and also zone y, it will be processed twice.

This is incorrect. If a file matches a zone, it will be processed in that zone, and never again in later zones. You can imagine that every zone in ascending numeric order includes an implicit "and not already processed" filter. Both the documentation and my own empirical testing have confirmed as far as I can tell that this is the way it works, so your main assumption about multiple file moves is incorrect.

Multiple file moves can happen, if a file is well down the disk from it's final destination, it could be pushed up into later free gaps several times until it finally settles into a zone. However, this is only likely to be a common occurance when you are first performing a majorly different sort/zoning of your disk, which, generally you will do very rarely, so optimizing the performance of this at the potentinal expense of the general case of daily defrags would be a net bad thing to do.


Assumption 2) it is possible if you preprocess all the sorting to determine the exact final positions of the zone.

Sadly, this is also incorrect. One thing that it is not possible under the windows API to determine is whether a file will be moveable or not, until you try to move it. This means that if there are large immovable files closer to the front of the disk than the sorting would put them, this will invalidate the zone position calculations. This is exactly when both MyDefrag and its predecessor JKDefrag worked on a zone-by-zone basis.

Jeroen Kessels knows a lot about the issues and pitfalls of various techniques for file sorting and optimization algorithms, which he has learned by experience of writing this excellent application, and there are many subtle but excellent reasons why the program has been coded to work the way it does. This doesn't mean that you won't come up with a brilliant idea on how it can be improved, it just means that there could well be reasons why your brilliant idea isn't as brilliant as it first appears.


So, Teddy, would it be possible for you to re-phrase the rest of your suggestions in a clearer way, taking into considerations the issues I raised here with your assumptions, as I for one am definitely interested in hearing your ideas.

[Lexar]

Teddy, I really don't see your point. With a cleverly written script, he file selection for one zone does not overwrap that for another zone. The following is part of the script that I am using in an attempt to keep MyDefrag running in the background.

Code:

 FileSelect
    Directory(yes)
  FileActions
    FastFill()
  FileEnd

  FileSelect
    Size(0,1048576) and LastChange("","1 DAYS AGO")
  FileActions
    Defragment()
  FileEnd

  FileSelect
    Size(1048577,4194304) and LastChange("","4 DAYS AGO")
  FileActions
    Defragment(ChunkSize(2))
  FileEnd

  FileSelect
    Size(4194305,16777216) and LastChange("","16 DAYS AGO")
  FileActions
    Defragment(ChunkSize(2))
  FileEnd

  FileSelect
    Size(16777217,67108864) and LastChange("","64 DAYS AGO")
  FileActions
    Defragment(ChunkSize(2))
  FileEnd

Files selected with Size(0,1048576) and LastChange("","1 DAYS AGO") will not be selected again later in the operation.

[poutnik]

 FileSelect
    Size(0,1048576) and LastChange("","1 DAYS AGO")
  FileActions
    Defragment()
  FileEnd

Hi,

Are you sure you wanted to use LastChange("","1 DAYS AGO") (modified more than 1 day ago )

and not    LastChange("1 days ago","") (modified in last 24 hours  )  ?

[TeDDy]

poutnik ,
 I'll try to make an ordered explanation.
Also I will write in notepad instead of the forum's form.
__________
jimbo ,
About what you say it is my Assumption 1: if a rar archive wich must go to the end( just by extension) by second rule is modified today and the first rule has no explicit exclude filter for rar, this particular file is moved to begin of drive and never to the end (the second rule will ignore it?)?
___________

I make exagerate assumptions just to demonstrate how many conditions must be writed in selections for the begin of drive to be sure the desired files really go to the end.

I try to suggest the importance to have at least one of those extra options:

 - files to be placed at the end are declared before fastoptimize selections and file placement begins with moving those archives to the end

 - declare selections at the begin (mask1=*.mp3,*.wav; mask2=*.avi,*.mov; mask3=*.rar,*.zip;) and then referred in the exclude (all and not(mask1 or mask3)).
   This way we can make precise selections and repeating same exclude mask is simplified.
   Imagine you wish to move to end about 100 filetypes and the remaining files sorted in 2-3 zones.
   Add/modify this selections become nightmare.

 - declare selections with predefined destination (keyword) can avoid processing those files by optimisation rules for acessed files.
   Ex: spacehog=.,.,. filetoend=.,.,.
   If those selections are <>nul then masked files, when vacated from fast zones are directly to zone_n and zone_n-2 (freespace between is zone_n-1 ).

 - a) SetBeginOfZone(Absolute(6000)) then b) SetBeginOfZone(Absolute(1000)
   will not move a_files from 6000 to end and then move b_files from 1000.
   The same way, if first filemove action was to put files to end, remaining files can't be optimized at begin of drive.


 Those 2 zones(end begin middle) will never interfere. Why not process them in any order?

Why to carry the huge amount of endfiles to exclude in each rule dedicated to first zones?

Files not selected to process are jumping on drive until, maybe the last rule select it.

Dedicated selection can instruct program to vacate by moving files to dedicated(precalculated) zone.

[TeDDy]

Teddy, I really don't see your point. With a cleverly written script, he file selection for one zone does not overwrap that for another zone. The following is part of the script that I am using in an attempt to keep MyDefrag running in the background.

Files selected with Size(0,1048576) and LastChange("","1 DAYS AGO") will not be selected again later in the operation.

True, but if not explicit excluded, files changed or acessed if lastchange specification will be optimised even if by later rule I want to put at the end.

If someone wish to optimize his drive in 8 phisical zones, no free space between them, he must:
z1 acess<10day AND NOT files for zones 4,5,6,7,8
z2 10day<acess<20day AND NOT files for zones 4,5,6,7,8
z3 20day<acess<30day AND NOT files for zones 4,5,6,7,8
z4 mask4 AND NOT files for zones 5,6,7,8
z5 mask5 AND NOT files for zones 6,7,8
z6 mask6 AND NOT files for zones 7,8
z7 mask7 AND NOT files for zone 8
z8 mask8

Now you got my point?

[jeroen]

I'll try to make an ordered explanation.

Let me begin by saying that I admire your stamina and dedication. I'm sure not many people would have put so much time into explaining their ideas as you have.

- files to be placed at the end are declared before fastoptimize selections and file placement begins with moving those archives to the end

Allow me to rephrase your idea, correct me if I am wrong: You are asking for the possibility in a script to first specify a zone (or zones) that go to the end of the disk, before specifying the zones that go to the beginning of the disk. You are asking for this because in your case it will simplify the file selection criteria a bit. I'm not sure if I stated this before, but I already have this idea on the wishlist.

Personally I prefer the current paradigm, where the script starts at the beginning of the disk and files are selected that go into each zone. What is left over automatically goes to the end of the disk. This makes the script relatively easy to understand because there is a direct mapping relationship with the disk.

declare selections at the begin (mask1=*.mp3,*.wav; mask2=*.avi,*.mov; mask3=*.rar,*.zip;) and then referred in the exclude (all and not(mask1 or mask3)).

I will think about it, but in my opinion text manipulation does not belong in a configuration script language. It belongs in a programming language. I think it is better that people use their favorite programming language to generate MyDefrag scripts, than to bloat MyDefrag scripts with things like number calculations, string manipulations, if-then, while, goto, and many other things commonly found in programming languages.

declare selections with predefined destination (keyword) can avoid processing those files by optimisation rules for acessed files.

JkDefrag v3 had pre-defined zones, and this generated a lot of criticism and discussion. That's why I switched to scripting. Not every script will have 8 zones, and not everybody (including me) thinks it is a good idea to put the spacehogs at the very end of the disk.

a) SetBeginOfZone(Absolute(6000)) then b) SetBeginOfZone(Absolute(1000) will not move a_files from 6000 to end and then move b_files from 1000.

Yes, already on the wishlist.

Those 2 zones(end begin middle) will never interfere. Why not process them in any order?

On a disk with little free space they will interfere.

[TeDDy]

q1
 My time is zero compared with the time you put to make this program better or answer to our questions.
 Making what we think is right motivate the time spent to do it.
 As long my ideas are not rejected, I can explain and rafinate them.
 I think they are good enough as they come from long observations of the SpeedDisk's behaviour.

q2
 Declared before just to be known by next rules.
 I explained how rules can be tricked for SortByLastAccess without precise selection for EOD&SH files.
 To avoid "mistakes" the EOD & SH rules must be stronger than other selections, and to free the fast zone they should have bigger priority.

 Please take a look here http://www.defraggler.com
 Go to settings/options/defrag and enable the move large files at end of drive, then select one drive who has this kind of files just to see they are moved as first action and after this processing the rest of files.
 Just an example because is far, far less flexible as MyDefrag.

 The current paradigm, without some fix from you, will force the repetition of "and not(*lot of masks*)" to be present in each of (n-1)rules. Before, I explained how I think the fix can be done (in script declared once or in external file).

q3
 The scripting has keywords. 'Select' is one of them. 'Select1', 'Select2' can be labels for later selections.
 Or to make selections inline with + - ?
 Just an idea!

 My favourite programming language is unable to read/write unicode files, will make only commandline, no gui dos like programs and for sure can't solve the problems for mydefrag or usual users (it will restrict the flexibility).
 I can make just an program who read some ini files to produce another script to be used by mydefrag.
 This is the longer way.
 The short way is for Mydefrag to have built-in gui for make the config for each and save it in the root of each drive;
 Or to have a simplified way to make rules in the script as I try so hard to explain.
 
 True, my extra program can read hundred of selections and repeat them in the generated myd script. Bad idea because the lines can be much longer the mydefrag can read perhaps.
 Or
 The script wich has /*ENDOFDRIVE*/ to read the lines from 'eod.txt' and similar for /*SPACEHOGS*/ with 'SH.txt'. Better I say because this reuse the masks in every generated script by one or two macro/keywords in script body.

q4
 Predefined/prioritary zones are very good. They separate the files in fast/slow zones and by deleting the macro/keyword from script this zones will not be used.

q5
 Slow/Fast Optimize scripts has 5 zones not counting freespace zones. Pretty close to 8 wich has the endofdrive zone.

 Spacehogs go just before the big free space. Endofdrive files go after the big free space, at the very end of drive.

q6
 True if little free space. But zones can be processed in any order as mydefrag really can write anywhere in the free space on drive.
 If first rule places files at some offset of the drive, future rules will not be able to place files before this offset point. Only after or in strange positions!

[poutnik]

I will think about it, but in my opinion text manipulation does not belong in a configuration script language. It belongs in a programming language. I think it is better that people use their favorite programming language to generate MyDefrag scripts, than to bloat MyDefrag scripts with things like number calculations, string manipulations, if-then, while, goto, and many other things commonly found in programming languages.

Command line and bat/cmd are still superior to Mydefrag script in some aspects.

As far as CMD and BAT have since old good DOS a macro feature,
the script IMHO need to have its generally usable counterpart.

BTW, one can use even CMD to generate MyD scripts.