Or add more RAM and switch to 64 bit Windows.
Or buy more RAM, buy 64-bit Windows, and watch it run out of memory in a few times longer time. Of course, never underestimate an expert.
My wild guess on source of the problem (without knowing or having confirmed any details, so likely wrong or at least oversimplified):
Step 1: MyDefrag sends the MS API command to move a file fragment.
Step 2: MS API fails to move the fragment because it did not succeed updating directory record due to its too long pathname.
Step 3: MyDefrag detects it as if the destination clusters are occupied by previously unknown file so it finds next suitable spot and tries again from Step 1 - ad infinitum. Probably some memory structures are created along with it, permanent or temporary, which gradually fill up the memory.
If it's the reason then of course it may depend on what file system and OS is used, MS defragmentation API is known to have different properties on different systems. Or the reason may be completely different, we would never know without investigating it properly and the only person who can do it is Jeroen who stated his position clearly:
The problem is caused by not enough memory in your computer, not by a bug in MyDefrag. And MyDefrag has no problem with paths longer than MAX_PATH.