Using 3ds Max 2010 under Windows 10, when panning the viewport, the mouse cursor frequently glitches away and the viewport pans off to nowhere. I worked around the issue by making some small changes in the binary, based on the disassembled x86 code.
The implementation of a wrapping mouse cursor in Windows tends to be tricky, expecting and depending on the mouse messages following the call to change the mouse cursor position to match with the request.
Mouse cursor changes are sent to the application as a stream of Win32 messages. There is, however, no hard specification in the
WM_MOUSEMOVE documentation on the actual frequency of those messages, whether mouse messages may be collapsed to just the last message, or whether the mouse movement messages are sent at a lower frequency than the actual mouse sampling rate. A backlog of mouse messages might just form as well, if you’re queuing messages up asynchronously from another thread. This means that after you call the
SetCursorPos Win32 function to wrap an ongoing mouse movement, the next mouse message your application sees could be either an older message from near the current position that was queued up, a message that actually matches the requested coordinates, or a message with some other coordinates near the requested position in case the mouse has already moved away from there.
In general, implementing it does tend to end up being unreliable in certain cases, and requires quite a bit of special handling and checking to get completely right. It seems that 3ds Max 2010 depended on some specific behaviour that has since, apparently, silently changed in recent versions of Windows.
The fix, that I decided on here, is to somehow just disable the wrapping altogether. Stopping
SetCursorPos from being called. I tried this out very quickly by just changing the function name in the import table of
SetCaretPos, another arbitrary function that I simply picked from the Win32 documentation, to act as a no-op call. It has the same function prototype, but doesn’t do anything useful here and so should be relatively safe to call in this context.
With this change applied, and as I hoped for, the mouse cursor was no longer jumping across the screen while panning. However, because the 3ds Max panning routine still expected it to be at it’s requested position, the panning itself still jumped off each time. From it’s point of view the cursor was now being “moved”, from the position it attempted to set it to, to it’s current position.
Based on this, I would guess that the wrapping code expects the mouse position messages immediately following
SetCursorPos to always be a position relative to the requested one, but that in reality there could still be mouse movements from before the call queued up. This would cause the wrapping routine to be processed repeatedly, based on the outdated cursor position, panning the viewport even further away, until an acceptable cursor position is encountered.
Fixing this needs deeper investigation. Time to bring out the disassembler. We know now that we should be able to find what we need in the
3dsmax.exe binary, so let’s search for those
... CASE_00478F5C_PROC0002: cmp byte ptr [esi+0Ch],00h jnz L00478C85 mov ecx,[esp+54h] sub esp,00000008h mov eax,esp mov [eax],ecx mov edx,[esp+60h] mov ecx,[esp+4Ch] mov [eax+04h],edx mov eax,[esp+54h] push ebx push eax push ecx mov ecx,esi call SUB_L00478930 test al,al jz L00478F50 L00478C85: mov edx,[esi+50h] push edx call jmp_USER32.dll!SetCursor call [email protected]@YAHXZ mov edi,eax call jmp_maxutil.dll!?GetScreenH[email protected]@YAHXZ mov ecx,[esp+58h] mov edx,[esi+18h] mov ebp,eax mov eax,[esp+54h] mov [esp+14h],eax mov [esp+18h],ecx mov ecx,[edx+000002ACh] mov edx,[ecx] lea eax,[esp+14h] push eax mov eax,[edx+20h] call eax push eax call jmp_USER32.dll!ClientToScreen mov eax,[esp+18h] mov ecx,0000000Ah cmp eax,ecx jge L00478CFD mov edx,ebp sub edx,eax lea eax,[ebp-0Ah] sub edx,ecx add [esi+14h],edx mov [esp+18h],eax push eax L00478CE5: mov eax,[esp+18h] push eax call jmp_USER32.dll!SetCursorPos ...
The first occurrence appears in this part of the code, right after label
L00478CE5 which follows an interesting large block of code labeled
L00478C85. Preceding that is a small block at label
CASE_00478F5C_PROC0002 which has a conditional jump directly to
L00478C85, and also a conditional jump skipping over
In order to skip the entire
L00478C85 block, I changed the
jnz L00478C85 instruction to
nop nop, and
jz L00478F50 to
It turns out, when testing this change, that the entire panning feature gets disabled. So, the good news is, we’re at the right location in the code. On the other hand, we’re going to have to do some more work to get it working the way we expect it to.
The approach is relatively straightforward, let it enter the
L00478C85 label so the panning routine gets processed, but skip any small blocks where
SetCursorPos is being called, as there we can also reasonably expect the code to be setting up it’s internal wrapping calculations.
L00478CE5 label, which is nearer to the cursor call, could be skipped by an earlier instruction,
jge L00478CFD, so that’s the first one that I changed. This only requires changing the first byte of the instruction from
BE, which changes it to the short
... jmp L00478CFD ; Changed from jge mov edx,ebp sub edx,eax lea eax,[ebp-0Ah] sub edx,ecx add [esi+14h],edx mov [esp+18h],eax push eax L00478CE5: mov eax,[esp+18h] push eax call jmp_USER32.dll!SetCursorPos ; Avoid this CASE_00478F5C_PROC0003: movzx eax,[esi+0Ch] pop edi pop esi pop ebp pop ebx add esp,00000030h retn 0018h L00478CFD: add ebp,FFFFFFF6h cmp eax,ebp jle L00478D12 ; Jump here to avoid L00478CE5 mov edx,ecx sub edx,eax add [esi+14h],edx mov [esp+18h],ecx push ecx jmp L00478CE5 ; Don't jump here ...
However, within the
L00478CFD block we can see another jump back to
L00478CE5, which is a label we want to avoid. The
jle L00478D12 instruction in that block is therefore also changed to a
jmp, so we skip past it.
... L00478D12: mov edx,[esp+14h] cmp edx,ecx jge L00478D3F ; Jump to avoid mov ebx,edi sub ebx,edx sub ebx,ecx add [esi+10h],ebx lea ecx,[edi-0Ah] push eax push ecx mov [esp+1Ch],ecx call jmp_USER32.dll!SetCursorPos ; Avoid this movzx eax,[esi+0Ch] pop edi pop esi pop ebp pop ebx add esp,00000030h retn 0018h L00478D3F: add edi,FFFFFFF6h cmp edx,edi jle L00478D66 ; Jump to avoid mov edi,ecx push eax sub edi,edx add [esi+10h],edi push ecx mov [esp+1Ch],ecx call jmp_USER32.dll!SetCursorPos ; Avoid this movzx eax,[esi+0Ch] pop edi pop esi pop ebp pop ebx add esp,00000030h retn 0018h ...
The block at label
L00478D12 is calling
SetCursorPos as well, and can also be changed with a
jmp instruction to skip forward to
L00478D3F, which looks quite similar to the previous label, and in turn can be modified to always jump forward further to
Once in the
L00478D66 block, no further calls to
SetCursorPos show up, with the function eventually returning a bit further, indicating that we’ve taken care of all the calls in this function.
I tested these four changes, and they are right on point. Panning works as it should, with cursor wrapping completely disabled, and thus no more glitching. Success!
As a side note, there’s another bug occurring with 3ds Max 2010 under Windows 10. Context menus may end up getting their overlay stuck on the screen. Simply renaming
3dsmax.exe to something else resolves this issue. Possibly Windows, or the graphics driver, is implementing some modified behaviour for executables named
3dsmax.exe, which doesn’t play well with this particular version. It’s probably being done for some legacy performance or compatibility reasons.