Using 3ds Max 2010 under Windows 10, when panning the viewport, the mouse cursor frequently glitches away and the viewport pans off to nowhere. I worked around the issue by making some small changes in the binary, based on the disassembled x86 code.
The implementation of a wrapping mouse cursor in Windows tends to be tricky, expecting and depending on the mouse messages following the call to change the mouse cursor position to match with the request.
Mouse cursor changes are sent to the application as a stream of Win32 messages. There is, however, no hard specification in the WM_MOUSEMOVE documentation on the actual frequency of those messages, whether mouse messages may be collapsed to just the last message, or whether the mouse movement messages are sent at a lower frequency than the actual mouse sampling rate. A backlog of mouse messages might just form as well, if you’re queuing messages up asynchronously from another thread. This means that after you call the SetCursorPos Win32 function to wrap an ongoing mouse movement, the next mouse message your application sees could be either an older message from near the current position that was queued up, a message that actually matches the requested coordinates, or a message with some other coordinates near the requested position in case the mouse has already moved away from there.
In general, implementing it does tend to end up being unreliable in certain cases, and requires quite a bit of special handling and checking to get completely right. It seems that 3ds Max 2010 depended on some specific behaviour that has since, apparently, silently changed in recent versions of Windows.
The fix, that I decided on here, is to somehow just disable the wrapping altogether. Stopping SetCursorPos from being called. I tried this out very quickly by just changing the function name in the import table of 3dsmax.exe to SetCaretPos, another arbitrary function that I simply picked from the Win32 documentation, to act as a no-op call. It has the same function prototype, but doesn’t do anything useful here and so should be relatively safe to call in this context.
With this change applied, and as I hoped for, the mouse cursor was no longer jumping across the screen while panning. However, because the 3ds Max panning routine still expected it to be at it’s requested position, the panning itself still jumped off each time. From it’s point of view the cursor was now being “moved”, from the position it attempted to set it to, to it’s current position.
Based on this, I would guess that the wrapping code expects the mouse position messages immediately following SetCursorPos to always be a position relative to the requested one, but that in reality there could still be mouse movements from before the call queued up. This would cause the wrapping routine to be processed repeatedly, based on the outdated cursor position, panning the viewport even further away, until an acceptable cursor position is encountered.
Fixing this needs deeper investigation. Time to bring out the disassembler. We know now that we should be able to find what we need in the 3dsmax.exe binary, so let’s search for those SetCursorPos calls.
...
CASE_00478F5C_PROC0002:
cmp byte ptr [esi+0Ch],00h
jnz L00478C85
mov ecx,[esp+54h]
sub esp,00000008h
mov eax,esp
mov [eax],ecx
mov edx,[esp+60h]
mov ecx,[esp+4Ch]
mov [eax+04h],edx
mov eax,[esp+54h]
push ebx
push eax
push ecx
mov ecx,esi
call SUB_L00478930
test al,al
jz L00478F50
L00478C85:
mov edx,[esi+50h]
push edx
call jmp_USER32.dll!SetCursor
call jmp_maxutil.dll!?GetScreenWidth@@YAHXZ
mov edi,eax
call jmp_maxutil.dll!?GetScreenHeight@@YAHXZ
mov ecx,[esp+58h]
mov edx,[esi+18h]
mov ebp,eax
mov eax,[esp+54h]
mov [esp+14h],eax
mov [esp+18h],ecx
mov ecx,[edx+000002ACh]
mov edx,[ecx]
lea eax,[esp+14h]
push eax
mov eax,[edx+20h]
call eax
push eax
call jmp_USER32.dll!ClientToScreen
mov eax,[esp+18h]
mov ecx,0000000Ah
cmp eax,ecx
jge L00478CFD
mov edx,ebp
sub edx,eax
lea eax,[ebp-0Ah]
sub edx,ecx
add [esi+14h],edx
mov [esp+18h],eax
push eax
L00478CE5:
mov eax,[esp+18h]
push eax
call jmp_USER32.dll!SetCursorPos
...The first occurrence appears in this part of the code, right after label L00478CE5 which follows an interesting large block of code labeled L00478C85. Preceding that is a small block at label CASE_00478F5C_PROC0002 which has a conditional jump directly to L00478C85, and also a conditional jump skipping over L00478C85.
In order to skip the entire L00478C85 block, I changed the jnz L00478C85 instruction to nop nop, and jz L00478F50 to jmp L00478F50.
It turns out, when testing this change, that the entire panning feature gets disabled. So, the good news is, we’re at the right location in the code. On the other hand, we’re going to have to do some more work to get it working the way we expect it to.
The approach is relatively straightforward, let it enter the L00478C85 label so the panning routine gets processed, but skip any small blocks where SetCursorPos is being called, as there we can also reasonably expect the code to be setting up it’s internal wrapping calculations.
The L00478CE5 label, which is nearer to the cursor call, could be skipped by an earlier instruction, jge L00478CFD, so that’s the first one that I changed. This only requires changing the first byte of the instruction from 7D to BE, which changes it to the short jmp instruction.
...
jmp L00478CFD ; Changed from jge
mov edx,ebp
sub edx,eax
lea eax,[ebp-0Ah]
sub edx,ecx
add [esi+14h],edx
mov [esp+18h],eax
push eax
L00478CE5:
mov eax,[esp+18h]
push eax
call jmp_USER32.dll!SetCursorPos ; Avoid this
CASE_00478F5C_PROC0003:
movzx eax,[esi+0Ch]
pop edi
pop esi
pop ebp
pop ebx
add esp,00000030h
retn 0018h
L00478CFD:
add ebp,FFFFFFF6h
cmp eax,ebp
jle L00478D12 ; Jump here to avoid L00478CE5
mov edx,ecx
sub edx,eax
add [esi+14h],edx
mov [esp+18h],ecx
push ecx
jmp L00478CE5 ; Don't jump here
...However, within the L00478CFD block we can see another jump back to L00478CE5, which is a label we want to avoid. The jle L00478D12 instruction in that block is therefore also changed to a jmp, so we skip past it.
...
L00478D12:
mov edx,[esp+14h]
cmp edx,ecx
jge L00478D3F ; Jump to avoid
mov ebx,edi
sub ebx,edx
sub ebx,ecx
add [esi+10h],ebx
lea ecx,[edi-0Ah]
push eax
push ecx
mov [esp+1Ch],ecx
call jmp_USER32.dll!SetCursorPos ; Avoid this
movzx eax,[esi+0Ch]
pop edi
pop esi
pop ebp
pop ebx
add esp,00000030h
retn 0018h
L00478D3F:
add edi,FFFFFFF6h
cmp edx,edi
jle L00478D66 ; Jump to avoid
mov edi,ecx
push eax
sub edi,edx
add [esi+10h],edi
push ecx
mov [esp+1Ch],ecx
call jmp_USER32.dll!SetCursorPos ; Avoid this
movzx eax,[esi+0Ch]
pop edi
pop esi
pop ebp
pop ebx
add esp,00000030h
retn 0018h
...The block at label L00478D12 is calling SetCursorPos as well, and can also be changed with a jmp instruction to skip forward to L00478D3F, which looks quite similar to the previous label, and in turn can be modified to always jump forward further to L00478D66.
Once in the L00478D66 block, no further calls to SetCursorPos show up, with the function eventually returning a bit further, indicating that we’ve taken care of all the calls in this function.
I tested these four changes, and they are right on point. Panning works as it should, with cursor wrapping completely disabled, and thus no more glitching. Success!
As a side note, there’s another bug occurring with 3ds Max 2010 under Windows 10. Context menus may end up getting their overlay stuck on the screen. Simply renaming 3dsmax.exe to something else resolves this issue. Possibly Windows, or the graphics driver, is implementing some modified behaviour for executables named 3dsmax.exe, which doesn’t play well with this particular version. It’s probably being done for some legacy performance or compatibility reasons.
Recent Comments