Patching the mouse wrapping glitch in 3ds Max 2010 on Windows 10

Using 3ds Max 2010 under Windows 10, when panning the viewport, the mouse cursor frequently glitches away and the viewport pans off to nowhere. I worked around the issue by making some small changes in the binary, based on the disassembled x86 code.

The implementation of a wrapping mouse cursor in Windows tends to be tricky, expecting and depending on the mouse messages following the call to change the mouse cursor position to match with the request.

Mouse cursor changes are sent to the application as a stream of Win32 messages. There is, however, no hard specification in the WM_MOUSEMOVE documentation on the actual frequency of those messages, whether mouse messages may be collapsed to just the last message, or whether the mouse movement messages are sent at a lower frequency than the actual mouse sampling rate. A backlog of mouse messages might just form as well, if you’re queuing messages up asynchronously from another thread. This means that after you call the SetCursorPos Win32 function to wrap an ongoing mouse movement, the next mouse message your application sees could be either an older message from near the current position that was queued up, a message that actually matches the requested coordinates, or a message with some other coordinates near the requested position in case the mouse has already moved away from there.

In general, implementing it does tend to end up being unreliable in certain cases, and requires quite a bit of special handling and checking to get completely right. It seems that 3ds Max 2010 depended on some specific behaviour that has since, apparently, silently changed in recent versions of Windows.

The fix, that I decided on here, is to somehow just disable the wrapping altogether. Stopping SetCursorPos from being called. I tried this out very quickly by just changing the function name in the import table of 3dsmax.exe to SetCaretPos, another arbitrary function that I simply picked from the Win32 documentation, to act as a no-op call. It has the same function prototype, but doesn’t do anything useful here and so should be relatively safe to call in this context.

With this change applied, and as I hoped for, the mouse cursor was no longer jumping across the screen while panning. However, because the 3ds Max panning routine still expected it to be at it’s requested position, the panning itself still jumped off each time. From it’s point of view the cursor was now being “moved”, from the position it attempted to set it to, to it’s current position.

Based on this, I would guess that the wrapping code expects the mouse position messages immediately following SetCursorPos to always be a position relative to the requested one, but that in reality there could still be mouse movements from before the call queued up. This would cause the wrapping routine to be processed repeatedly, based on the outdated cursor position, panning the viewport even further away, until an acceptable cursor position is encountered.

Fixing this needs deeper investigation. Time to bring out the disassembler. We know now that we should be able to find what we need in the 3dsmax.exe binary, so let’s search for those SetCursorPos calls.

...
CASE_00478F5C_PROC0002:
  		cmp	byte ptr [esi+0Ch],00h
  		jnz	L00478C85
  		mov	ecx,[esp+54h]
  		sub	esp,00000008h
  		mov	eax,esp
  		mov	[eax],ecx
  		mov	edx,[esp+60h]
  		mov	ecx,[esp+4Ch]
  		mov	[eax+04h],edx
  		mov	eax,[esp+54h]
  		push	ebx
  		push	eax
  		push	ecx
  		mov	ecx,esi
  		call	SUB_L00478930
  		test	al,al
  		jz 	L00478F50
 L00478C85:
  		mov	edx,[esi+50h]
  		push	edx
  		call	jmp_USER32.dll!SetCursor
  		call	jmp_maxutil.dll!?GetScreenWidth@@YAHXZ
  		mov	edi,eax
  		call	jmp_maxutil.dll!?GetScreenHeight@@YAHXZ
  		mov	ecx,[esp+58h]
  		mov	edx,[esi+18h]
  		mov	ebp,eax
  		mov	eax,[esp+54h]
  		mov	[esp+14h],eax
  		mov	[esp+18h],ecx
  		mov	ecx,[edx+000002ACh]
  		mov	edx,[ecx]
  		lea	eax,[esp+14h]
  		push	eax
  		mov	eax,[edx+20h]
  		call	eax
  		push	eax
  		call	jmp_USER32.dll!ClientToScreen
  		mov	eax,[esp+18h]
  		mov	ecx,0000000Ah
  		cmp	eax,ecx
  		jge	L00478CFD
  		mov	edx,ebp
  		sub	edx,eax
  		lea	eax,[ebp-0Ah]
  		sub	edx,ecx
  		add	[esi+14h],edx
  		mov	[esp+18h],eax
  		push	eax
 L00478CE5:
  		mov	eax,[esp+18h]
  		push	eax
  		call	jmp_USER32.dll!SetCursorPos
...

The first occurrence appears in this part of the code, right after label L00478CE5 which follows an interesting large block of code labeled L00478C85. Preceding that is a small block at label CASE_00478F5C_PROC0002 which has a conditional jump directly to L00478C85, and also a conditional jump skipping over L00478C85.

In order to skip the entire L00478C85 block, I changed the jnz L00478C85 instruction to nop nop, and jz L00478F50 to jmp L00478F50.

It turns out, when testing this change, that the entire panning feature gets disabled. So, the good news is, we’re at the right location in the code. On the other hand, we’re going to have to do some more work to get it working the way we expect it to.

The approach is relatively straightforward, let it enter the L00478C85 label so the panning routine gets processed, but skip any small blocks where SetCursorPos is being called, as there we can also reasonably expect the code to be setting up it’s internal wrapping calculations.

The L00478CE5 label, which is nearer to the cursor call, could be skipped by an earlier instruction, jge L00478CFD, so that’s the first one that I changed. This only requires changing the first byte of the instruction from 7D to BE, which changes it to the short jmp instruction.

 ...
 		jmp	L00478CFD ; Changed from jge
  		mov	edx,ebp
  		sub	edx,eax
  		lea	eax,[ebp-0Ah]
  		sub	edx,ecx
  		add	[esi+14h],edx
  		mov	[esp+18h],eax
  		push	eax
 L00478CE5:
  		mov	eax,[esp+18h]
  		push	eax
  		call	jmp_USER32.dll!SetCursorPos ; Avoid this
 CASE_00478F5C_PROC0003:
  		movzx	eax,[esi+0Ch]
  		pop	edi
  		pop	esi
  		pop	ebp
  		pop	ebx
  		add	esp,00000030h
  		retn	0018h
 L00478CFD:
  		add	ebp,FFFFFFF6h
  		cmp	eax,ebp
  		jle	L00478D12 ; Jump here to avoid L00478CE5
  		mov	edx,ecx
  		sub	edx,eax
  		add	[esi+14h],edx
  		mov	[esp+18h],ecx
  		push	ecx
  		jmp	L00478CE5 ; Don't jump here
...

However, within the L00478CFD block we can see another jump back to L00478CE5, which is a label we want to avoid. The jle L00478D12 instruction in that block is therefore also changed to a jmp, so we skip past it.

...
 L00478D12:
  		mov	edx,[esp+14h]
  		cmp	edx,ecx
  		jge	L00478D3F ; Jump to avoid
  		mov	ebx,edi
  		sub	ebx,edx
  		sub	ebx,ecx
  		add	[esi+10h],ebx
  		lea	ecx,[edi-0Ah]
  		push	eax
  		push	ecx
  		mov	[esp+1Ch],ecx
  		call	jmp_USER32.dll!SetCursorPos ; Avoid this
  		movzx	eax,[esi+0Ch]
  		pop	edi
  		pop	esi
  		pop	ebp
  		pop	ebx
  		add	esp,00000030h
  		retn	0018h
 L00478D3F:
  		add	edi,FFFFFFF6h
  		cmp	edx,edi
  		jle	L00478D66 ; Jump to avoid
  		mov	edi,ecx
  		push	eax
  		sub	edi,edx
  		add	[esi+10h],edi
  		push	ecx
  		mov	[esp+1Ch],ecx
  		call	jmp_USER32.dll!SetCursorPos ; Avoid this
  		movzx	eax,[esi+0Ch]
  		pop	edi
  		pop	esi
  		pop	ebp
  		pop	ebx
  		add	esp,00000030h
  		retn	0018h
...

The block at label L00478D12 is calling SetCursorPos as well, and can also be changed with a jmp instruction to skip forward to L00478D3F, which looks quite similar to the previous label, and in turn can be modified to always jump forward further to L00478D66.

Once in the L00478D66 block, no further calls to SetCursorPos show up, with the function eventually returning a bit further, indicating that we’ve taken care of all the calls in this function.

I tested these four changes, and they are right on point. Panning works as it should, with cursor wrapping completely disabled, and thus no more glitching. Success!

As a side note, there’s another bug occurring with 3ds Max 2010 under Windows 10. Context menus may end up getting their overlay stuck on the screen. Simply renaming 3dsmax.exe to something else resolves this issue. Possibly Windows, or the graphics driver, is implementing some modified behaviour for executables named 3dsmax.exe, which doesn’t play well with this particular version. It’s probably being done for some legacy performance or compatibility reasons.