[d3d9] `DrawIndexedPrimitive` with an underflowed primitive count "works" natively, but breaks DXVK

[CookiePLMonster]

Software information

The issue is general, but I reproduced it on Colin McRae Rally 3 with an earlier version of my patch. I used to have a bug where I attempted to draw 0 rectangles using the in-game drawing functions, but the problem is that this always "worked" fine on native D3D9, but resulted in issues in DXVK, hence it took me this long to notice my mistake.

In the case of this game, a 0-rectangle draw issues a call to DrawIndexedPrimitive with 2 * (3 * 0 - 2)) = 4294967292 primitive count. This is obviously invalid, but in native D3D9 this doesn't produce visible side effects, while with DXVK it wrecks performance and results in random corruption showing on screen:

System information

• GPU: NVIDIA GTX 1070
• Driver: 560.94
• Wine version: Native, on Windows 10 22H2
• DXVK version: 2.6

Apitrace file(s)

Something to note - this issue crashes both PIX and apitrace. I managed to capture a trace but it crashes the moment you try to preview the last frame of the capture. Nonetheless, I'm attaching the file in case it is useful in this broken state. .7z compresses it much better than .zip but it cannot be attached on GitHub, hence the double packing:

Rally_3PC.7z.zip

[K0bin]

The large draw call is not in the apitrace unfortunately. The number shows up for the locking call before that though: 26281 @0 IDirect3DIndexBuffer9::Lock(this = 0x8c48ee0, OffsetToLock = 13848, SizeToLock = 4294967292, ppbData = [0xacfa618], Flags = D3DLOCK_NOOVERWRITE) = D3D_OK.

I wonder how we should deal with this.
We can calculate the maximum number of indices/vertices (depending on whether the draw is indexed).
In the trace the buffer size is 32768 bytes, so assuming 32 bit integers (didn't check tbh), the maximum number of indices would be 8192. The last draw before the problematic one uses firstIndex = 6920. That leaves 1272 indices. With TRIANGLESTRIP that leaves a maximum number of 213 primitives.

We could

• clamp the number of primitives to that maximum
• skip the draw if it exceeds that maximum

Given that it also causes artifacts for you, we should probably skip it but I need to test what Windows does.

[CookiePLMonster]

Given that it also causes artifacts for you, we should probably skip it but I need to test what Windows does.

Indeed. Considering the game crashes if this happens with PIX attached (even when not capturing), I don't know what more data I could provide from Windows to help solve this.

[K0bin]

I'll just have to write a test and see what happens.

[WinterSnowfall]

I guess it most likely locks the entire buffer if the lock size is over the buffer size (so clamp), but testing is the way to be sure.

As for the offending draw call, IIRC it had NumVertices = 0, which we should probably check if it's outright skipped by native.

[K0bin]

I guess it most likely locks the entire buffer if the lock size is over the buffer size (so clamp), but testing is the way to be sure.

We already clamp, the lock is fine.

IIRC it had NumVertices = 0

That would make a lot of sense and make it a very easy fix.

[WinterSnowfall]

I guess @CookiePLMonster should comment, but yeah, should be primCount = 4294967292, NumVertices = 0, which is a bit silly, but it's d3d9 after all. Currently we only derp out early on !primCount I believe.

P.S.:

[in] primCount

Type: UINT

Number of primitives to render. The number of vertices used is a function of the primitive count and the primitive type. The maximum number of primitives allowed is determined by checking the MaxPrimitiveCount member of the D3DCAPS9 structure.

[CookiePLMonster]

The exact parameters of the offending call were:

• D3DPT_TRIANGLESTRIP
• 0 base vertex index
• 6535 min vertex index
• 0 num vertices
• 7702 start index
• 4294967292 prim count

(of course the min vertex index and start index are "moving" as they're reusing dynamic vertex buffers)