Xformers optimization causes inconsistent generations with the same parameters

[JustMaier]

Describe the bug
The new xformers optimization has something that makes it so that when given the same parameters for generation the same image won't be generated, see examples below. It seems to be less noticeable given more steps.

To Reproduce
Steps to reproduce the behavior:

1. Activate xformers enhancement by adding --xformers to launch params
2. Generate 'a robot dog' with seed 1, euler a, and 30 steps
3. Generate it again and notice that the result is slightly different

Expected behavior
Generations should be consistent given the same parameters.

Screenshots

Prompt: A robot dog
Seed: 1, CFG: 7

With xformers:

Without xformers:

With xformers + differences highlighted:

Without xformers + consistency highlighted:

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Chrome
• Commit revision: b458fa4

Additional context
Perhaps this is just the nature of the optimization?

[Arron17]

I'm having the same issue. Using the AND contidional formatter seems to greatly exacerbates it.

Prompt: A stuffed bear toy wearing a tophat AND a table ornament
CFG scale: 7, Seed: 123456789, Size: 512x512

Without xformers:

With xformers:

The differences in this example are subtle, but whislt doing other testing, there were some where it was actually a completely different image.

[grexzen]

Looks like sometimes it makes the image better, sometimes worse.

[azumukupoe]

Perhaps this is just the nature of the optimization?

maybe?
same thing happened when split attention became enabled by default
cae5c5f
k-diffusion updates seem to alter outputs as well
b6f80bd c30c06d

[Doggettx]

I've ran into this issue a lot while optimizing code on my own branches, It seems to happen mainly due to rounding differences, you can usually see by using fp32 instead of fp16 which will reduce differences quite a lot. Of course, the step from fp16 to fp32 itself also introduces a lot of small differences as well.

[cvar66]

OK so this is normal I guess. This issue became very noticeable to me today when I was using a long complicated prompt using AND while also using a higher resolution with the hi res fix. I could generate the same identical prompt 5 times in a row and the output images will all be different variations of the same thing, sometimes drastically. Changes at smaller resolutions with simple prompts seem to be much more subtle.

[f-rank]

I notice that with Euler and DDIM (512x512) it is able to reproduce the same image over and over. Diff from the one created with same info while not using --xformers. But that part is to be expected.
At bigger sizes all the methods produce different variations with the same starting point info.

I think it would be normal to expect --xformers to change the outcome a bit, but that the changed outcome would be repeatable, not variant on each generation from the same starting point.

It also screws up [a|b], a AND b and TI embeddings big time.

[0xdevalias]

A friend mentioned that using xformers could make things non-deterministic, and that there were a lot of references to it on the repo issues here. Wanting to understand a bit more about it, and link a bunch of potentially related issues together, I tried to find as many issues as I could that seemed to be related to xformers and the potential for it to be causing non-deterministic / unstable / inconsistent results:

The following may potentially be related (ordered by issue number):

Issues:

• Xformers optimization causes inconsistent generations with the same parameters #1999
• No token limit for prompts broken? Different results with the same prompts and seed #2005
• The same image can no longer be output under the same conditions. #2197
• On the problem that the same resolution can not be generated stably #2293
• Different results after swapping graphics card #2423
• Different images for same seed with xformers and "Highres. fix" enabled when ancestral samplers are used #2528
• Different outputs since WebUI update #2943
• xformers and euler_a variant results for same generation settings #3054
• [Bug]: The prompts do not produce the same results as before the latest updates on 10/22/22 #3476
• [Bug]: xformers always generate different results, seed is the same! #3967
• [Bug]: In img2img, setting denoising strength to 0 does not return original image #4011

Discussions:

• Different results after swapping graphics card #2418
• --xformers CMD line option causes slightly poorer quality images on GTX 1650 (Turing), possibly others? #2592
• Highres.fix composition #2705
• Exactly same prompt no more the same ? #2849
• xformers option has good and bad aspects #2958
• What has changed recently? Different and Worse Results from Same Settings #3461

I also came across this thread in the xformers repo, which while I can't guarantee is related, am wondering if it might be:

• xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

And a question I raised on a PR in the diffusers repo:

• Up to 2x speedup on GPUs using memory efficient attention huggingface/diffusers#532 (comment)

  • A friend just mentioned that apparently using xformers tends to lower the quality of generated images, and apparently even causes StableDiffusion to generate different images for the same seed/settings. I haven't looked too deeply into things to try and validate that, but I was wondering if a) that is something that is currently known/documented, and b) if that's likely to be fixed at any point, or if it's just 'the price we pay' for the benefits it adds. Would any of that be a concern with integrating things here? I did stumble upon this issue on the xformers repo, but not sure if it would be the same/similar root cause as my friend described: xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219