xformers and euler_a variant results for same generation settings

[f-rank]

Describe the bug
With -xformers the same generation settings produce variant results with euler_a, the latest fix hasn't solved this.
Didn't notice the other samplers being affected.

To Reproduce
This specific one is dramatic.
Try: "tiny cars", 512x512, euler_a, CFG:7, seed: 26798323584
One car keeps popping in and out.

Expected behavior
Being able to reproduce same result

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser : Chrome
• Commit revision : Commit hash: c71008c

Additional context
I get deterministic results with other samplers that also started changing things up after the introduction of xformers, with euler_a not all images do this, some just present minimal diffs. Maybe it is supposed to work like this even, specific to the sampler... don't know. Thinking all samplers should be deterministic, but could be me.

[younyokel]

I'm sure it was already addressed. Xformers change the output, it's how it works apparently

[aleksusklim]

Create a grid against Steps (18, 19, 20, 21, 22) and CFG (6.4, 6.7, 7, 7.3, 7.7) – to better understand, are the cars supposed to behave like that in EulerA in the first place.

[f-rank]

I'm sure it was already addressed. Xformers change the output, it's how it works apparently

xformers turns euler_a non-deterministic ?

[f-rank]

Create a grid against Steps (18, 19, 20, 21, 22) and CFG (6.4, 6.7, 7, 7.3, 7.7) – to better understand, are the cars supposed to behave like that in EulerA in the first place.

The values around the specific example display much tamer variations.

[mayofiddler]

Euler_a will produce different images regularly. Most of the ancestral samplers are like this but Euler_a is very prone to it.

[Skifoid]

masterpiece, best quality, 2girls, (looking at viewer:1.1), hug, hand on another's face, street, day, school uniform, skirt, pink hair
Negative prompt: lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name, hands, fingers
Steps: 40, Sampler: Euler a, CFG scale: 12, Seed: 2560115584, Size: 512x512, Model hash: c1002b19, Model: WD_NA_diff, Clip skip: 2

With --xformers:

without:

[mayofiddler]

This relates to xformers and the high-res fix but it might have some useful info near the bottom of the comments for you.

#2528

[f-rank]

Euler_a will produce different images regularly. Most of the ancestral samplers are like this but Euler_a is very prone to it.

It produces the same image when not using --xformers.

[Jellybit]

Yes, I had the same mutation. The images on euler_a are steady as a rock without xformers running. Once I run xformers, it changes every single image. Check it out, using a fresh install of the 10/22 Auto1111 WebUI, with the SD 1.4 model, using this prompt:

A happy man in a city eats a cake while a crowd of people stand behind him, close-up
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3225699936, Size: 512x512, Model hash: 7460a6fa, Clip skip: 2

Person has four buttons on their shirt. I hit generate again...

Person has three buttons. I hit generate again...

Person has two buttons.

Again, before anyone says this is normal for Euler a, this does not happen without xformers on. If xformers is turned off, the images are pixel perfect steady. Xformers is doing something to the results.

PC:
Ryzen 7 5800X
64GB RAM
Nvidia Geforce RTX 3090 with 24GB VRAM
Windows 10
Python 3.10.6
--xformers is my only command line option

[Jellybit]

I just confirmed that it happens not just with Euler a, but with all of the following. I used multiple prompts/tests to make sure.

Euler a
Euler
LMS
Heun (minor change)
DPM2
DPM2 a
DPM fast (drastic change)
DPM adaptive
LMS Karras
DPM2 Karras
DPM2 a Karras (fairly big change)
DDIM
PLMS

Yep, that's the entire list of sampling methods. Not a one was safe from it, though Heun almost held onto consistency for me. It tried its best.

But after reading the thread that mayofiddler linked to above, it seems that this is an essential part of how xformers works, and maybe cannot be fixed. It's just a price we pay for faster processing.

[younyokel]

I don't think you can do anything about it, it's just how xformers work.

[0xdevalias]

I don't think you can do anything about it, it's just how xformers work

Is there a source that talks about the 'why' of it being 'just how xformers works'? I'd be interested to read more into the specifics of what that limitation is.

[0xdevalias]

A friend mentioned that using xformers could make things non-deterministic, and that there were a lot of references to it on the repo issues here. Wanting to understand a bit more about it, and link a bunch of potentially related issues together, I tried to find as many issues as I could that seemed to be related to xformers and the potential for it to be causing non-deterministic / unstable / inconsistent results:

The following may potentially be related (ordered by issue number):

Issues:

• Xformers optimization causes inconsistent generations with the same parameters #1999
• No token limit for prompts broken? Different results with the same prompts and seed #2005
• The same image can no longer be output under the same conditions. #2197
• On the problem that the same resolution can not be generated stably #2293
• Different results after swapping graphics card #2423
• Different images for same seed with xformers and "Highres. fix" enabled when ancestral samplers are used #2528
• Different outputs since WebUI update #2943
• xformers and euler_a variant results for same generation settings #3054
• [Bug]: The prompts do not produce the same results as before the latest updates on 10/22/22 #3476
• [Bug]: xformers always generate different results, seed is the same! #3967
• [Bug]: In img2img, setting denoising strength to 0 does not return original image #4011

Discussions:

• Different results after swapping graphics card #2418
• --xformers CMD line option causes slightly poorer quality images on GTX 1650 (Turing), possibly others? #2592
• Highres.fix composition #2705
• Exactly same prompt no more the same ? #2849
• xformers option has good and bad aspects #2958
• What has changed recently? Different and Worse Results from Same Settings #3461

I also came across this thread in the xformers repo, which while I can't guarantee is related, am wondering if it might be:

• xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

And a question I raised on a PR in the diffusers repo:

• Up to 2x speedup on GPUs using memory efficient attention huggingface/diffusers#532 (comment)

  • A friend just mentioned that apparently using xformers tends to lower the quality of generated images, and apparently even causes StableDiffusion to generate different images for the same seed/settings. I haven't looked too deeply into things to try and validate that, but I was wondering if a) that is something that is currently known/documented, and b) if that's likely to be fixed at any point, or if it's just 'the price we pay' for the benefits it adds. Would any of that be a concern with integrating things here? I did stumble upon this issue on the xformers repo, but not sure if it would be the same/similar root cause as my friend described: xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219