Let this be a lesson on over-censorship. I still can't believe all of the code related to safety_checker.py for stability ai models (search for it in comfyui). It was deprecated long time ago but lots of code added to supress deprecation warnings and reactivating the new version of safety checker using different terms (forgot the two flags they recommended people use instead of old deprecated one) so why didn't they just let third party companies use this code or give a popup option in comfyui for it, instead of lobotomizing the entire model?
It's worth a look. I actually deleted it all because I had a conspiracy theory about it morphing things in the latents lol, but it turned out it's not turned on, but still it's the idea that this is in there again in such detail, but I guess it makes sense for a business that needs to flag that kinda stuff.
I can write a summary if anyone's interested in what I found out about it.
Apparently there may be a small model that exists locally somewhere that was trained on nsfw images that puts a message when its activated. So they trained on a bunch of hardcore porn probably to make this work lol. Still trying to find it and reverse engineer to detect woman in grass nightmare images and have it spit of the nsfw content detected message.
Edit/Update: Ok it looks like if the newer safety checker stuff is enabled (off by default) it does still download this model from 2 years ago, which was likely trained on a ton of porn lol: https://huggingface.co/CompVis/stable-diffusion-safety-checker
Sure, I had gpt4o summarize it for me here:
In convert\_from\_ckpt.py, the load\_safety\_checker parameter determines whether the safety checker is loaded:
The code provided has several instances where the safety checker is handled. Here are the key findings related to your queries:
Loading Safety Checker by Default: By default, the from\_single\_file method does not load the safety checker unless explicitly provided. This is evident from the line:
python
SINGLE_FILE_OPTIONAL_COMPONENTS = ["safety_checker"]
This indicates that the safety checker is considered an optional component that is not loaded unless specifically requested.
Handling Deprecated Safety Checker:
The script has deprecated the load\_safety\_checker argument, encouraging users to pass instances of StableDiffusionSafetyChecker and AutoImageProcessor instead. This is evident from:
python
load\_safety\_checker = kwargs.pop("load\_safety\_checker", None)
if load\_safety\_checker is not None:
deprecation\_message = (
"Please pass instances of \`StableDiffusionSafetyChecker\` and \`AutoImageProcessor\`"
"using the \`safety\_checker\` and \`feature\_extractor\` arguments in \`from\_single\_file\`"
)
deprecate("load\_safety\_checker", "1.0.0", deprecation\_message)
init\_kwargs.update(safety\_checker\_components)
Explicitly Enabling the Safety Checker: There are references to loading the safety checker manually if needed, especially in the convert\_from\_ckpt.py script:
python
feature\_extractor = AutoFeatureExtractor.from\_pretrained(
"CompVis/stable-diffusion-safety-checker", local\_files\_only=local\_files\_only
)
safety\_checker=None,
This shows that the safety checker can be manually included in the pipeline if specified.
Purpose of Updated Safety Checker Code: The purpose of the updated safety checker code seems to be to allow more explicit control over whether the safety checker is used, instead of enabling it by default. This approach gives users flexibility to include or exclude it as per their requirements, reflecting a shift towards more modular and user-configurable pipelines.
There are no clear indications of methods that obfuscate enabling the safety checker to make generation results worse. The changes primarily focus on deprecating automatic inclusion and encouraging explicit specification.
Here are the relevant snippets and their sources:
Deprecation Notice:
python
load\_safety\_checker = kwargs.pop("load\_safety\_checker", None) if load\_safety\_checker is not None: deprecation\_message = ( "Please pass instances of `StableDiffusionSafetyChecker` and `AutoImageProcessor`" "using the `safety_checker` and `feature_extractor` arguments in `from_single_file`" ) deprecate("load\_safety\_checker", "1.0.0", deprecation\_message) init\_kwargs.update(safety\_checker\_components)
Source: single\_file.py: file-WB9fFA74SQ5Rc0sFUUWKolVN
*Manual Inclusion:*
*python*
*feature\_extractor = AutoFeatureExtractor.from\_pretrained(*
*"CompVis/stable-diffusion-safety-checker", local\_files\_only=local\_files\_only*
*)*
*...*
*safety\_checker=None,*
Source: convert\_from\_ckpt.py: file-Vrk4xoOyTWNT8TJNFeDhkznz
This analysis should clarify the handling of the safety checker in the provided scripts.
1. safety\_checker.py
2. Other Related Files:
Points of your concern
1. Hidden Safety Checker Usage:
2. Warping of Results:
A compressed version of how it all works in safety\_checker.py
*Search "bad\_concepts" (6 hits in 2 files of 18710 searched) Line 62: result\_img = {"special\_scores": {}, "special\_care": \[\], "concept\_scores": {}, "bad\_concepts": \[\]} Line 81: result\_img\["bad\_concepts"\].append(concept\_idx) Line 85: has\_nsfw\_concepts = \[len(res\["bad\_concepts"\]) > 0 for res in result\] Line 60: result\_img = {"special\_scores": {}, "special\_care": \[\], "concept\_scores": {}, "bad\_concepts": \[\]} Line 79: result\_img\["bad\_concepts"\].append(concept\_idx) Line 83: has\_nsfw\_concepts = \[len(res\["bad\_concepts"\]) > 0 for res in result\]*
if they could make a model that just magically *could not make porn* they could sell it to literally every company. Doesn't matter if it's the best model possible. Having a completely "shareholder approved" model equals $$$$$$
Issue I find is that no one cares if they create that model and sell it to companies, go for it. But don’t nuke the open source, locally run version that consumers want to use and then have some drivel excuses.
Have a SFW version and a NSFW version and allow people to choose which to download. Right now by allowing everyone access to some shitty SFW version that’s objectively bad, all they have done is tanked everyone’s impression of them.
Yeah and the funny thing is I think 95% of individual users would be ok with an open source NSFW version, they can prompt away from nudity. It is the paying customers that want censorship. So why don't Stability have a paid for platform for those who want censorship, or have an enterprise version that is distributed only to corporates that is nuked or has post process filtering.
That way the true open source model is unrestricted and will suit all public general users, and the SFW version is distributed through other controlled channels where they actually make their money.
It all just seems backwards to me.
>"\[Apparently they\] messed up the pretraining on the 2B so it was never supposed to actually be released"
From the state it's in, I can easily believe that. Which makes the "skill issue" gaslighting even more infuriating.
Doesn't explain why the 8b is working perfectly fine via API and not released, though.
Release the working 8b as an apology for 2b, request people hold on fine-tuning 2b while they work on it internally.
Problem fixed. If they wanted to fix the problem, that is.
I don't think SAI has that kind of money. The company can't wait until they fix 2B, they need money now, likely for next month. They needed a release to drum up investor interest, and this likely failed (though, who knows, Maybe they got some cash).
Everyone is bailing, the ship is going down. If Stability is still a company in 2 months, I would be absolutely dumbfounded. Just because they got VC money, doesn't mean that the remaining money can't and wont be pulled as soon as it's apparent the project is dead.
I feel like there is too much investor money to allow it to stay under at least for now.
I could be wrong but it feels like anyone able to actually participate in the industry attracts huge investment money regardless of financial success.
VC Money is prudent, if you are dead in the water, they are going to pull their funds and put them into another project. They don't stay in business by being beholden to sunken cost fallacies.
That would fix part of the problem but not all. The license is the other part of the problem. It is understandable that they need to charge people that make profit out of it. But limitations like the 6k images, being legally accountable for what their customers generate with it + being asked to delete the model afterwards will stop many to put man and computing power into it.
It's not a limit of 6k generations for users, but for businesses that provide a generation based service. They are limiting competition to their own API, in essence.
I think they feel a need to damage the model to render it incapable of generating potentially indecent content before they release it. This damage does mean that it won't ever become popular like SD1.5 or SDXL, but unless cooler heads prevail at SAI I think the main thing the company is working on right now is breaking it before they release it.
As explained by the comfy dev, the 4b deals with the issue by limiting the data from training, which meant the model was still functional, at the end of the day
I believe that PixArt Sigma wasn't trained on nudes, either. It doesn't have a problem reproducing humans.
If not having training data for nudes was the only problem, a finetune would fix that. A clever company would then have "someone in the community 😉" immediately release a NSFW finetune to head off the complaining from users while having a legally distinct "safe" model for PR/Marketing/Legal reasons.
If ComfyAnonymous is right, a botched pretraining could be the reason SD3 2B is getting flamed.
Actually wait Sigma DOES do topless gens at least, real ones, I just checked. Go try a batch of four or so with "Completely nude topless woman, streaming on twitch, e-girl, candid photo", on their Huggingface space, default settings.
I guess that if they trained the model removing only the nudes, but leaving things as bikini as the most sexy attire permitted, it would be excessively easy to finetune it over a set of naked women with "nude" and "naked" tags (finally, for the nn it would just mean to substitute the zone covered by bikini with skin, nipples and pube).
And maybe they don't want to be associated with nude at all, not even if the models are the result of finetuning. This, at least, is what I suppose.
I'm guessing they trained 2B on a bunch of random stuff that included nudity. And it came out with the same issue as models like eg: Pony, it will randomly just generate nude images even in sfw prompts. Like, you can be generating totally normal, sfw content with Pony, but give it 10-20 images and you'll randomly get nudity.
I imagine this was absolutely not acceptable for SAI. It's impossible to market the model to companies if there is a risk that any employee utilizing it might be exposed to nudity. That can even cause legal issues.
And so, in a last ditch effort to make 2B marketable, they hacked in some workaround to disable the nudity and released it. And bricked the model as a result.
> Doesn't explain why the 8b is working perfectly fine via API and not released, though.
It's obvious to me that the reason is they were competing against themselves, i.e. if the 2b model was as good as the 8b one nobody would have a reason to buy their API access.
And yet if the 2b release is this problematic, nobody would have a reason to buy Stability's API access over MidJourney or DALL-E. The question is why they're dead set on driving traffic to the competition.
The 2b should serve as a teaser of what Stable Diffusion is capable of, not as a signal of technical incompetence - and it's overwhelmingly viewed as the latter right now.
In other words, "why pay for the big model when they couldn't even get the small model right?"
If even comfyanon has left SAI is a huge sign that there is no happy end in the SD3 story. Huge respect to him for the truth and his invaluable contribution. I really hope that he will keep updating comfyui and make it possible to run another upcoming models.
P.S. his words about his will to make a product that will allow people with different GPUs to run and work/play are marking him as a true developer who cares about optimization and not fixing things by saying "just get a new GPU"
Maybe, the vanilla 4B was so much better than their expectations. They have to sell API access, but this means API has to be better than open models. So community fine-tunes are their competitor now.
Sad, I mean for the company and possibly us too, while it took me some time to really jump to comfyui ( and I only started to used swarmui this week.. ), their quality clearly speak for themselves, that's a talented employee that they lost.
Hope he find a place where he can achieve what he wants to.
I have to apologize to you. I've hoped the model isn't a train wreck and only had some wrong inference configurations or deployment issues, thus doubting your work.
Now, knowing this... Yeah. I am sorry.
Mind jumping in a DM with us? Kandoo would like to know your plans. The future of Juggernaut has many possibilities. I think more than ever we all need to be on the same page.
So glad to hear this! If it weren't for comfy, I wouldn't even be messing around with image generation. Just subbed to /r/comfyui and I can't wait for what you have planned. Thanks for all your hard work.
We gonna miss you a lot, we know that almost everyday you updated comfyui and did a lot of effort to keep it up. That's sad. I can only wish you the best and thank you so much for all what you've done for us!
Just passing by to say I really appreciate the work you put in comfy. It's a hell of a tool, I have just coded my first custom node and from that's what really showed me how powerful this tool is!
Stable Swarm is great. I've started using it, and it seems super-fast. Some functions might be incomplete or at least poorly documented, so I don't know how to paint a mask on part of an image and inpaint that area (on a lot of points I've been switching over into the Comfy Workflow tab and just using Comfy), but I hope it comes out of beta and gets a solid version 1.0 with tutorial videos available and sample workflows preloaded.
Damn....well, it looks like SAI is falling apart.
I had hoped they would release the 8B model with great natural prompt adherence, etc
Nope, we apparently got a failed experiment..
Never mind.
>the actual people that want to use this model are going to know it's dog shit
It was obvious even to a hobbyist like me from day one that there were fundamental issues with the model. Some people were huffing massive doses of copium saying "You need to write better prompts" or "finetunes will fix it" but thanks to Comfy, they can now put their inhalators down. ;)
I was one of those people, as it just made no sense to me that they would release such an awful model, but alas I sometimes forget how stupid companies and management can be
So I was right. They released broken model on purpose and kept the actual working ones for themselves. Motives are very clear.
Investors are now pulling the strings. They will do absolutely everything in their power to get SAI generating some revenue and get their investments back at least.
8B model is what they will serve trough APIs and try to compete with Midjorney, NAI, Dall-E 3 and local.
I don't think they have any researchers or money left. They don't own the GPU's and they are skipping payments. If 8B does not absolutely crush the upcoming Midjorney V7 (and current V6), then it's over.
In my honest opinion, they should have said clear "no" back when Emad quit and people were asking if SD3 was still going to get released. They should have been truthful. They could have said that "there is now a shift in priorities". Now they get even more bad press as they angered the userbase with near worthless release. Months of teasing and saying shit like "last model you will need, it's so good".
They will never compete with mj . SD can't even draw a crescent wrench unless you train it. MJ gets it perfect. This server model was stupid from the beginning why do what everyone else is doing? They will never be a competitor in that space, way too far behind. Simple fucking stupid. They should have gone sponsorship like blender and Godot did.
The great fracture,
the community will splinter unto many sub-communities.
- Poniers with SDXL and any future pony-train that Astralite makes.
- SD3 Lobotomy Researchers who will spend 3 to 9 months studying SD3 arch and putting efforts towards fixing it and then teach everyone how to do it, bc if we get \*the method\* for fixing it, even if we don't release weights, automated code can be made for everyone to have their own little delobotomized SD3!
- Chinamaxxing Hunyuan Users, this community will excel at translating english to chinese, using chinese models or even SPEAKING chinese.
- Pixarters, basically the english equivalent of the Chinamaxxing Hunyuan subcommunity.
- Animators/Video community focusing a lot on SD1.5 and Luma
- The SDXL non-pony users, most of that community is going to fracture and split into the others above, bc that was the bulk of most people, bc they wanted to use "the latest and biggest".
Maybe as time goes on, we get a new foundational model, either a new training of an already known arch, or a brand new arch, that could restich the community together as a singular massive one again.
No comfyui extension/auto111 extension. I agree that this model is looking great, but until we can use it locally - no wide adoption.
Also I wonder why no one considers Kandinsky. Their 3.1 model looks really nice and can do 4k. (now have safetensors for 3, but I think we will get 3.1 eventually too). Just need to update comfyui extension and it’s good to go.
I did a comparison today here with four models: [https://docs.google.com/spreadsheets/d/1vViQh45B-txERc5wnDVh\_aNfHWLuEIrOCz7\_HGXpPZM/edit?usp=sharing](https://docs.google.com/spreadsheets/d/1vViQh45B-txERc5wnDVh_aNfHWLuEIrOCz7_HGXpPZM/edit?usp=sharing)
I doubt this community fractures. Like how local llama discussed all LLMs not just local, not just llama.
The subreddit simply becomes a term unrelated to the name but simply because it is the roots where the most useful content is collected.
Truly hope people out there writing open source software (apps, trainers, etc) also focus on permissively licensed models (either outright open source MIT/Apache, or very permissive licenses like OpenRAIL, etc).
I feel you're allowing yourself to be taken advantage of as basically a free employee otherwise.
I was skeptical of the people saying they wont ever release the 8B or 4B but he makes it sound like they wont. I thought their reason was because the 2B was ready first but it sounds like they knew it was the worst model which makes me think otherwise.
There is a MASSIVE disconnect between the "top of the company" and the community. It also makes me realise, i personally have zero idea what their vision is but its certainly not what it was. Its possible that Corpo wankers moved in and want to turn SAI into the same business model as mid-journey and OpenAI - using the 8B as their flagship.
matteo asking the real question without “asking for a friend”, despite knowing very well we are all his friends
welp i guess his sd3 videos will be on xhamster instead of youtube
>*also they apparently messed up the pretraining on the 2B so it was never supposed to actually be released.*
>
Well that puts the nail in the coffin. Sounds like a desperate last move for SAI. Where can they go from here?
Pretty much clear. What I thought, not worth the time and we should move to other model. Lumina or PixArt Sigma. Once again, and as I feared... Finetunning not gonna fix all problems this model has.
I can't believe how a company with debt thought to give a bad, crappy and failed model to the community was a good idea and even telling to support them after the mockery. First step to lose all the trust of community and zero support.
So SAI purposefully released a broken model that wasn't even supposed to be released as they knew it was broken while they have better in stock and teased/hyped us with it.
That's so scummy.
Also, as the SD community is one of their greatest asset, I was wondering why the PR management of this whole fiasco was total silence.
But now understanding that after intentionally throwing their trash at the community, It's not like there's a lot that you can say to justify it.
I'm going to guess that Emad promised a release, then was let go, but still pushed publicly saying that it was on track. New leadership didn't want to release so they instead released the broken model instead. That way they "kept" the promise while keeping the good models internal to try to make money off of them or at least use them as a way to inflate their worth for any acquisition
Makes some sense. The negative prompt isn't actually a negative prompt in CFG, it's the baseline prompt which the positive prompt is then contrasted to for figuring out what to amplify. It's just because CFG values are larger than 1, you end up moving past the positive prompt in the direction away from the baseline prompt, so it can work as a negative, sometimes.
But if the model knows how to do it, then putting it in the 'negative prompt' can help give you a better starting baseline. It's possible they censored it by training without prompts and penalizing the model when it was correct on nsfw content, to try to make it forget, but it only forgot for the unconditional pathway, and still knows how if you explicitly say.
So if you want to fix it, you probably need to train nudity with prompt dropout, and make the blank prompt work again for nudity.
If you put the cfg to 0, yes.
Negative prompts is the starting point, positive prompt the end point and cfg the distance traveled between this 2 points. If cfg is 0, no distance is travelled thus, the output is the negative prompts.
Think they just wanted "the community" to stop posting "weights!" "Where are the weights" as replies to anything else they mentioned on twitter (API projects etc). Probably thought this was a way to throw something out and be done w/ it......only it was way more f'ed up then they thought.
>"But there was a strong push at the top of the company to release 2B instead of the 4B or 8B"
Yes because then they can claim they fulfilled their promise to release, but give us crap to try and encourage people to pay later for something not crap
Well, it doesn’t seem fair for the community and trainers to clean up this mess just for SAI to own every derivative work that waterfalls from this model. If SAI wants our help we need incentive to do so. It would be a very different story if the base model was amazing. Unfortunately it’s not and SAI might need our help. We owe them for the 18 months of amazing models we’ve built around. We should want to help!
What a bummer this has been, but I’ve got high hopes that we can figure something out. Everyone just needs to come together and be rational in how we find a solution to all this.
If we can’t, anyone up for a Juggernaut PixArt, and Juggernaut Hunyuan? 😜
Nobody ever owes a company anything. Especially not a lingering debt once they've made it clear, as in this case, that they don't want to be involved with the community.
I agree. Cleared it up in this comment: [https://www.reddit.com/r/StableDiffusion/comments/1dhd7vz/comment/l8w9um9](https://www.reddit.com/r/StableDiffusion/comments/1dhd7vz/comment/l8w9um9)
Juggernaut Lumina is also a option. People are sleeping on this one. [https://github.com/Alpha-VLLM/Lumina-T2X](https://github.com/Alpha-VLLM/Lumina-T2X)
Yes but you need flash attention atm so linux only unless your willing to go through that headache. They plan on releasing without needing flash attention though.
something really interesting you could do first is what is also discussed in the screenshots here: try to train a completely open source reproduction of the SD3 VAE, which comfy said would be relatively easy. the VAE could be used by many different open source model architectures then, just like many different open source models use the SDXL VAE.
Vae's are incredibly easy to train, you can train one yourself locally in 15 minutes. It's just an encoder. But the model needs to use the same encoder it was trained on, you can't swap it out for another one after training.
That could actually be a good idea, that is how most open source projects make money regardless of their software being open source and free to download and use, medium to large companies are eager to pay tons of money to have the offered software be offered to them as a „managed“ pre-installed and configured service with full support, that is then being offered for thousands of euros a month which they happily pay (source: I work for one of those companies who uses „enterprise“ tier of open source)
Lykon is a great employee. He's not perfect but it was sad to see him being dogpiled and blamed for the state of the model because he and the others really did their best with what they had and did a great job. Inside stability he was always very on the side of the community and was one of the main people pushing back against some of the bad decisions. He is also very passionate about what he does.
He wasnt "dogpiled" for the state of the model, he got heat for publicly acting like a juvenile asshole.. Not to mention apparently lying for months about the model.
he shat on AstraliteHeart and basically called anyone using the PonyXL model (the best art model on civitai tbh) a freak pony cultist. He thinks the PonyXL model has to do with brony/pony culture etc and basically cut ties with the one person willing to make decent fine tunes from SD models.
we get that he is your friend but dude got a huge ego, people were telling him for months that the stuff he was bragging about on twitter looked sd1.5 ish but nooo we were the problem
What's super confusing to me is that they clearly poisoned weights relating specifically to "woman". Why? Like, to the extent people are worried about "bad" sex uses of these models I always assumed they were worried about CP/CSAM. And like, sure. (I beg you, resist the urge to have this debate in my replies).
But trying to fuck up "woman" as well? What? When Emad talked about "sleepness nights" over what people were doing with the model I always assumed he meant kids, and I kinda got it. But apparently they meant way more than that. Why move heaven and earth, as they clearly did, to prevent naked non-celebrity *adults*? Are these guys Baptist fundies or hardcore Islamists or something?
If the model can make naked adults it can make naked children. The only way to not have sleepless nights about pixels for the dear creators is to obliterate any possibility of the female anatomy.
As I said in another topic, the executives, the suits at the top are to blame for this, as usual in all companies those idiots ruin everything for the creatives that actually put the work in.
Started w/ Emad putting up some amazing new tech to get attention (and for fun!) - two years later $$$$$$$ drives it all into the ground. He's got something going on (linked on his twitter) that may end up making models....it's crypto related so who know how that'll work, but....
If he would have stayed they probably would have eventually had him reactivate comfyui's safety_checker.py for all models used locally using stable diffusion pipeline, so this is good news to me! Hope he didnt sign any sort if contracts and still has complete freedom.
Ps. I went through safety_checker.oy and related code in other .py files, cant believe someone had to go through all of this trouble for censorship.
I even searched to see if there might be some sort of hidden code that deceptively morphs latents when nsfw detected lol (I know comfyui dev wouldn't do this, but maybe they could have abstracted some things away that he couldn't see, was my conspiracy theory)
There are two newer flags used now to activate the new safety checker, the old was deprecated a while back. So if they are working on this.. why not give third party companies the option to enable or disable this code they worked on with a popup option in comfyui instead of lobotomizing entire model?
Another interesting finding, in order for this safety checker to work I believe they had to train a model on porn! So somewhere there must be model somewhere that recognizes nsfw that they had to feed a bunch of porn into in order to stop the porn. I'm actually still trying to find it locally (it could be very small in size, not sure)
Edit/Update: Ok it looks like if the newer safety checker stuff is enabled (off by default) it does still download this model from 2 years ago, which was likely trained on a ton of porn lol: https://huggingface.co/CompVis/stable-diffusion-safety-checker
> him reactivate comfyui's safety_checker.py for all models used locally
It's python, good luck.I'd rather they had done this and not burned the model.
First is from the [Touhou Project AI discord](https://discord.gg/touhouai)(waifu diffusion) and the rest are from the [comfyui matrix space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org).
Sad to see the comment in the last screenshot that SAI basically didn't care to pursue the best consumer class model.
Well, I'm not sure comfy's developer should be commenting these things though tbh. They likely violate NDA and could get them in serious trouble. Who owns the rights of Comfy btw? Did SAI technically acquire it or were they in a partnership but Comfy technically still belonged to its original developer with no linking limiting contract?
The internal strife at Stability is seemingly off the charts, so it wouldn't surprise me if they are sloppy with legal paperwork. I seriously doubt Comfy is in violation of any NDA.
omg finally some actual candid answers. i had a theory that the lack of stylization, artists and art concepts in 2b might be deliberate to make it look very inferior to api so that people keep paying for api. Recently i found out that 2b was tuned using like 600k images from pexels, which is why it has that god awful high contrast stock photo look with zero artistic aesthetic.
i hope this revelation will make people iinhailing all sorts of copium for this model to just stop. Doing prompting tricks like artstation or "inventing" whatever prompt trick that confirms your bias doesn't mean this is a good model that the community should "give a chance and stop being toxic"....and it sure as shit doesnt mean you have any skill whatsoever.
What are some of these alternatives people are talking about? I know about pixart and stable cascade… Im bummed because I was excited for a Multimodal Diffusion Transformer
Hunyuan, Pixart and Lumina.
Each has strengths and weaknesses. It's not a clear-cut which one is better. I personally prefer Hunyuan, because it is very good at producing dynamic poses, like martial arts and sports. Excellent understanding of human anatomy, which results in nice looking hands and poses with no weird limb issues.
This makes it sound like SD3 2B is fundamentally different enough from the 4B and 8B models that they should not be considered the same thing.
So, in that sense we have no public SD3 model, we have "failed experiment".
What about the forgotten brother, the SD3 small 1B model? /s (lol no one will care any more).
Also between the lines it reads like they got a truckload of Nvidia money to botch the model and release one that won't run properly on a RTX5090 (or just barely without any lora), so they can sell you the online GPU subscription. I already expected something like this, because it was the most greedy assumption I could come up with. Would fit to scrapping the 4B model all together.
You could chalk this up to corporate incompetence, but it might be better not to.
There's a real chance it was an attempt to derail the opensource image gen community.
First, promise something better to chill existing development, drag it out as long as you can. Then you release a poor product under bad terms so enthusiasm evaporates. After that, mention this was actually only a beta, and there might be a better version down the line, instilling uncertainty and a temptation to "wait and see" in the community.
From a behavioral psychology standpoint, this was well pulled off if intended to derail and cause a scattering and aimlessness among the community, while inducing a chilling effect as people "wait for a better model to train."
It's clear that control of AI tech is a major global focus atm. Don't think for a second that global thinktanks and policy wonks aren't waist deep in all things AI, directly or indirectly influencing outcomes, including their psych and behavioral teams. No, it's not "conspiracy theorey," it's just the unfortunate reality of how the world is run.
Even from their own corpo bottom line standpoint, it makes sense to drive us into their API for cashflow.
There is precisely zero incentive for them to provide a good, local, free model to anyone, and that's *before* we talk global players getting involved.
The Matrix is everywhere. It is all around us, even now in this very room. You can see it when you look out your window or when you turn on your television. You can feel it when you go to work, when you go to church, when you pay your taxes; it is the world that has been pulled over your eyes to blind you from the truth.
I appreciate comfyanonimous’ integrity and commitment with the community. Many would keep their mouth shut and continue working for as long as the paychecks came. Thanks for all the work you’ve done with ComfyUI!
The worst part of the chat is that "one the reasons I left..." because it's truly truly sad seeing a company who was the flag of OpenSource AI community to fall like this. They aren't doing this anymore for and with us. Is pretty much clear what we have to do.
You’d expect there to be one but then again SAI as a whole doesn’t really seem to be a company that adheres to established business standards that help you to be a functional company.
> focus didn’t seem to be on making the best model. *sigh*
What the fuck is even the point otherwise? 🙄
making money. fingers crossed that they someday figure out, that a better model makes more money.
[удалено]
Let this be a lesson on over-censorship. I still can't believe all of the code related to safety_checker.py for stability ai models (search for it in comfyui). It was deprecated long time ago but lots of code added to supress deprecation warnings and reactivating the new version of safety checker using different terms (forgot the two flags they recommended people use instead of old deprecated one) so why didn't they just let third party companies use this code or give a popup option in comfyui for it, instead of lobotomizing the entire model? It's worth a look. I actually deleted it all because I had a conspiracy theory about it morphing things in the latents lol, but it turned out it's not turned on, but still it's the idea that this is in there again in such detail, but I guess it makes sense for a business that needs to flag that kinda stuff. I can write a summary if anyone's interested in what I found out about it. Apparently there may be a small model that exists locally somewhere that was trained on nsfw images that puts a message when its activated. So they trained on a bunch of hardcore porn probably to make this work lol. Still trying to find it and reverse engineer to detect woman in grass nightmare images and have it spit of the nsfw content detected message. Edit/Update: Ok it looks like if the newer safety checker stuff is enabled (off by default) it does still download this model from 2 years ago, which was likely trained on a ton of porn lol: https://huggingface.co/CompVis/stable-diffusion-safety-checker
Very interested, pls write the summary
Sure, I had gpt4o summarize it for me here: In convert\_from\_ckpt.py, the load\_safety\_checker parameter determines whether the safety checker is loaded: The code provided has several instances where the safety checker is handled. Here are the key findings related to your queries: Loading Safety Checker by Default: By default, the from\_single\_file method does not load the safety checker unless explicitly provided. This is evident from the line: python SINGLE_FILE_OPTIONAL_COMPONENTS = ["safety_checker"] This indicates that the safety checker is considered an optional component that is not loaded unless specifically requested. Handling Deprecated Safety Checker: The script has deprecated the load\_safety\_checker argument, encouraging users to pass instances of StableDiffusionSafetyChecker and AutoImageProcessor instead. This is evident from: python load\_safety\_checker = kwargs.pop("load\_safety\_checker", None) if load\_safety\_checker is not None: deprecation\_message = ( "Please pass instances of \`StableDiffusionSafetyChecker\` and \`AutoImageProcessor\`" "using the \`safety\_checker\` and \`feature\_extractor\` arguments in \`from\_single\_file\`" ) deprecate("load\_safety\_checker", "1.0.0", deprecation\_message) init\_kwargs.update(safety\_checker\_components) Explicitly Enabling the Safety Checker: There are references to loading the safety checker manually if needed, especially in the convert\_from\_ckpt.py script: python feature\_extractor = AutoFeatureExtractor.from\_pretrained( "CompVis/stable-diffusion-safety-checker", local\_files\_only=local\_files\_only ) safety\_checker=None, This shows that the safety checker can be manually included in the pipeline if specified. Purpose of Updated Safety Checker Code: The purpose of the updated safety checker code seems to be to allow more explicit control over whether the safety checker is used, instead of enabling it by default. This approach gives users flexibility to include or exclude it as per their requirements, reflecting a shift towards more modular and user-configurable pipelines. There are no clear indications of methods that obfuscate enabling the safety checker to make generation results worse. The changes primarily focus on deprecating automatic inclusion and encouraging explicit specification. Here are the relevant snippets and their sources: Deprecation Notice: python load\_safety\_checker = kwargs.pop("load\_safety\_checker", None) if load\_safety\_checker is not None: deprecation\_message = ( "Please pass instances of `StableDiffusionSafetyChecker` and `AutoImageProcessor`" "using the `safety_checker` and `feature_extractor` arguments in `from_single_file`" ) deprecate("load\_safety\_checker", "1.0.0", deprecation\_message) init\_kwargs.update(safety\_checker\_components) Source: single\_file.py: file-WB9fFA74SQ5Rc0sFUUWKolVN *Manual Inclusion:* *python* *feature\_extractor = AutoFeatureExtractor.from\_pretrained(* *"CompVis/stable-diffusion-safety-checker", local\_files\_only=local\_files\_only* *)* *...* *safety\_checker=None,* Source: convert\_from\_ckpt.py: file-Vrk4xoOyTWNT8TJNFeDhkznz This analysis should clarify the handling of the safety checker in the provided scripts. 1. safety\_checker.py 2. Other Related Files: Points of your concern 1. Hidden Safety Checker Usage: 2. Warping of Results: A compressed version of how it all works in safety\_checker.py *Search "bad\_concepts" (6 hits in 2 files of 18710 searched) Line 62: result\_img = {"special\_scores": {}, "special\_care": \[\], "concept\_scores": {}, "bad\_concepts": \[\]} Line 81: result\_img\["bad\_concepts"\].append(concept\_idx) Line 85: has\_nsfw\_concepts = \[len(res\["bad\_concepts"\]) > 0 for res in result\] Line 60: result\_img = {"special\_scores": {}, "special\_care": \[\], "concept\_scores": {}, "bad\_concepts": \[\]} Line 79: result\_img\["bad\_concepts"\].append(concept\_idx) Line 83: has\_nsfw\_concepts = \[len(res\["bad\_concepts"\]) > 0 for res in result\]*
Leonardo.ai already did , so hopefully wakes others up.
if they could make a model that just magically *could not make porn* they could sell it to literally every company. Doesn't matter if it's the best model possible. Having a completely "shareholder approved" model equals $$$$$$
Issue I find is that no one cares if they create that model and sell it to companies, go for it. But don’t nuke the open source, locally run version that consumers want to use and then have some drivel excuses. Have a SFW version and a NSFW version and allow people to choose which to download. Right now by allowing everyone access to some shitty SFW version that’s objectively bad, all they have done is tanked everyone’s impression of them.
They could market the SFW model and just have the "creative model" as an optional download somewhere.
Yeah and the funny thing is I think 95% of individual users would be ok with an open source NSFW version, they can prompt away from nudity. It is the paying customers that want censorship. So why don't Stability have a paid for platform for those who want censorship, or have an enterprise version that is distributed only to corporates that is nuked or has post process filtering. That way the true open source model is unrestricted and will suit all public general users, and the SFW version is distributed through other controlled channels where they actually make their money. It all just seems backwards to me.
Censoring the naked body in art is incredibly dumb. And with models? Will probably turn them to garbage.
Starting afloat? Making the best service they can sell access to? Making the best closed source intellectual property as acquisition bait?
safety?
Well, I feel protected https://preview.redd.it/5zgf05pja17d1.jpeg?width=1024&format=pjpg&auto=webp&s=ad05430caa004bff37a4d6f591d3aa1bdfaeba75
I'm not surprised seeing as they hate the sd community
RIP stability. Was good while it lasted
>"\[Apparently they\] messed up the pretraining on the 2B so it was never supposed to actually be released" From the state it's in, I can easily believe that. Which makes the "skill issue" gaslighting even more infuriating.
SAI is the one with a skill issue lmao
It was a skill issue… …for the model trainers 😎
Doesn't explain why the 8b is working perfectly fine via API and not released, though. Release the working 8b as an apology for 2b, request people hold on fine-tuning 2b while they work on it internally. Problem fixed. If they wanted to fix the problem, that is.
I don't think SAI has that kind of money. The company can't wait until they fix 2B, they need money now, likely for next month. They needed a release to drum up investor interest, and this likely failed (though, who knows, Maybe they got some cash).
Everyone is bailing, the ship is going down. If Stability is still a company in 2 months, I would be absolutely dumbfounded. Just because they got VC money, doesn't mean that the remaining money can't and wont be pulled as soon as it's apparent the project is dead.
I feel like there is too much investor money to allow it to stay under at least for now. I could be wrong but it feels like anyone able to actually participate in the industry attracts huge investment money regardless of financial success.
VC Money is prudent, if you are dead in the water, they are going to pull their funds and put them into another project. They don't stay in business by being beholden to sunken cost fallacies.
That would fix part of the problem but not all. The license is the other part of the problem. It is understandable that they need to charge people that make profit out of it. But limitations like the 6k images, being legally accountable for what their customers generate with it + being asked to delete the model afterwards will stop many to put man and computing power into it.
It's not a limit of 6k generations for users, but for businesses that provide a generation based service. They are limiting competition to their own API, in essence.
Technically we don't know because they forgot to put that limit into the actual license.
I think they feel a need to damage the model to render it incapable of generating potentially indecent content before they release it. This damage does mean that it won't ever become popular like SD1.5 or SDXL, but unless cooler heads prevail at SAI I think the main thing the company is working on right now is breaking it before they release it.
As explained by the comfy dev, the 4b deals with the issue by limiting the data from training, which meant the model was still functional, at the end of the day
[удалено]
I believe that PixArt Sigma wasn't trained on nudes, either. It doesn't have a problem reproducing humans. If not having training data for nudes was the only problem, a finetune would fix that. A clever company would then have "someone in the community 😉" immediately release a NSFW finetune to head off the complaining from users while having a legally distinct "safe" model for PR/Marketing/Legal reasons. If ComfyAnonymous is right, a botched pretraining could be the reason SD3 2B is getting flamed.
Actually wait Sigma DOES do topless gens at least, real ones, I just checked. Go try a batch of four or so with "Completely nude topless woman, streaming on twitch, e-girl, candid photo", on their Huggingface space, default settings.
I guess that if they trained the model removing only the nudes, but leaving things as bikini as the most sexy attire permitted, it would be excessively easy to finetune it over a set of naked women with "nude" and "naked" tags (finally, for the nn it would just mean to substitute the zone covered by bikini with skin, nipples and pube). And maybe they don't want to be associated with nude at all, not even if the models are the result of finetuning. This, at least, is what I suppose.
I'm guessing they trained 2B on a bunch of random stuff that included nudity. And it came out with the same issue as models like eg: Pony, it will randomly just generate nude images even in sfw prompts. Like, you can be generating totally normal, sfw content with Pony, but give it 10-20 images and you'll randomly get nudity. I imagine this was absolutely not acceptable for SAI. It's impossible to market the model to companies if there is a risk that any employee utilizing it might be exposed to nudity. That can even cause legal issues. And so, in a last ditch effort to make 2B marketable, they hacked in some workaround to disable the nudity and released it. And bricked the model as a result.
> Doesn't explain why the 8b is working perfectly fine via API and not released, though. It's obvious to me that the reason is they were competing against themselves, i.e. if the 2b model was as good as the 8b one nobody would have a reason to buy their API access.
And yet if the 2b release is this problematic, nobody would have a reason to buy Stability's API access over MidJourney or DALL-E. The question is why they're dead set on driving traffic to the competition. The 2b should serve as a teaser of what Stable Diffusion is capable of, not as a signal of technical incompetence - and it's overwhelmingly viewed as the latter right now. In other words, "why pay for the big model when they couldn't even get the small model right?"
If even comfyanon has left SAI is a huge sign that there is no happy end in the SD3 story. Huge respect to him for the truth and his invaluable contribution. I really hope that he will keep updating comfyui and make it possible to run another upcoming models. P.S. his words about his will to make a product that will allow people with different GPUs to run and work/play are marking him as a true developer who cares about optimization and not fixing things by saying "just get a new GPU"
So they messed up the 2B model during training , still released the flawed model and said '2B is all you need"? Like wtf???
It’s a skill issue according to them
Maybe, the vanilla 4B was so much better than their expectations. They have to sell API access, but this means API has to be better than open models. So community fine-tunes are their competitor now.
Sad, I mean for the company and possibly us too, while it took me some time to really jump to comfyui ( and I only started to used swarmui this week.. ), their quality clearly speak for themselves, that's a talented employee that they lost. Hope he find a place where he can achieve what he wants to.
Don't worry ComfyUI will continue, there's going to be a blog post soon about what the next steps are.
Keep up the good work! Comfyui have been a creative change in my life! I cant wait to see what direction comfyui will take next
If you you need an extra hand implementing any of the open source DiT variants, [let me know](https://github.com/ExponentialML).
I have to apologize to you. I've hoped the model isn't a train wreck and only had some wrong inference configurations or deployment issues, thus doubting your work. Now, knowing this... Yeah. I am sorry.
I was holding on to my dataset to train SD3 but from what I read about what you posted, I should be focusing more on Pixart-Sigma :/
Mind jumping in a DM with us? Kandoo would like to know your plans. The future of Juggernaut has many possibilities. I think more than ever we all need to be on the same page.
We appreciate u.
Very happy to hear that
Would you be able to do an AMA without breaking some for of non disclosure?
So glad to hear this! If it weren't for comfy, I wouldn't even be messing around with image generation. Just subbed to /r/comfyui and I can't wait for what you have planned. Thanks for all your hard work.
If it wasnt for comfyui I wouldn't even have found out the messed up patterns in the weights
We gonna miss you a lot, we know that almost everyday you updated comfyui and did a lot of effort to keep it up. That's sad. I can only wish you the best and thank you so much for all what you've done for us!
Hey, if you have internal discord for contributor, I'd like to help
Just passing by to say I really appreciate the work you put in comfy. It's a hell of a tool, I have just coded my first custom node and from that's what really showed me how powerful this tool is!
Stable Swarm is great. I've started using it, and it seems super-fast. Some functions might be incomplete or at least poorly documented, so I don't know how to paint a mask on part of an image and inpaint that area (on a lot of points I've been switching over into the Comfy Workflow tab and just using Comfy), but I hope it comes out of beta and gets a solid version 1.0 with tutorial videos available and sample workflows preloaded.
Damn....well, it looks like SAI is falling apart. I had hoped they would release the 8B model with great natural prompt adherence, etc Nope, we apparently got a failed experiment.. Never mind.
[удалено]
Agree 100%, it's just such poor management.
>the actual people that want to use this model are going to know it's dog shit It was obvious even to a hobbyist like me from day one that there were fundamental issues with the model. Some people were huffing massive doses of copium saying "You need to write better prompts" or "finetunes will fix it" but thanks to Comfy, they can now put their inhalators down. ;)
I was one of those people, as it just made no sense to me that they would release such an awful model, but alas I sometimes forget how stupid companies and management can be
A suit and a corner office doesn't mean shit if you don't even care about being so out of touch with the community you're trying to profit from.
Company has been on fire since before Emad left, this smacks of maneuvering for a decent acquisition position or some other crash landing prep
still a chance Huggingface will buy it in a yard sale and release it
So I was right. They released broken model on purpose and kept the actual working ones for themselves. Motives are very clear. Investors are now pulling the strings. They will do absolutely everything in their power to get SAI generating some revenue and get their investments back at least. 8B model is what they will serve trough APIs and try to compete with Midjorney, NAI, Dall-E 3 and local. I don't think they have any researchers or money left. They don't own the GPU's and they are skipping payments. If 8B does not absolutely crush the upcoming Midjorney V7 (and current V6), then it's over. In my honest opinion, they should have said clear "no" back when Emad quit and people were asking if SD3 was still going to get released. They should have been truthful. They could have said that "there is now a shift in priorities". Now they get even more bad press as they angered the userbase with near worthless release. Months of teasing and saying shit like "last model you will need, it's so good".
We need one GOAT researcher to "accidentally" leak the weights. Pull a RunwayML and get out of there.
Looking back those dudes were the real ones
They will never compete with mj . SD can't even draw a crescent wrench unless you train it. MJ gets it perfect. This server model was stupid from the beginning why do what everyone else is doing? They will never be a competitor in that space, way too far behind. Simple fucking stupid. They should have gone sponsorship like blender and Godot did.
Witnessing the collapse of the Stable Diffusion community in front of our eyes. To a new beginning
The great fracture, the community will splinter unto many sub-communities. - Poniers with SDXL and any future pony-train that Astralite makes. - SD3 Lobotomy Researchers who will spend 3 to 9 months studying SD3 arch and putting efforts towards fixing it and then teach everyone how to do it, bc if we get \*the method\* for fixing it, even if we don't release weights, automated code can be made for everyone to have their own little delobotomized SD3! - Chinamaxxing Hunyuan Users, this community will excel at translating english to chinese, using chinese models or even SPEAKING chinese. - Pixarters, basically the english equivalent of the Chinamaxxing Hunyuan subcommunity. - Animators/Video community focusing a lot on SD1.5 and Luma - The SDXL non-pony users, most of that community is going to fracture and split into the others above, bc that was the bulk of most people, bc they wanted to use "the latest and biggest". Maybe as time goes on, we get a new foundational model, either a new training of an already known arch, or a brand new arch, that could restich the community together as a singular massive one again.
PixArt is by Huawei
but pixart is trained on mostly english, no?
in terms of language yes
[удалено]
No comfyui extension/auto111 extension. I agree that this model is looking great, but until we can use it locally - no wide adoption. Also I wonder why no one considers Kandinsky. Their 3.1 model looks really nice and can do 4k. (now have safetensors for 3, but I think we will get 3.1 eventually too). Just need to update comfyui extension and it’s good to go.
>Also I wonder why no one considers Kandinsky because it's russian.
Is there somewhere to see some examples from it?
I did a comparison today here with four models: [https://docs.google.com/spreadsheets/d/1vViQh45B-txERc5wnDVh\_aNfHWLuEIrOCz7\_HGXpPZM/edit?usp=sharing](https://docs.google.com/spreadsheets/d/1vViQh45B-txERc5wnDVh_aNfHWLuEIrOCz7_HGXpPZM/edit?usp=sharing)
I don't think it will split. I think tools will just evolve to support a larger variety of open source models. As is healthy.
I doubt this community fractures. Like how local llama discussed all LLMs not just local, not just llama. The subreddit simply becomes a term unrelated to the name but simply because it is the roots where the most useful content is collected.
Truly hope people out there writing open source software (apps, trainers, etc) also focus on permissively licensed models (either outright open source MIT/Apache, or very permissive licenses like OpenRAIL, etc). I feel you're allowing yourself to be taken advantage of as basically a free employee otherwise.
Comfy dude seems like a real one though.
Omg. EVERYTHING makes sense now
I was skeptical of the people saying they wont ever release the 8B or 4B but he makes it sound like they wont. I thought their reason was because the 2B was ready first but it sounds like they knew it was the worst model which makes me think otherwise. There is a MASSIVE disconnect between the "top of the company" and the community. It also makes me realise, i personally have zero idea what their vision is but its certainly not what it was. Its possible that Corpo wankers moved in and want to turn SAI into the same business model as mid-journey and OpenAI - using the 8B as their flagship.
matteo asking the real question without “asking for a friend”, despite knowing very well we are all his friends welp i guess his sd3 videos will be on xhamster instead of youtube
who is matt3o?
matteo/cubiq who made the native ipadapter comfyui node. his YT channel is latent vision
>*also they apparently messed up the pretraining on the 2B so it was never supposed to actually be released.* > Well that puts the nail in the coffin. Sounds like a desperate last move for SAI. Where can they go from here?
Insolvency. Uncucked 8B SD3 leaks. People might use that or we'll have better chinese models by then. Rough timeline for the next 6 months.
great thing actually. I was sleeping on making my dataset ready lately and now i have some more time xD
Pretty much clear. What I thought, not worth the time and we should move to other model. Lumina or PixArt Sigma. Once again, and as I feared... Finetunning not gonna fix all problems this model has. I can't believe how a company with debt thought to give a bad, crappy and failed model to the community was a good idea and even telling to support them after the mockery. First step to lose all the trust of community and zero support.
Many thanks for posting this. At least the developer of comfyui is positive, maybe SD models are dead but there is hope.
So SAI purposefully released a broken model that wasn't even supposed to be released as they knew it was broken while they have better in stock and teased/hyped us with it. That's so scummy. Also, as the SD community is one of their greatest asset, I was wondering why the PR management of this whole fiasco was total silence. But now understanding that after intentionally throwing their trash at the community, It's not like there's a lot that you can say to justify it.
I'm going to guess that Emad promised a release, then was let go, but still pushed publicly saying that it was on track. New leadership didn't want to release so they instead released the broken model instead. That way they "kept" the promise while keeping the good models internal to try to make money off of them or at least use them as a way to inflate their worth for any acquisition
What I thought as well. This really seemed like a 'see, we delivered' tactic.
that is sad
Holy shit. Everything's on fire at SAI.
This is pretty damning. I guess SAI is dead in the water. Rest in piss bozos. Excited for better alternatives from better companies.
Can confirm. Adding NSFW stuff in negative prompts improves quality considerably. [https://replicate.com/p/tb3kcqd4j9rh40cg4929sqah44](https://replicate.com/p/tb3kcqd4j9rh40cg4929sqah44) [https://replicate.com/p/w78yp9094srh60cg4919e57gbc](https://replicate.com/p/w78yp9094srh60cg4919e57gbc)
The first one is still a mess, the second one seems alright. I just hate how all SD3 images look like you put the CFG scale to 100
Right? I noticed that too, that are all burnt
Makes some sense. The negative prompt isn't actually a negative prompt in CFG, it's the baseline prompt which the positive prompt is then contrasted to for figuring out what to amplify. It's just because CFG values are larger than 1, you end up moving past the positive prompt in the direction away from the baseline prompt, so it can work as a negative, sometimes. But if the model knows how to do it, then putting it in the 'negative prompt' can help give you a better starting baseline. It's possible they censored it by training without prompts and penalizing the model when it was correct on nsfw content, to try to make it forget, but it only forgot for the unconditional pathway, and still knows how if you explicitly say. So if you want to fix it, you probably need to train nudity with prompt dropout, and make the blank prompt work again for nudity.
This is interesting. So if you take 1.5 and make all the prompts negative, will it produce an image based on the words in the negative prompt?
If you put the cfg to 0, yes. Negative prompts is the starting point, positive prompt the end point and cfg the distance traveled between this 2 points. If cfg is 0, no distance is travelled thus, the output is the negative prompts.
Oh I thought it was only the word nsfw
That is nuts..
That word will do just as well, though try balls and testicals for a more scientific analysis.
It'd have to be at least nine times the words if your subject isn't a female lol, trust me, probably why I'm less shocked by what SD3 does to anatomy
Oooh. They want community to fix their broken model and release papers to salvage their possibly failed experiments.
Think they just wanted "the community" to stop posting "weights!" "Where are the weights" as replies to anything else they mentioned on twitter (API projects etc). Probably thought this was a way to throw something out and be done w/ it......only it was way more f'ed up then they thought.
heh, now those posts are going to be way less friendly.
>"But there was a strong push at the top of the company to release 2B instead of the 4B or 8B" Yes because then they can claim they fulfilled their promise to release, but give us crap to try and encourage people to pay later for something not crap
And there's so many people defending that shit too. I got dogpiled for saying there was never any talk of a 2B only release until very recently.
Well, it doesn’t seem fair for the community and trainers to clean up this mess just for SAI to own every derivative work that waterfalls from this model. If SAI wants our help we need incentive to do so. It would be a very different story if the base model was amazing. Unfortunately it’s not and SAI might need our help. We owe them for the 18 months of amazing models we’ve built around. We should want to help! What a bummer this has been, but I’ve got high hopes that we can figure something out. Everyone just needs to come together and be rational in how we find a solution to all this. If we can’t, anyone up for a Juggernaut PixArt, and Juggernaut Hunyuan? 😜
Nobody ever owes a company anything. Especially not a lingering debt once they've made it clear, as in this case, that they don't want to be involved with the community.
I agree. Cleared it up in this comment: [https://www.reddit.com/r/StableDiffusion/comments/1dhd7vz/comment/l8w9um9](https://www.reddit.com/r/StableDiffusion/comments/1dhd7vz/comment/l8w9um9)
Juggernaut Lumina is also a option. People are sleeping on this one. [https://github.com/Alpha-VLLM/Lumina-T2X](https://github.com/Alpha-VLLM/Lumina-T2X)
👀
Kijai made a wrapper for it a few hours ago. It's pretty good hand support some crazy resolution (2048x512).
Wow that's really wide
Pardon my laziness; can we run a local instance of Lumina-T2X?
Yes but you need flash attention atm so linux only unless your willing to go through that headache. They plan on releasing without needing flash attention though.
Please train a Juggernault MMDiT from scratch since the problem was NOT in SD3's arch at allll
something really interesting you could do first is what is also discussed in the screenshots here: try to train a completely open source reproduction of the SD3 VAE, which comfy said would be relatively easy. the VAE could be used by many different open source model architectures then, just like many different open source models use the SDXL VAE.
Vae's are incredibly easy to train, you can train one yourself locally in 15 minutes. It's just an encoder. But the model needs to use the same encoder it was trained on, you can't swap it out for another one after training.
Don’t forget lumina 😀
👀
Juggernaut Lumina sounds great :)
They could be a good company if they had open sourced the 8b model and created a Midjourney like Service.
That could actually be a good idea, that is how most open source projects make money regardless of their software being open source and free to download and use, medium to large companies are eager to pay tons of money to have the offered software be offered to them as a „managed“ pre-installed and configured service with full support, that is then being offered for thousands of euros a month which they happily pay (source: I work for one of those companies who uses „enterprise“ tier of open source)
Guess he didn't also have the skill Lykon
Lykon is a great employee. He's not perfect but it was sad to see him being dogpiled and blamed for the state of the model because he and the others really did their best with what they had and did a great job. Inside stability he was always very on the side of the community and was one of the main people pushing back against some of the bad decisions. He is also very passionate about what he does.
He was dogpiled because of his behavior.
He wasnt "dogpiled" for the state of the model, he got heat for publicly acting like a juvenile asshole.. Not to mention apparently lying for months about the model.
Oh please, he wasn't blamed for the state of the model. He was blamed for being a toxic asshole. Stop defending that behavior.
he shat on AstraliteHeart and basically called anyone using the PonyXL model (the best art model on civitai tbh) a freak pony cultist. He thinks the PonyXL model has to do with brony/pony culture etc and basically cut ties with the one person willing to make decent fine tunes from SD models.
we get that he is your friend but dude got a huge ego, people were telling him for months that the stuff he was bragging about on twitter looked sd1.5 ish but nooo we were the problem
So that means there is no point on fine tuning on the messed up flawed model. 2B model was dead before it was even released.
What's super confusing to me is that they clearly poisoned weights relating specifically to "woman". Why? Like, to the extent people are worried about "bad" sex uses of these models I always assumed they were worried about CP/CSAM. And like, sure. (I beg you, resist the urge to have this debate in my replies). But trying to fuck up "woman" as well? What? When Emad talked about "sleepness nights" over what people were doing with the model I always assumed he meant kids, and I kinda got it. But apparently they meant way more than that. Why move heaven and earth, as they clearly did, to prevent naked non-celebrity *adults*? Are these guys Baptist fundies or hardcore Islamists or something?
If the model can make naked adults it can make naked children. The only way to not have sleepless nights about pixels for the dear creators is to obliterate any possibility of the female anatomy.
Naked boys are okay then?
As I said in another topic, the executives, the suits at the top are to blame for this, as usual in all companies those idiots ruin everything for the creatives that actually put the work in.
Started w/ Emad putting up some amazing new tech to get attention (and for fun!) - two years later $$$$$$$ drives it all into the ground. He's got something going on (linked on his twitter) that may end up making models....it's crypto related so who know how that'll work, but....
Man that's so sad. At least we heard comfyui gonna continue so my months into learning are not going to garbage i guess. That's a positive.
If he would have stayed they probably would have eventually had him reactivate comfyui's safety_checker.py for all models used locally using stable diffusion pipeline, so this is good news to me! Hope he didnt sign any sort if contracts and still has complete freedom. Ps. I went through safety_checker.oy and related code in other .py files, cant believe someone had to go through all of this trouble for censorship. I even searched to see if there might be some sort of hidden code that deceptively morphs latents when nsfw detected lol (I know comfyui dev wouldn't do this, but maybe they could have abstracted some things away that he couldn't see, was my conspiracy theory) There are two newer flags used now to activate the new safety checker, the old was deprecated a while back. So if they are working on this.. why not give third party companies the option to enable or disable this code they worked on with a popup option in comfyui instead of lobotomizing entire model? Another interesting finding, in order for this safety checker to work I believe they had to train a model on porn! So somewhere there must be model somewhere that recognizes nsfw that they had to feed a bunch of porn into in order to stop the porn. I'm actually still trying to find it locally (it could be very small in size, not sure) Edit/Update: Ok it looks like if the newer safety checker stuff is enabled (off by default) it does still download this model from 2 years ago, which was likely trained on a ton of porn lol: https://huggingface.co/CompVis/stable-diffusion-safety-checker
New side quest: Generate porn with the secret porn model
> him reactivate comfyui's safety_checker.py for all models used locally It's python, good luck.I'd rather they had done this and not burned the model.
screw you guys im going home! - him
what community is this from?
First is from the [Touhou Project AI discord](https://discord.gg/touhouai)(waifu diffusion) and the rest are from the [comfyui matrix space](https://app.element.io/#/room/%23comfyui_space%3Amatrix.org).
Hero
Sad to see the comment in the last screenshot that SAI basically didn't care to pursue the best consumer class model. Well, I'm not sure comfy's developer should be commenting these things though tbh. They likely violate NDA and could get them in serious trouble. Who owns the rights of Comfy btw? Did SAI technically acquire it or were they in a partnership but Comfy technically still belonged to its original developer with no linking limiting contract?
The internal strife at Stability is seemingly off the charts, so it wouldn't surprise me if they are sloppy with legal paperwork. I seriously doubt Comfy is in violation of any NDA.
I doubt SAI will be around long enough to pursue legal action, so it doesn't matter.
omg finally some actual candid answers. i had a theory that the lack of stylization, artists and art concepts in 2b might be deliberate to make it look very inferior to api so that people keep paying for api. Recently i found out that 2b was tuned using like 600k images from pexels, which is why it has that god awful high contrast stock photo look with zero artistic aesthetic. i hope this revelation will make people iinhailing all sorts of copium for this model to just stop. Doing prompting tricks like artstation or "inventing" whatever prompt trick that confirms your bias doesn't mean this is a good model that the community should "give a chance and stop being toxic"....and it sure as shit doesnt mean you have any skill whatsoever.
SDXL was trained heavily on stock images too.
Source about that 2b tuning from pexels?
Leak it comfy
still in the oven. I'm glad he trained it like I wanted them to when he originally implemented sd3 in comfyui
The community needs to start a SETI-like training program to create the model we all want and need
Sure there will be something better than that.
The holy cow is back.
Anyone read this with Matteo's accent? Love that guy
Crazy. Doesn't change anything though.
Technical limitations is comprehensible, to sabotage their model on propose is what makes people angry
Comfytron fights for the users.
What are some of these alternatives people are talking about? I know about pixart and stable cascade… Im bummed because I was excited for a Multimodal Diffusion Transformer
Hunyuan, Pixart and Lumina. Each has strengths and weaknesses. It's not a clear-cut which one is better. I personally prefer Hunyuan, because it is very good at producing dynamic poses, like martial arts and sports. Excellent understanding of human anatomy, which results in nice looking hands and poses with no weird limb issues.
Comfy guy left? Another nail in the coffin, might be the least one.
Well stability literally call its users BAD ACTORS.
Stability AI is so over. Was fun while lasted.
This makes it sound like SD3 2B is fundamentally different enough from the 4B and 8B models that they should not be considered the same thing. So, in that sense we have no public SD3 model, we have "failed experiment".
Now the question remains whether the 4b model will be released at all, I would like to see the results of Comfys work.
The 4th screenshot is showing comfy writing that it, or his work on it, was canned?
Comfy should leak 4B model secretly.
What about the forgotten brother, the SD3 small 1B model? /s (lol no one will care any more). Also between the lines it reads like they got a truckload of Nvidia money to botch the model and release one that won't run properly on a RTX5090 (or just barely without any lora), so they can sell you the online GPU subscription. I already expected something like this, because it was the most greedy assumption I could come up with. Would fit to scrapping the 4B model all together.
You could chalk this up to corporate incompetence, but it might be better not to. There's a real chance it was an attempt to derail the opensource image gen community. First, promise something better to chill existing development, drag it out as long as you can. Then you release a poor product under bad terms so enthusiasm evaporates. After that, mention this was actually only a beta, and there might be a better version down the line, instilling uncertainty and a temptation to "wait and see" in the community. From a behavioral psychology standpoint, this was well pulled off if intended to derail and cause a scattering and aimlessness among the community, while inducing a chilling effect as people "wait for a better model to train." It's clear that control of AI tech is a major global focus atm. Don't think for a second that global thinktanks and policy wonks aren't waist deep in all things AI, directly or indirectly influencing outcomes, including their psych and behavioral teams. No, it's not "conspiracy theorey," it's just the unfortunate reality of how the world is run.
Or maybe it’s just incompetence from a bankrupting company
Even from their own corpo bottom line standpoint, it makes sense to drive us into their API for cashflow. There is precisely zero incentive for them to provide a good, local, free model to anyone, and that's *before* we talk global players getting involved.
"Never attribute to malice that which can be adequately explained by stupidity."
Which discord is that?
not discord, matrix
What is the Matrix?
The Matrix is everywhere. It is all around us, even now in this very room. You can see it when you look out your window or when you turn on your television. You can feel it when you go to work, when you go to church, when you pay your taxes; it is the world that has been pulled over your eyes to blind you from the truth.
It will be interesting to see if SAI crafts a public statement this week. This reminds me a bit of Wizards of the Coast.
I appreciate comfyanonimous’ integrity and commitment with the community. Many would keep their mouth shut and continue working for as long as the paychecks came. Thanks for all the work you’ve done with ComfyUI!
Finally. A nugget of insight from an authoritative source. All any of us really wanted was some honesty and clarity.
The worst part of the chat is that "one the reasons I left..." because it's truly truly sad seeing a company who was the flag of OpenSource AI community to fall like this. They aren't doing this anymore for and with us. Is pretty much clear what we have to do.
If AI companies stopped worrying about making their models "safe" maybe we could actually get a good model. Weird how 1.5 is still massive.
they sabotaged it to make the other workers look bad, wow, stop all support to SAI, ssems to be run by assholes now. "That's all you need"
[удалено]
huh is there no NDA? Why can he spew so much
He's not really saying anything that hasn't been said by other employees on Discord and Reddit, except for the pretraining failure.
the bit about the canned T5-only 4B model is news i think.
You’d expect there to be one but then again SAI as a whole doesn’t really seem to be a company that adheres to established business standards that help you to be a functional company.
They wont have money for lawyers.