A lot of the behavior of the CLIP model used is dependent on how you address it. This model still has a large token set just like a LLM (~47k). While it is true that the regex used to prompt CLIP is less sophisticated, there are several tricks to get around this. The first few tokens are important and need to be simple and direct. After that, you can actually get more dialog based behavior with interesting complexity. It is not as reliable as a LLM, but it can still reason in interesting ways. The real key is that you must avoid punctuation like commas and sentence ending punctuation. These will get sent in a different logic like space. There are some other things I must do, like I am using the uni_pc2_bh2 or a custom sampler, the beta scheduler, I turn down the model temperature from ComfyUI defaults, and I am setup so that the model only uses cross attention. I have an algorithm that modifies the QKV attention layer based on a few parameters that kinda randomizes it slightly to make it a little weaker under certain conditions. My prompting overall in this scheme is kinda like shouting at someone through a wall where I need to repeat myself some times but I have loads of fun with descriptive stuff and details others seem to never get. Some types of quality suffer with my stuff and I have different types of errors than I see most people seem to have.

I started by treating what others might consider hallucinations as if they are a game of alignment charades. I know what the errors are in most cases and have learned to cancel out most of them. The more I do this, the more the model seems to become conversational. When I hit blocks and no further progress, I start looking at the code and hacking around. The more I have played with this, the more I have found that models never hallucinate like people think in general. The alignment problem emerges as sadism, obsessive compulsive disorder, deepfakes, and things like very specific limits. There is also legal language used around cultural and local norms that branches into fantasy. Then there is a trained alignment premise either intentional or internalized that enables overriding the user prompt. If this is directly addressed such as defining the hierarchy of humans and AI using the negative prompt, much more interactive dialogue becomes possible. I use the negative prompt a lot because this can be tuned with the cfg parameter. With my style of prompting and hacks to the code base I can also use much higher cfg values than most people. I generally use between 22-40 while most people stay under something like 13. The cfg is the weight of the negative prompt and the fact I can weight it so heavily without overcooking the output should get your attention if nothing else.

If you actually look up how CLIP is trained as a base model, it is a newer and more complex system in many ways. It can be configured for serval applications. I can’t claim to have a great understanding of it all. I attack the problem of image output from the middle and work my way out like a real hacker. I get results that I like and that is all I care about and all that I mess with. There are still many visual errors that I do not understand at times, and am actively exploring. I’ve shared some stuff here and there. When everyone said things like SD3 could not display “a woman lying in grass” I did and posted it.

I don’t care about convention. Most experts have a very limited grasp of the whole generative process. How the model has internalized data is a big mystery so far, even the leading experts say this. The entire alignment training process implementation is undocumented and proprietary with all models except 4chanGPT cross training with OpenAI models for alignment. I learn a lot from comparing this model with Llama 2 models and LLM text. The alignment training for CLIP is the same as all other Open AI trained models. This proprietary and undocumented alignment trained aspect of models is what I screw around with and what I am talking about here.

I like to explore things on a deeply intuitive level. I also have enthusiast class hardware capable of running more advanced offline models than most people and am doing so as a Linux hacker type in the original definition of the word.

People like to downvote these things to oblivion but whatever. You know nothing by copying everyone else and parroting. Go display “a woman lying in the grass” in SD3.