Aah, for this we're just trying to filter not generate. When it comes to conditioning, you'll still need a model that understands text since the primary control is text.

In the original Stable Diffusion, CLIP doubled as part of this LAION Aesthetic Filter as well as the text control for the generative model. But in the last couple of years, the field has drifted away from CLIP towards LLM hidden states (e.g. Qwen, Mistral, etc.). It turns out that text-to-image and text-to-video models do not need the visual prior from CLIP; instead the hidden states of multimodal LLMs act as better conditioning mechanisms.

But you are on-to something when it comes to leveraging pre-trained vision models as priors in generative image/video.

Nowadays, models like Dino-v3 (which have the strong spatial priors, you're talking about with LingBot) are being used in 3 distinct ways:

1) REPA: The DiT in the generative model minimizes cosine distance to another vision prior, like Dinov3. This paper from earlier this year, iREPA (https://arxiv.org/pdf/2512.10794) digs into WHAT specifically makes a good regression target for generative image.

2) VA-VAE: Most generative image/video models work in a latent space that's pre-trained separately. We wrote a lot about this in an earlier blog post (https://www.linum.ai/field-notes/vae-reconstruction-vs-gener...). But basically, VAEs that overfit to the reconstruction task create brittle latent spaces for generative models. One solution folks have figured out is to regress to a strong spatial prior like Dino, during VAE training. This is discussed at length in the VA-VAE paper(https://arxiv.org/abs/2501.01423). Recent image models like Flux 2, likely achieve their improvements through this type of VAE training.

3) RAE (Representation Autoencoder): This is the latest/greatest right now. Where generative image models are trained directly in Dino-v3 space; and then they just train a decoder to read back out into pixel space. (https://arxiv.org/pdf/2605.18324)

Hopefully this helps!

Getting video models to learn better, faster