Keep the 7B DiT as the renderer, make a frozen LLM the conditioner, and bridge them with a small bidirectional refiner

Asked (summary):

What are the most promising research directions for integrating a ~7B text-to-image diffusion model into a pretrained LLM for efficient image generation and editing — ideally throwing away the original text encoder?

This report distils 30 ranked technical precedents (15 shortlisted with bounded summaries, 13 audited primary sources) on replacing or removing the diffusion text encoder with a pretrained LLM/VLM. Sources are arXiv papers plus one project page; results are as reported by the authors, not independently reproduced.

The recommended retrofit path, and where the branches fork

Boxes are stages; green is the low-risk retrofit spine, amber the high-risk unified-transformer branch. Hover a stage for the precedents behind it.

low-risk retrofit (do first) high-risk unified transformer (later ablation) deleted component

Evidence matrix: what each precedent reports

Green = low-risk retrofit precedent, amber = unified-model precedent, grey = not reported in the row. Hover a dot for the reported detail.

The 15 shortlisted precedents

Bounded technical summaries; two works appear under more than one source document.

Work	Route	Text-encoder strategy	Reported result	Efficiency fact · source

Method: built from 30 ranked technical precedents on LLM-conditioned diffusion, a deduplicated 15-row shortlist with bounded summaries, and a 13-entry source audit. Fields record each paper's architecture, text-encoder disposition, freezing/initialization, editing design, efficiency and results as the authors report them (arXiv papers and one project page; no result independently reproduced; no single URL backs more than half the rows). The matrix shows presence of a reported field, not its quality. Long result strings are truncated for space; full detail is at the linked sources. Compiled December 2025.

This report was generated automatically by Keenable SELECT at a user's request, from publicly available web sources linked herein. Keenable does not review, verify, or endorse its contents and makes no representation as to accuracy, completeness, or timeliness; AI-based extraction may contain errors. Nothing in this report is investment, legal, financial, or other professional advice. All trademarks and referenced content remain the property of their respective owners; no affiliation or endorsement is implied. To report an error, rights concern, or request removal: legal@keenable.ai.