No found humor dataset clears rights, realism and preferences at once — build a rights-first hybrid corpus
Asked (summary):
Find real-world photo humor datasets for multimodal SFT and DPO — dataset discovery, licensing verification, and a small real-data pilot, reporting concrete findings before scaling. Goal: at least 1,000 diverse real photographs with high-quality humorous captions and preferably 3,000–5,000 human-validated preference pairs, without starting model training.
The evidence is a 30-row candidate and source shortlist spanning 20 independent hosts, with no URL backing more than one row. Each row reports media type, image and caption scale, human signals, stated rights, and access method. Many rows are single-source claims; blanks are reported as “not verified”, never as “no”. No files were downloaded and no visual sampling was executed — all quality judgments below are desk research.
Pilot not executed. No files were downloaded in this research environment, no 100-image gallery was generated, and the requested 30-random-example visual inspection per dataset was not completed. Sample-quality conclusions are desk-research assessments, not image-by-image validation.
Decision: three sourcing strategies compared
Ordinal desk-research assessment (low / medium / high) of each strategy against the five decision criteria from the comparison above — not measured scores. C uses cleared existing examples as seed and evaluation data and builds the production core from 1,000–1,500 per-item-authorized photographs with 4–6 generated caption candidates each and three-person human ranking.
Findings the chart cannot say
OxfordTVG-HIC reports 2,885,326 image-text pairs over 53,728 unique images (~53.7 captions per image) with “humour scores”, but it is mixed-media — photos alongside paintings and animation — and nothing verifies the scores as direct human preference labels. arxiv.org
RedCaps offers 12,011,111 real Reddit image-caption pairs with community score signals, but images are URL-only, commercial redistribution is prohibited, and use is reported as non-commercial academic research — unusable for a commercial corpus without clearance. gwern.net
BoketeDB reports 999,571 Japanese funny captions over 70,981 mixed images with community stars — structurally the best DPO match found — but redistribution and training rights are not established; do not download or train absent permission. arxiv.org
FlickrStyle10K pairs 10,000 real Flickr photos with stylized captions including one humorous description per image, but only the 7,000-image training portion is reported public, it carries no per-item humor preferences, and upstream rights need verification. microsoft.com
Candidate table — all 30 shortlist rows
Click a header to sort; type to filter. Blank source claims appear as “not verified”. Dataset | real photos | images | captions | human scores | training rights | downloadable | source — the requested comparison columns.
Dataset
Real photos
Images
Captions
Human scores
Rights status
Downloadable
Source
OxfordTVG-HIC: downloadable, not yet trainable
The official repository advertises working Google Drive scripts, about 2.9M image-text pairs and an MIT license — but the MIT license covers the code and repository material, not the rights to every underlying image and caption. The corpus is mixed-media, and the advertised humour scores lack a verified protocol establishing them as human preference labels. Classify HIC as technically downloadable and potentially valuable for experimentation, but legally and compositionally unverified for production training until a file-level provenance and license audit and a random visual sample are completed. Repository MIT status is not training permission for underlying media. github.com · arxiv.org
Rights traffic-light across the 30 rows
Stated training rights classified conservatively from each row’s own text. “Not verified” means the row stated nothing — it never means “no”. Even “clear licence signal” (e.g. CC BY 4.0 annotations, CC0-filtered APIs) still needs per-image provenance before commercial training; annotation licensing does not settle right-of-publicity or image copyright.
Photo-source recommendation: prefer per-item public-domain / CC0 or clear CC BY assets from Wikimedia Commons or Openverse, plus a carefully filtered commercial-use subset of Flickr/YFCC with license snapshots. Treat CC BY-SA separately (share-alike needs counsel); exclude NC and ND assets unless counsel approves. Do not assume Unsplash or Pexels are training-safe — obtain written confirmation on ML training, bulk collection and redistribution. Preserve source URL, creator, license URL and version, retrieval timestamp, original and derivative file hashes, attribution text, and release notes.
Sample quality — sampling not executed
Photographs: FlickrStyle10K, Personality-Captions, captionsDB, RedCaps and YFCC/Flickr subsets report ordinary real photographs — none inspected visually here; characterization rests on the source documents.
Mixed media: OxfordTVG-HIC and BoketeDB mix photos with paintings, animation and composites; composition is unverified pending a random visual sample.
Cartoons and memes: New Yorker-style caption contests and Eimage are cartoons or meme images — no remedy for the real-photo domain gap.
The requested 30-random-example inspection per dataset was not completed; the 100-image gallery is pending.
Cost and effort, 1,000-image pilot-to-corpus
5 candidate captions per image → 5,000 captions to review.
Three reviewers at ~1–2 minutes per image ≈ 50–100 reviewer hours, plus 20–50 hours for adjudication, QA, rights and engineering.
Annotation/QA cash ≈ US$2,000–$6,000 by reviewer geography; engineering and legal review additional.
3–5 accepted rank-separated pairs per image targets 3,000–5,000 DPO pairs; expect to source 1,200–1,500 photos to net 1,000 usable images.
Gated next step: the 100-image pilot (pending)
Pull 100 images from one explicitly approved source — ideally 50 ordinary scenes and 50 potentially humorous or surprising scenes.
Validate MIME/decode, dimensions, perceptual hashes, license metadata, provenance, image-caption alignment, people/privacy flags and corruption; then generate a contact-sheet gallery for human review. The samples/ manifest and gallery are specified but pending — not generated.
For each approved photo, generate 4–6 one-line caption candidates from diverse prompts and models; use an LLM only to remove policy violations, duplicates, generic captions and grounding failures.
Three independent humans rank candidates for funniness and visible-image grounding, with a “none are good” option; one gold SFT caption only when quality passes; 3–5 chosen/rejected DPO pairs per image from rank-separated candidates, avoiding near-ties.
Include ordinary photos, not only inherently funny ones; audit for cruelty, protected-trait humor, sexualization and invented sensitive attributes.
Do not start training or bulk collection until the pilot, the rights audit, and the human-ranking calibration all pass.
Method: desk research over a 30-row candidate/source shortlist spanning 20 independent hosts (max one row per URL), compiled from official project pages and academic papers where available. Numbers are the sources’ reported figures (image counts, caption counts, human-signal descriptions); blanks are shown as “not verified”. Rights status is a conservative reading of each row’s stated text, not legal advice. No files were downloaded and no visual sampling was performed; long cell texts are truncated in the table for space — the linked source carries the full claim.