Search

GitHub - IFAKA/platonic-lab: Laptop fact check: do good views beat next-token prediction at aligning text models with an image model? (Balestriero & LeCun 2026) · GitHub

A human-caption-based replication of a claim about text-image alignment shows good captions better align text models to an image model than bad captions or next-word prediction, with detailed results across seeds and setups using a small model and COCO data. The study reports specific R@1 and mutual 10-NN scores, and notes the identical training setup aside from caption pairs.

[2610.37341] CodeCap-Bench: Can Code Serve as a Good Video Caption?

CodeCap-Bench studies using executable code as video captions to better capture spatial and temporal information, showing improved reasoning and reconstruction but revealing current multimodal models struggle with temporal dynamics.

[2611.06083] SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

SleuthVLM demonstrates that direct audio input is not necessary for omni video understanding; a text-transcribed audio route with a capable agent can outperform audio-visible baselines on multiple benchmarks, suggesting focus on agentic ability over data scale. The work reports leading accuracy on four benchmarks and a top score on Video-MME-v2, advocating simpler, text-based audio integration and stronger agents for omni video tasks.

GitHub - ahkamboh/agent-caption: Add captions to any video or song, in any language — 100% on your device, no cloud. Hinglish-first. Free, open-source, and works with any AI agent. · GitHub

Open-source, on-device captioning tool that produces accurate, timed captions for video and audio in many languages, with Hinglish support and music captioning features. It uses Whisper/WhisperX for transcription, Demucs for vocal isolation, and offers multiple caption styles and a CLI workflow for integration with AI agents.

[2610.33472] Qwen-Image-PE: Learning Visual Design for Text-to-Image Prompt Enhancement

Qwen-Image-PE introduces a two-stage training approach to improve visual design for text-to-image prompts, using dense captions for supervision and groupwise rewards for refinement. It demonstrates strong gains across benchmarks and backbone generalization, and announces the release of Qwen-Image-PE-9B.

[2611.04916] Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

The report analyzes how semantic control and multimodal understanding shape video generation, detailing a chain from data to evaluation. It outlines Kling-MLLM’s unified multimodal foundation, captioning strategies, prompt enhancement, perceptual-focused text encoding, reward modeling, and evaluation protocols to improve controllable video synthesis.

[2611.03602] Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning

Reviso is an active, priority-guided video captioning framework that learns what to observe in videos. It builds a hierarchical observation plan, uses goal-directed perception to gather evidence, and is trained with verification-guided trajectories and GRPO rewards to improve caption quality and downstream QA performance.

GitHub - ghost1412/shortsflow: Self-hosted Opus Clip / Klap / Vizard alternative: turn long videos into captioned 9:16 Shorts with local Whisper, face tracking, active-speaker cuts and Remotion captions. MCP-ready, MIT. · GitHub

ShortsFlow is an open-source, self-hosted tool for turning long videos into captioned vertical clips with local transcription, face tracking, and speaker-based cuts. It offers desktop, command-line, browser, and API options, with external AI services available as optional features.

{ "id": "e8f30439-267f-45de-b260-db0d114b2c85", "revision": 0, "config" - Pastebin.com

The document defines how to convert a rough Chinese image request into a finished English caption for image generation, with strict rules on structure, preservation of user content, and a multi-facet description approach. It prohibits edits or explanations and requires the caption to describe the final image in detail.

A story of developing an image encoder on a full scratch that rivals SigLIP2

Translated title. Original: SigLIP2に匹敵する画像エンコーダーをフルスクラッチで開発した話

Noetra builds a domestic, transparent multimodal LLM by training an image encoder from scratch, using curated, filtered interleaved data and Recaption-generated captions to reach high performance on Japanese benchmarks while matching SigLIP2 in English. The approach emphasizes data quality, simple contrastive learning, and evaluation across multilingual benchmarks before integrating the encoder into a multimodal model.