Image-to-text generation tasks, including image captioning and visual question answering, have been studied for many years. Historically, these systems typically use an object-detection network as the vision encoder to extract visual features, and then generate text using a text decoder. Given the extensive prior literature, this post focuses on a single direction for solving vision-language tasks: extending pre-trained generalized language models so that they can consume visual signals.

I broadly categorize these vision-language models (VLMs) into four groups:

1. Convert images into embedding features that can be trained jointly with token embeddings.
2. Learn high-quality image embeddings that can serve as a prefix to a frozen, pre-trained language model.
3. Fuse visual information into language-model layers using a purpose-built cross-attention mechanism.
4. Combine vision and language models without any training.

Jointly Training with Image and Text

A direct way to incorporate visual information into language models is to treat images as ordinary tokens, then train on a sequence that interleaves text and image representations. Concretely, an image is partitioned into many smaller patches, and each patch is treated as one “token” in the input sequence.

VisualBERT (Li et al. 2019) supplies both text inputs and image regions to BERT, enabling the model to learn internal alignment between images and text via the self-attention mechanism.

As in BERT text embeddings, each VisualBERT visual embedding is also formed by summing three embedding types: tokenized features $f_o$, segmentation embedding $f_s$, and position embedding $f_p$, specifically:

1. $f_o$ is a visual feature vector computed for a bounding region of the image using a convolutional neural network.
2. $f_s$ is a segment embedding indicating that the embedding corresponds to vision rather than text.
3. $f_p$ is a position embedding used to align the ordering of bounding regions.

The model is trained on the MS COCO image-caption dataset, using both text and image inputs to predict text captions, and optimizing two visually grounded language-modeling objectives:

1. MLM with the image. The model must predict masked text tokens, while image embeddings are never masked.
2. Sentence-image prediction. Given an image and two associated captions, one caption may be replaced with a random, unrelated caption with 50% probability. The model must classify which of the two cases applies.

Ablation results indicate that the most important setting is to fuse visual information early within the transformer layers and to pretrain on the COCO caption dataset. In contrast, initialization from a pre-trained BERT and including the sentence-image prediction objective have comparatively small effects.

VisualBERT surpasses the SoTA at the time on NLVR and Flickr30K, but still trails the SoTA on VQA.

SimVLM (Simple Visual Language Model; Wang et al. 2022) is a prefix language model in which the prefix segment uses bidirectional attention (as in BERT), while the main input segment uses causal attention (as in GPT). Images are encoded as prefix tokens, allowing the model to fully ingest visual information and then generate the corresponding text autoregressively.

Motivated by ViT and CoAtNet, SimVLM partitions each image into patches and flattens them into a 1D patch sequence. It uses a convolutional stage consisting of the first three ResNet blocks to extract contextualized patch representations, which was found to perform better than a simple linear projection.

SimVLM is trained on a mixture of image-text pairs from ALIGN (Jia et al. 2021) and text-only data from the C4 dataset (Raffel et al. 2019). The two datasets are mixed within each batch, which contains 4,096 image-text pairs (ALIGN) and 512 text-only documents (C4).

Ablations suggest that including both image-text and text-only data is important. The PrefixLM objective outperforms both span corruption and a naive LM objective.

CM3 (Causally-Masked Multimodal Modeling; Aghajanyan, et al. 2022) is a hyper-text language model trained to generate the contents of large-scale HTML web pages (hypertext markup, hyperlinks, and images) from CC-NEWS and Wikipedia articles. The resulting CM3 models can produce rich, structured, multimodal outputs while conditioning on arbitrary masked document context.

Architecturally, CM3 is autoregressive. However, to combine causal and masked language modeling, CM3 also masks a small number of long token spans and learns to generate those spans at the end of the sequence.

CM3 is trained on close to 1T of web data. During preprocessing, images are first downloaded from src and resized to 256 x 256 with random cropping. They are then tokenized using VQVAE-GAN, yielding 256 tokens per image. These tokens, joined by spaces, are inserted back into the src attribute.

With prompt engineering, CM3 can be applied to several task types:

• Image in-filling:

Infilling Prompt: <figure>
	<img src="{prefix}<mask:0>{postfix}"><mask:0>

• Conditional image in-filling:

Conditional Infilling Prompt:
<figure>
	<img alt="Photo: {text}" src="{prefix}<mask:0>{postfix}"><mask:0>

• Conditional image generation:

Conditional Generation Prompt: <figure>
	<img alt="{prompt}

• Image captions:

Captioning Masked Prompt #1: 
<figure>
	<img alt="Photo: A photo taken of<mask:0>" src="{image}">

Captioning Causal Prompt #1: 
<figure>
	<img src="{image}" title="Photo: A photo taken of

• Entity disambiguation

Original: Manetho writes that these kings ruled from <a title="Memphis, Egypt">Memphis</a>

Prompt: Manetho writes that these kings ruled from <a title="<mask:0>">Memphis</a>...<mask:0>

Target: Manetho writes that these kings ruled from <a title="<mask:0>">Memphis</a>...<mask:0> Memphis, Egypt

Learned Image Embedding as (Frozen) LM Prefix

Suppose we want to adapt a language model to visual inputs without modifying the language-model parameters. In that case, we can instead learn an image embedding space that is compatible with the language model’s embedding space.

Building on prefix or prompt tuning, both Frozen (Tsimpoukelli et al. 2021) and ClipCap (Mokady, Hertz & Hertz, 2021) update only the vision-module parameters during training, producing image embeddings that work with a pre-trained, frozen language model. Both are trained on aligned image-caption datasets to predict the next text token in the caption, conditioned on the image and the preceding text tokens. By freezing language-model parameters, the models preserve the LM’s language capability. Additionally, even though training uses limited image-caption data, inference can still leverage the encyclopedic knowledge present in the language model.

Frozen uses an NF-ResNet-50 vision encoder and takes the NF-ResNet final output vector after the global pooling layer. The resulting Frozen VLM can function as a multimodal few-shot learner, adapting at test time to new tasks for zero-shot or few-shot transfer using sequences that interleave images and text.

Experiments showed that fine-tuning the pre-trained LM can, interestingly, reduce performance on VQA tasks. Initializing the language model from a pre-trained checkpoint is important, since training from scratch (${Frozen}_\text{scratch}$) does not exhibit meaningful progress. The baseline ${Frozen}_\text{train-blind}$ removes the image (by blacking it out) yet still reaches decent performance, reflecting the inherent strength of a pre-trained LM.

ClipCap uses CLIP (Radford et al. 2021) for vision encoding, but applies a lightweight mapping network $F$ so that the image embedding vectors are transformed into the same semantic space as the pre-trained LM. The network $F$ maps CLIP embeddings into a sequence of $k$ embedding vectors, each matching the dimensionality of a GPT2 word embedding. Increasing the prefix length $k$ improves performance. During training, both the CLIP vision encoder and the LM remain frozen, and only the mapping network $F$ is trained. They observed that when the LM is frozen, $F$ should be a transformer (8 multi-head self-attention layers, with 8 heads each), whereas if the LM can be fine-tuned, an MLP suffices.

Although ClipCap trains only a minimal set of parameters, it still achieves strong image-captioning performance, comparable to the SoTA at the time (for example, Oscar, VLP, BUTD). Based on these results, they suggest that “the CLIP space already encapsulates the required information, and adapting it towards specific styles does not contribute to flexibility.”

An additional observation is that, because ClipCap translates CLIP image embeddings into the LM embedding space, the resulting prefixes can even be interpreted as words.

Text-Image Cross-Attention Fuse Mechanisms

To fuse visual information more efficiently across different language-model layers, one can use a specially designed cross-attention fusion mechanism that balances text generation capacity with visual grounding.

VisualGPT (Chen et al. 2021) uses a self-resurrecting encoder-decoder attention mechanism to adapt a pre-trained LM rapidly using only a small amount of in-domain image-text data.

Let $I$ denote the output of a visual encoder, and let $H$ denote the LM decoder hidden state. VisualGPT introduces a self-resurrecting activation unit (SRAU) that controls the tradeoff between a mixture of pre-trained linguistic information $H$ and the visual component $\text{EncDecAttn}(H, I)$, using two complementary gates $B^\text{vis}$ and $B^\text{lan}$:

$ \begin{aligned} & B^\text{vis} \otimes \text{EncDecAttn}(H, I) + B^\text{lan} \otimes H \\ \text{where } & B^\text{vis}[i,j] = \sigma(H[i,j]) \mathbb{1}[\sigma(H[i,j]) > \tau] \\ & B^\text{lan}[i,j] = (1 - \sigma(H[i,j])) \mathbb{1}[1 - \sigma(H[i,j]) > \tau] \\ \end{aligned} $ where $\otimes$ denotes element-wise multiplication, $[i,j]$ denotes an individual element of the matrix, and $\tau$ is a predefined threshold hyperparameter.

VC-GPT (Visual Conditioned GPT; Luo et al. 2022) pairs a pretrained visual transformer (CLIP-ViT) as the visual encoder with a pretrained LM as the language decoder.

CLIP-ViT takes a sequence of image patches as input and produces a representation for each patch. To mitigate catastrophic forgetting, rather than injecting visual information directly into GPT2, VC-GPT adds extra cross-attention layers on top of the outputs of the visual encoder and the language decoder. A self-ensemble module then linearly combines the single-model language-decoder logits $h^G$ and the cross-model vision-language fusion logits $h^\text{fuse}$. The self-ensemble component (see “VC-GPT w/o SE” in Fig. 13) is important for performance.

$ \text{logits} = W^G h^G + W^\text{fuse}h^\text{fuse} $

where $W^G$ is a linear projection of the language decoder, initialized from the GPT2 word embedding matrix, and $W^\text{fuse}$ is a linear projection of the fusion module, initialized randomly.

MERLOT (Zellers, et al. 2021) is trained on 6 millions YouTube videos with transcribed speech (YT-Temporal-180M) to learn both spatial (frame-level) and temporal (video-level) objectives. When fine-tuned, it demonstrates strong performance on VQA and visual reasoning tasks.

Each video $\mathcal{V}$ is divided into multiple segments $\{ \boldsymbol{s}_t \}$, where each segment $\boldsymbol{s}_t$ contains an image frame $\mathbf{I}_t$ (extracted from the middle timestep) and $L=32$ associated word tokens. Images are encoded using a learned image encoder, and words are encoded via a learned embedding. Both modalities are then processed jointly by a vision-language transformer.

MERLOT uses three learning objectives:

1. Masked language modeling (MLM), which is especially helpful because video speech often contains rambling patterns, including repeated keywords and filler words.
2. Contrastive frame-caption matching, which uses the language-only portion of the joint vision-language transformer. Representations for a matched frame $\mathbf{I}_t$ and caption $\boldsymbol{w}_t$ are positive examples, and negatives are all other frame-caption pairs in the minibatch.
3. Temporal reordering, which targets temporal reasoning by scrambling random $i$ frames and replacing segment-level position embeddings with random, unique position embeddings. These random position embeddings are learned, enabling the model to unshuffle these “'shuffled'” frames conditioned on correctly ordered ones. The loss predicts whether $t_i < t_j$ or $t_j < t_i$ for each frame-frame pair.

Ablations indicate that performance depends on: (1) training on videos rather than images, (2) scaling the training dataset in size and diversity, and (3) using diverse objectives to promote full-stack multimodal reasoning.

Flamingo (Alayrac et al. 2022) is a vision-language model that accepts text interleaved with images or videos and produces free-form text. Flamingo connects a pretrained LM and a pretrained vision encoder (for example, the CLIP image encoder) through a transformer-based mapper. To incorporate vision signals efficiently, Flamingo uses a Perceiver-based design to compress a large set of visual input features into a few hundred tokens, then fuses visual information into the language decoding process through cross-attention layers interleaved with LM layers. Training uses an autoregressive NLL objective.

• The Perceiver resampler takes spatio-temporal features from the vision encoder for image or video inputs and produces fixed-size visual tokens.
• The frozen LM is augmented with newly initialized cross-attention layers inserted between the pretrained LM layers, allowing text generation conditioned on the visual tokens.

As in ClipCap, both pretrained models remain frozen during training, so Flamingo learns only to connect two existing, strong language and vision models in a compatible way. The primary difference between ClipCap and Flamingo is that ClipCap uses the image embedding as a simple LM prefix, while Flamingo uses a gated cross-attention-dense layer to fuse image information. Additionally, Flamingo is trained with substantially more data than ClipCap.

To support text interleaved with images, Flamingo uses a masking scheme in which each text token cross-attends only to the visual tokens associated with the most recent preceding image. This substantially reduces how many visual tokens any given text token can access. The authors report that this works better than allowing a text token to attend directly to all preceding images. Text can still depend on all prior images via the causal self-attention dependencies in the text encoder. This design supports an arbitrary number of images in context.

They scraped 43 million webpages, forming the MultiModal MassiveWeb (M3W) dataset, which contains text interleaved with images. Flamingo is also trained on paired image-text and video-text datasets, including ALIGN, LTIP and VTP.

Internet dataset processing includes:

• Insert <image> tags into the webpage text at the positions of visual inputs, and add special tokens <BOS> (beginning of sentence) and <EOC> (end of chunks, always at the end of the document, before any image tag).
• For each document, sample a random subsequence of $L = 256$ tokens, and include up to $N = 5$ images that appear in the sampled sequence (use only the first $N$ within the sampled subsequence if more are present, or pad to $N$ if fewer are present).
• Compute a function $\phi: [1,L] \to [0,N]$ that tracks the interleaving order of text and images by assigning, to each text position, the index of the last image or video that appears before that position (0 if no preceding visual input exists).

Because Flamingo is trained on a mixture of three datasets, it optimizes a weighted sum of dataset-specific NLL losses. Selecting dataset weights is critical for final performance. In practice, rather than sampling datasets round-robin, they sample one batch from each dataset and apply a weighted sum of the gradients at every update. Gradient accumulation across heterogeneous datasets can be viewed as a way to stabilize training by reducing gradient variance between updates.

At inference time, Flamingo naturally supports few-shot learning because it accepts arbitrary sequences of interleaved text and images, and adding more in-context examples improves performance.

Using only few-shot prompting (without any fine-tuning), Flamingo outperforms SoTA fine-tuned models on 6 of the 16 tasks. Fine-tuning Flamingo is expensive and makes hyperparameter tuning difficult, but it does yield additional gains.

CoCa (Contrastive Captioner; Yu & Wang et al., 2022) combines the benefits of contrastive learning with image-to-caption generation. It is trained jointly with a contrastive loss on CLIP-style representations and a generative loss for image captioning, achieving SoTA zero-shot transfer across a broad set of multimodal evaluation tasks.

CoCa is pretrained from scratch using web-scale alt-text data ALIGN and annotated images by treating all labels as text in JTB-3B.

CoCa training includes two main components. The overall loss is a weighted sum of the two losses below, with weight scalars $\lambda_\text{cap}=2.0, \lambda_\text{con} = 1.0$.:

1. $\mathcal{L}_\text{con}$: Dual-encoder contrastive learning optimizes a symmetric contrastive objective, in the same spirit as CLIP.
2. $\mathcal{L}_\text{cap}$: Encoder-decoder captioning trains the decoder to predict a caption conditioned on the latent features produced by the image encoder, by optimizing an autoregressive loss. The text decoder is separated into two parts, unimodal and multimodal. A practical trade-off is to split the decoder evenly between these two components:
   • The lower, unimodal component encodes the input text using causally masked self-attention.
   • The upper, multimodal component applies both causally masked self-attention and cross-attention over the vision encoder output.

On VQA, CoCa outperforms a contrastive-only model and is comparable to a captioning-only model. The captioning loss is also shown to improve zero-shot classification capability.

They introduce task-specific attention pooling (an attention pooler) as a natural task adapter, motivated by the observation that a single pooled image embedding is effective for visual recognition tasks (e.g. ImageNet classification), whereas a higher-resolution embedding is more helpful for multimodal understanding tasks (e.g. VQA). The pooler is a single multi-head attention layer with $n_\text{query}$ learnable queries (note that $\mathbf{X} \in \mathbb{R}^{L \times d}$, $\mathbf{W}^q \in \mathbb{R}^{d \times d_q}$, and $d_k = d_q$), and it uses the encoder output as both keys and values. CoCa uses attentional poolers during pretraining for the generative loss $n_\text{query} = 256$ and the contrastive loss $n_\text{query} = 1$. This design allows the model to achieve strong performance as a frozen encoder, where only a new pooler is learned to aggregate features.

No Training

It is also possible to address vision-language tasks by combining pretrained language and vision models without training any additional parameters.

Decoding Guided with Vision-based Scores

MAGiC (iMAge-Guided text generatIon with CLIP; Su et al. 2022) performs guided decoding using a CLIP-based score called the magic score to sample each next token, without fine-tuning. This encourages the generated text to remain relevant to the input image while staying coherent with the already generated context.

At time step $t$, the next token $x_t$ is selected according to the equation below. To reduce corrupted generations from the language model, the method incorporates model confidence and a degeneration penalty (Su et al. 2022).

$ \begin{aligned} & x_t = \arg\max_{v \in \mathcal{V}^{(k)}} \big\{ (1-\alpha) \underbrace{p(v \vert \boldsymbol{x}_{

where $\mathcal{I}$ denotes the input image; $\mathcal{V}^{(k)}$ includes the top-$k$ candidate tokens predicted by the language model $p$; $\boldsymbol{x}_{

MAGiC achieves solid results relative to other unsupervised approaches, although it still lags significantly behind supervised methods.

Language as Communication Interface

For knowledge-based VQA, PICa (Prompts GPT-3 via the use of Image Captions; Yang et al. 2021) first converts images into captions or tags, then uses few-shot examples to prompt GPT3 to produce answers. Captions or tags are obtained using existing systems (e.g. VinVL) or the Azure Tagging API. In this setup, GPT3 is treated as an unstructured, implicit knowledge base.

PICa investigates two techniques for improving few-shot prompting to obtain stronger results:

• In-context examples are selected based on their similarity to the question, using CLIP embeddings.
• Multi-query ensembling prompts the model multiple times to produce multiple candidate answers, then selects the answer with the highest logprob.

With only 16 examples, this straightforward method improved SoTA on OK-VQA by +8.6 points and achieved competitive performance on VQAv2.

Socratic Models (SM) (Zeng et al. 2022) is a framework for composing multiple pretrained models across modalities into a single system via language (prompting), without any additional training. In this approach, language serves as the intermediate representation through which models exchange information. The central idea is multi-model multimodal prompting, where the output of a non-language model is inserted into a language prompt that the LM then uses for reasoning.

Consider a concrete example. Given an egocentric video (images + audio), SM can generate a summary of the person’s activity by combining a text-to-text LM, a image-to-text VLM, and a speech-to-text ALM. These components are chained as follows:

1. the VLM detects visual entities;
2. the LM proposes sounds that might be present;
3. the ALM selects the most likely sound;
4. the LM proposes possible activities;
5. the VLM ranks the most likely activity;
6. the LM produces a summary of the Socratic interaction.

For image captioning, SM first uses the VLM to zero-shot predict place categories, object categories, image type, and the number of people. Next, it inserts these VLM outputs into a language prompt, which is then provided to a causal LM to generate caption candidates. This Socratic approach remains behind ClipCap on image captioning, but it is notably strong given that it requires no training.

The SM framework is highly flexible and extends to tasks beyond image captioning. For example, the egocentric perception task (User inputs + VLM + LM + ALM) takes egocentric videos as input to: (1) summarize content; (2) answer free-form reasoning questions; and (3) perform forecasting.

Datasets

Image Caption Datasets

• MS COCO (Chen et al. 2015): includes 328K images, each paired with five independent captions.
• NoCaps (Agrawal et al., 2019) is intended to evaluate generalization to unseen classes and concepts. The in-domain split contains images depicting only COCO classes, the near-domain split contains both COCO and novel classes, and the out-of-domain split consists exclusively of novel classes.
• Conceptual Captions (Sharma et al. 2018) contains 3 million image-caption pairs, mined from the web and post-processed. To emphasize concepts, specific entities are replaced with more general terms (e.g. a politician’s name is replaced with “politician”).
• Crisscrossed Captions (CxC) (Parekh et al. 2021) provides 247,315 human-labeled annotations, including positive and negative associations among image pairs, caption pairs, and image-caption pairs.
• Concadia (Kreiss et al. 2021) is a Wikipedia-based dataset containing 96,918 images with corresponding English descriptions, captions, and surrounding context.

Pair Image-Text Datasets

(*) Not a public dataset.

• ALIGN (Jia et al., 2021) contains 1.8 billion images paired with alt-text. The dataset is large but noisy, with only minimal frequency-based filtering.
• (*) LTIP (Long text & image pairs; Alayrac et al. 2022): 312 million images paired with descriptive captions.
• (*) VTP (Video & text pairs; Alayrac et al. 2022): 27 million short videos (~22 seconds on average), paired with descriptive captions.
• (*) JFT-300M / JFT-3B are internal Google datasets that contain 300M / 3B images annotated via a semi-automatic pipeline with a class hierarchy of around 30k labels. As a result, both the data and associated labels are noisy.

Evaluation Tasks

Visual Question-Answering

Given an image and a question, the goal is to produce the correct answer.

• VQAv2 (Goyal et al., 2017) includes 1+ million questions about 200K COCO images.
• OK-VQA (Marino et al. 2019) contains 14K open-ended questions that require external knowledge (e.g. Wikipedia).
  • A-OKVQA: the augmented successor to OK-VQA, with no overlapped questions with OK-VAQ.
• TextVQA (Singh, et al. 2019) contains 45,336 questions over 28,408 images, requiring reasoning over text in the image.
• VizWiz (Gurari, et al. 2018) contains over 31,000 visual questions from blind users, who each took a photo on a mobile phone and recorded a spoken question about it, along with 10 crowdsourced answers per question.

Visual Language Reasoning

• VCR (Visual Commonsense Reasoning; Zellers et al. 2018) contains 290k multiple-choice QA items derived from 110k movie scenes, with an emphasis on visual commonsense.
• NLVR2 (Natural Language for Visual Reasoning; Suhr et al. 2019) provides 100k+ sentence and web-image examples; the task is to determine whether a natural-language statement is true for a pair of images, with a focus on semantic diversity.
• Flickr30K (Jia et al. 2015) includes 30k images from Flickr and 250k annotations; the task is to select bounding regions given spans in a sentence.
• SNLI-VE (Visual Entailment; Xie et al. 2019) is built on SNLI and Flickr30K, and the task is to infer the relationship between an image premise and a text hypothesis.

Video QA and Understanding

• MSR-VTT (MSR Video to Text; Xu et al. 2016) contains 10K web video clips totaling 41.2 hours and 200K clip-sentence pairs; the task is to translate videos into text.
• ActivityNet-QA (Yu et al. 2019) includes 58,000 human-annotated QA pairs over 5,800 videos derived from the ActivityNet dataset.
• TGIF (Tumblr GIF; Li et al. .2016) contains 100K animated GIFs and 120K sentences describing their visual content, collected from posts published on Tumblr between May and June 2015.
  • TGIF-QA contains 165K QA pairs for the animated GIFs in TGIF.
• LSMDC (Large Scale Movie Description Challenge; Rohrbach et al. 2015) contains 118,081 short video clips extracted from 202 movies. Each clip has a caption, either extracted from the movie script or transcribed from DVS (descriptive video services) intended for the visually impaired.
• TVQA (Lei et al. 2018) / TVQA+ (Lei et al. 2019) is a large-scale video QA dataset drawn from six popular TV shows (Friends, The Big Bang Theory, How I Met Your Mother, House M.D., Grey’s Anatomy, Castle). It includes 152.5K QA pairs from 21.8K clips, spanning more than 460 hours of video.
• DramaQA (Choi et al. 2020) is a large-scale video QA dataset based on the Korean TV show “Another Miss Oh”. It includes four difficulty levels and multi-level, character-centered story descriptions.
• VLEP (Video-and-Language Event Prediction; Lei et al. 2020) contains 28,726 future event prediction examples (with rationales) from 10,234 diverse TV show clips and YouTube lifestyle vlog clips.

Citation

Cited as:

Weng, Lilian. (Jun 2022). Generalized visual language models. Lil’Log. https://lilianweng.github.io/posts/2022-06-09-vlm/.

Or

@article{weng2022vlm,
  title   = "Generalized Visual Language Models",
  author  = "Weng, Lilian",
  journal = "Lil'Log",
  year    = "2022",
  month   = "Jun",
  url     = "https://lilianweng.github.io/posts/2022-06-09-vlm/"
}

References

[1] Li et al. “VisualBERT: A Simple and Performant Baseline for Vision and Language.” arXiv preprint:1908.03557 (2019).

[2] Wang et al. “SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.” ICLR 2022.

[3] Aghajanyan, et al. “CM3: A Causal Masked Multimodal Model of the Internet.” arXiv preprint arXiv: 2201.07520 (2022).

[4] Tsimpoukelli et al. “Multimodal Few-Shot Learning with Frozen Language Models.” NeuriPS 2021.

[5] Mokady, Hertz & Hertz. “ClipCap: CLIP Prefix for Image Captioning.” 2021.

[6] Chen et al. “VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning.” arXiv preprint arXiv:2111.09734 (2021).

[7] Luo et al. “A Frustratingly Simple Approach for End-to-End Image Captioning.” arXiv preprint arXiv:2201.12723 (2022).

[8] Zellers et al. “MERLOT: Multimodal neural script knowledge models.” NeuriPS 2021.

[9] Alayrac et al. “Flamingo: a Visual Language Model for Few-Shot Learning.” arXiv preprint arXiv:2204.14198 (2022).

[10] Yu & Wang et al. “CoCa: Contrastive Captioners are Image-Text Foundation Models.” arXiv preprint arXiv:2205.01917 (2022).

[11] Yang et al. “An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.” arXiv preprint arXiv:2109.05014 (2021).

[12] Su et al. “Language models can see: Plugging visual controls in text generation.” arXiv preprint arXiv:2205.02655 (2022).

[13] Zeng et al. “Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.” arXiv preprint arXiv:2204.00598 (2022).