Llama 3.2-11B Vision-Instruct

The Llama 3.2-Vision-Instruct model is a multimodal large language model (LLM) that can process both text and images to generate text. This model is part of the Llama 3.2 family, developed by Meta, and is designed for commercial and research applications. Llama 3.2-11B Vision-Instruct is in 11B parameter size. It is optimized for visual recognition, image reasoning, captioning, and answering questions about images. The model is built upon the Llama 3.1 text-only model, incorporating a vision adapter for image processing. The model use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF).

Key Features of Llama 3.2-11B Vision-Instruct

• Multimodal Input: Takes both text and images as input.

• Output: Generates text as output.

• Image Reasoning Tasks: Supports Visual Question Answering (VQA), Document Visual Question Answering (DocVQA), Image Captioning, Image-Text Retrieval, and Visual Grounding.

• Language Support: For text-only tasks, it officially supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. For image+text, only English is supported.

• Context Length: 128

Technical Specifications

• Architecture: Uses an auto-regressive language model with an optimized transformer architecture.

• Vision Adapter: Employs a separately trained vision adapter consisting of cross-attention layers to integrate image encoder representations into the core LLM.

• Training Data: Trained on 6 billion (image, text) pairs, with a data cutoff of December 2023. Instruction tuning data includes public vision instruction datasets and over 3 million synthetically generated examples.

• Inference Scalability: Uses Grouped-Query Attention (GQA).

Intended Use Cases

• Commercial and research use

• Visual recognition, image reasoning, captioning, and assistant-like chat with images.

• Adaptable for a variety of image reasoning tasks.

• Leveraging model outputs to improve other model.