# documents

id: 527bb3ad-5f68-4888-b783-41ea610651ee
filename: 2503.11576v1-1-2.pdf
content_type: application/pdf
created_at: 2026-03-23 09:46:50.094408+00:00
storage_path:
/Users/pjmalandrino/Documents/Pro/workspace/poc/Docling-Studio/document-parser/uploads/1f5e4880-d8f5-44fe-8324-b8b33d50e691.pdf


# analysis_jobs

id: 516d470c-7c03-4053-9c45-17a831c2cbfd
document_id: 527bb3ad-5f68-4888-b783-41ea610651ee
status: COMPLETED
started_at: 2026-03-23 09:47:01.335978+00:00
completed_at: 2026-03-23 09:47:03.325503+00:00
created_at: 2026-03-23 09:47:01.327030+00:00
content_markdown:
## SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

Ahmed Nassar 1 , Andres Marafioti 2 , Matteo Omenetti 1 , Maksym Lysak 1 , Nikolaos Livathinos 1 , Christoph Auer 1 , Lucas Morin 1 , Rafael Teixeira de Lima 1 , Yusik Kim 1 , A. Said Gurbuz 1 , Michele Dolfi 1 , Miquel Farré 2 , Peter W. J. Staar 1

IBM Research 1 , HuggingFace 2

https://huggingface.co/ds4sd/SmolDocling-256M-preview Version 1.0, March 14 th 2025

## Abstract

Weintroduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms - significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.

## 1 Introduction

For decades, converting complex digital documents into a structured, machine-processable format has been a significant technical challenge. This challenge primarily stems from the substantial variability in document layouts and styles, as well as the inherently opaque nature of the widely used PDF format, which is optimized for printing rather than semantic parsing. Intricate layout styles and visually challenging elements such as forms, tables, and complex charts can significantly impact the reading order and general understanding of documents. These problems have driven extensive research and development across multiple domains of computer science. On one hand, sophisticated ensemble systems emerged, which decompose the conversion problem into several sub-tasks (e.g. OCR, layout analysis, table structure recognition, classification) and tackle each sub-task independently. Although such systems can achieve high-quality results for many document types while maintaining relatively low computational demands, they are often difficult to tune and generalize.

On the other hand, in the recent past, great interest has developed around large foundational multimodal models that can solve the whole conversion task in one shot, while simultaneously offering flexible querying and parametrization through prompts. This approach has been made possible by the advent of multimodal pre-training of large visionlanguage models (LVLMs), which opened up a vast array of opportunities for leveraging diverse data sources, including PDF documents. However, the literature on this topic highlights a significant gap in the availability of high-quality and open-access datasets suitable for training robust multi-modal models for the task of document understanding. Furthermore, relying on LVLMs may introduce common issues associated with such models, including hallucinations and the use of significant computational resources, making them impractical from both a quality and cost perspective.

Figure 1: SmolDocling/SmolVLM architecture. SmolDocling converts images of document pages to DocTags sequences. First, input images are encoded using a vision encoder and reshaped via projection and pooling. Then, the projected embeddings are concatenated with the text embeddings of the user prompt, possibly with interleaving. Finally, the sequence is used by an LLM to autoregressively predict the DocTags sequence.

<!-- image -->

In this work, we outline how we close the gaps left by publicly available datasets and establish a training approach to achieve end-to-end, full-featured document conversion through a vision-language model, which we name SmolDocling . Our main contributions can be summarized as follows:

- SmolDocling: An ultra-compact VLM for end-to-end document conversion, based on Hugging Face's most recent SmolVLM-256M [56], which is between 5 and 10 times smaller in parameters than comparable VLMs tuned on document understanding tasks. Hence, we reduce the computational complexity to roughly the same order of magnitude as typical representatives of ensemble methods. Uniquely, SmolDocling learns and produces a unified document representation that captures content, structure and spatial location of all elements in a page.
- We augment existing document pre-training datasets with additional feature annotations, and create new datasets for tasks with insufficient coverage in open multi-modal training datasets, including code listings, equations, and chart understanding. Importantly, we construct full-page ground truth data that combines key annotation features such as layout, table structure, code listings, charts, and formulas in their natural context, and derive instruction datasets from all training datasets used.
- We introduce DocTags , a document markup format optimized to efficiently represent the full content and layout characteristics of a document, inspired by [55].
- We evaluate the achieved prediction quality of SmolDocling on several tasks, including comparisons to popular community models.

## 2 Related Work

## 2.1 Large Vision-Language Models (LVLMs)

With the advent of large language models (LLMs) [74, 66, 82, 32, 90, 4, 8, 25, 2] characterized by their powerful contextual understanding and adaptability, several approaches have emerged to extend them into large visionlanguage models. Some proprietary, closed-source LLMs have been adapted into multimodal large language models (MLLMs), such as GPT-4o [65], Gemini [80], and Claude 3.5 [5], demonstrating exceptional capabilities across various modalities, including vision. Open-source methods are actively advancing to match the capabilities of proprietary MLLMs. BLIP-2 [47] is one of the earliest works combining a vision encoder with a frozen LLM (OPT [102] or FlanT5 [14]) using a lightweight transformer (Q-former [101]). Building upon BLIP-2, MiniGPT-4 [105] integrated a frozen visual encoder with a frozen LLM (Vicuna [85]) using a Q-Former network and a single projection layer. LLaVA [52, 51] employs a similar architecture but uses a minimal adapter layer. LLaVA-OneVision [45] and LLaVA-NeXTInterleave [46] build upon LLaVA to support multi-image, higher resolution, and video understanding. For handling high resolutions and text-image compositions and comprehension, InternLM-XComposer introduces a family of models [99, 100, 20, 19] designed specifically for these tasks. Qwen-VL [8] introduces a position-aware adapter to address efficiency issues caused by the vision encoder generating long image feature sequences. Qwen2.5-VL [9]

content_html:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"/>
<title>1f5e4880-d8f5-44fe-8324-b8b33d50e691</title>
<meta name="generator" content="Docling HTML Serializer"/>
<style>
    html {
        background-color: #f5f5f5;
        font-family: Arial, sans-serif;
        line-height: 1.6;
    }
    body {
        max-width: 800px;
        margin: 0 auto;
        padding: 2rem;
        background-color: white;
        box-shadow: 0 0 10px rgba(0,0,0,0.1);
    }
    h1, h2, h3, h4, h5, h6 {
        color: #333;
        margin-top: 1.5em;
        margin-bottom: 0.5em;
    }
    h1 {
        font-size: 2em;
        border-bottom: 1px solid #eee;
        padding-bottom: 0.3em;
    }
    table {
        border-collapse: collapse;
        margin: 1em 0;
        width: 100%;
    }
    th, td {
        border: 1px solid #ddd;
        padding: 8px;
        text-align: left;
    }
    th {
        background-color: #f2f2f2;
        font-weight: bold;
    }
    figure {
        margin: 1.5em 0;
        text-align: center;
    }
    figcaption {
        color: #666;
        font-style: italic;
        margin-top: 0.5em;
    }
    img {
        max-width: 100%;
        height: auto;
    }
    pre {
        background-color: #f6f8fa;
        border-radius: 3px;
        padding: 1em;
        overflow: auto;
    }
    code {
        font-family: monospace;
        background-color: #f6f8fa;
        padding: 0.2em 0.4em;
        border-radius: 3px;
    }
    pre code {
        background-color: transparent;
        padding: 0;
    }
    .formula {
        text-align: center;
        padding: 0.5em;
        margin: 1em 0;
        background-color: #f9f9f9;
    }
    .formula-not-decoded {
        text-align: center;
        padding: 0.5em;
        margin: 1em 0;
        background: repeating-linear-gradient(
            45deg,
            #f0f0f0,
            #f0f0f0 10px,
            #f9f9f9 10px,
            #f9f9f9 20px
        );
    }
    .page-break {
        page-break-after: always;
        border-top: 1px dashed #ccc;
        margin: 2em 0;
    }
    .key-value-region {
        background-color: #f9f9f9;
        padding: 1em;
        border-radius: 4px;
        margin: 1em 0;
    }
    .key-value-region dt {
        font-weight: bold;
    }
    .key-value-region dd {
        margin-left: 1em;
        margin-bottom: 0.5em;
    }
    .form-container {
        border: 1px solid #ddd;
        padding: 1em;
        border-radius: 4px;
        margin: 1em 0;
    }
    .form-item {
        margin-bottom: 0.5em;
    }
    .image-classification {
        font-size: 0.9em;
        color: #666;
        margin-top: 0.5em;
    }
</style>
</head>
<body>
<div class='page'>
<h2>SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion</h2>
<p>Ahmed Nassar 1 , Andres Marafioti 2 , Matteo Omenetti 1 , Maksym Lysak 1 , Nikolaos Livathinos 1 , Christoph Auer 1 , Lucas Morin 1 , Rafael Teixeira de Lima 1 , Yusik Kim 1 , A. Said Gurbuz 1 , Michele Dolfi 1 , Miquel Farré 2 , Peter W. J. Staar 1</p>
<p>IBM Research 1 , HuggingFace 2</p>
<p>https://huggingface.co/ds4sd/SmolDocling-256M-preview Version 1.0, March 14 th 2025</p>
<h2>Abstract</h2>
<p>Weintroduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms - significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.</p>
<h2>1 Introduction</h2>
<p>For decades, converting complex digital documents into a structured, machine-processable format has been a significant technical challenge. This challenge primarily stems from the substantial variability in document layouts and styles, as well as the inherently opaque nature of the widely used PDF format, which is optimized for printing rather than semantic parsing. Intricate layout styles and visually challenging elements such as forms, tables, and complex charts can significantly impact the reading order and general understanding of documents. These problems have driven extensive research and development across multiple domains of computer science. On one hand, sophisticated ensemble systems emerged, which decompose the conversion problem into several sub-tasks (e.g. OCR, layout analysis, table structure recognition, classification) and tackle each sub-task independently. Although such systems can achieve high-quality results for many document types while maintaining relatively low computational demands, they are often difficult to tune and generalize.</p>
<p>On the other hand, in the recent past, great interest has developed around large foundational multimodal models that can solve the whole conversion task in one shot, while simultaneously offering flexible querying and parametrization through prompts. This approach has been made possible by the advent of multimodal pre-training of large visionlanguage models (LVLMs), which opened up a vast array of opportunities for leveraging diverse data sources, including PDF documents. However, the literature on this topic highlights a significant gap in the availability of high-quality and open-access datasets suitable for training robust multi-modal models for the task of document understanding. Furthermore, relying on LVLMs may introduce common issues associated with such models, including hallucinations and the use of significant computational resources, making them impractical from both a quality and cost perspective.</p>
<figure><figcaption><div class="caption">Figure 1: SmolDocling/SmolVLM architecture. SmolDocling converts images of document pages to DocTags sequences. First, input images are encoded using a vision encoder and reshaped via projection and pooling. Then, the projected embeddings are concatenated with the text embeddings of the user prompt, possibly with interleaving. Finally, the sequence is used by an LLM to autoregressively predict the DocTags sequence.</div></figcaption></figure>
<p>In this work, we outline how we close the gaps left by publicly available datasets and establish a training approach to achieve end-to-end, full-featured document conversion through a vision-language model, which we name SmolDocling . Our main contributions can be summarized as follows:</p>
<ul>
<li style="list-style-type: '· ';">SmolDocling: An ultra-compact VLM for end-to-end document conversion, based on Hugging Face's most recent SmolVLM-256M [56], which is between 5 and 10 times smaller in parameters than comparable VLMs tuned on document understanding tasks. Hence, we reduce the computational complexity to roughly the same order of magnitude as typical representatives of ensemble methods. Uniquely, SmolDocling learns and produces a unified document representation that captures content, structure and spatial location of all elements in a page.</li>
<li style="list-style-type: '· ';">We augment existing document pre-training datasets with additional feature annotations, and create new datasets for tasks with insufficient coverage in open multi-modal training datasets, including code listings, equations, and chart understanding. Importantly, we construct full-page ground truth data that combines key annotation features such as layout, table structure, code listings, charts, and formulas in their natural context, and derive instruction datasets from all training datasets used.</li>
<li style="list-style-type: '· ';">We introduce DocTags , a document markup format optimized to efficiently represent the full content and layout characteristics of a document, inspired by [55].</li>
<li style="list-style-type: '· ';">We evaluate the achieved prediction quality of SmolDocling on several tasks, including comparisons to popular community models.</li>
</ul>
<h2>2 Related Work</h2>
<h2>2.1 Large Vision-Language Models (LVLMs)</h2>
<p>With the advent of large language models (LLMs) [74, 66, 82, 32, 90, 4, 8, 25, 2] characterized by their powerful contextual understanding and adaptability, several approaches have emerged to extend them into large visionlanguage models. Some proprietary, closed-source LLMs have been adapted into multimodal large language models (MLLMs), such as GPT-4o [65], Gemini [80], and Claude 3.5 [5], demonstrating exceptional capabilities across various modalities, including vision. Open-source methods are actively advancing to match the capabilities of proprietary MLLMs. BLIP-2 [47] is one of the earliest works combining a vision encoder with a frozen LLM (OPT [102] or FlanT5 [14]) using a lightweight transformer (Q-former [101]). Building upon BLIP-2, MiniGPT-4 [105] integrated a frozen visual encoder with a frozen LLM (Vicuna [85]) using a Q-Former network and a single projection layer. LLaVA [52, 51] employs a similar architecture but uses a minimal adapter layer. LLaVA-OneVision [45] and LLaVA-NeXTInterleave [46] build upon LLaVA to support multi-image, higher resolution, and video understanding. For handling high resolutions and text-image compositions and comprehension, InternLM-XComposer introduces a family of models [99, 100, 20, 19] designed specifically for these tasks. Qwen-VL [8] introduces a position-aware adapter to address efficiency issues caused by the vision encoder generating long image feature sequences. Qwen2.5-VL [9]</p>
</div>
</body>
</html>

pages_json:
[{"page_number": 1, "width": 612.0, "height": 792.0, "elements": [{"type": "section_header", "bbox": [75.333, 90.88667499999997, 536.6712892000002, 134.60335780000003], "content": "SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion", "level": 1}, {"type": "text", "bbox": [96.914, 186.28416659999993, 517.5758744, 215.79938820000007], "content": "Ahmed Nassar 1 , Andres Marafioti 2 , Matteo Omenetti 1 , Maksym Lysak 1 , Nikolaos Livathinos 1 , Christoph Auer 1 , Lucas Morin 1 , Rafael Teixeira de Lima 1 , Yusik Kim 1 , A. Said Gurbuz 1 , Michele Dolfi 1 , Miquel Farr\u00e9 2 , Peter W. J. Staar 1", "level": 1}, {"type": "text", "bbox": [245.00899999999982, 223.87747500000012, 366.49281699999983, 237.8993882000001], "content": "IBM Research 1 , HuggingFace 2", "level": 1}, {"type": "text", "bbox": [153.43199999999982, 251.5261008, 458.5675583999997, 271.39638820000005], "content": "https://huggingface.co/ds4sd/SmolDocling-256M-preview Version 1.0, March 14 th 2025", "level": 1}, {"type": "section_header", "bbox": [278.3899999999999, 315.2505632, 333.6110687999999, 320.797776], "content": "Abstract", "level": 1}, {"type": "text", "bbox": [107.387, 325.918075, 505.8226149056, 481.5043882], "content": "Weintroduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms - significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.", "level": 1}, {"type": "section_header", "bbox": [72.0, 501.1055632, 159.21318400000004, 506.652776], "content": "1 Introduction", "level": 1}, {"type": "text", "bbox": [72.0, 517.858075, 541.0973397709998, 629.8083882], "content": "For decades, converting complex digital documents into a structured, machine-processable format has been a significant technical challenge. This challenge primarily stems from the substantial variability in document layouts and styles, as well as the inherently opaque nature of the widely used PDF format, which is optimized for printing rather than semantic parsing. Intricate layout styles and visually challenging elements such as forms, tables, and complex charts can significantly impact the reading order and general understanding of documents. These problems have driven extensive research and development across multiple domains of computer science. On one hand, sophisticated ensemble systems emerged, which decompose the conversion problem into several sub-tasks (e.g. OCR, layout analysis, table structure recognition, classification) and tackle each sub-task independently. Although such systems can achieve high-quality results for many document types while maintaining relatively low computational demands, they are often difficult to tune and generalize.", "level": 1}, {"type": "text", "bbox": [72.0, 632.428075, 541.6859401416003, 722.5603882], "content": "On the other hand, in the recent past, great interest has developed around large foundational multimodal models that can solve the whole conversion task in one shot, while simultaneously offering flexible querying and parametrization through prompts. This approach has been made possible by the advent of multimodal pre-training of large visionlanguage models (LVLMs), which opened up a vast array of opportunities for leveraging diverse data sources, including PDF documents. However, the literature on this topic highlights a significant gap in the availability of high-quality and open-access datasets suitable for training robust multi-modal models for the task of document understanding. Furthermore, relying on LVLMs may introduce common issues associated with such models, including hallucinations and the use of significant computational resources, making them impractical from both a quality and cost perspective.", "level": 1}]}, {"page_number": 2, "width": 612.0, "height": 792.0, "elements": [{"type": "picture", "bbox": [98.08104705810547, 71.2328872680664, 512.7471923828125, 213.59715270996094], "content": "", "level": 1}, {"type": "text", "bbox": [72.0, 233.74307499999998, 541.538533512, 280.2383882], "content": "Figure 1: SmolDocling/SmolVLM architecture. SmolDocling converts images of document pages to DocTags sequences. First, input images are encoded using a vision encoder and reshaped via projection and pooling. Then, the projected embeddings are concatenated with the text embeddings of the user prompt, possibly with interleaving. Finally, the sequence is used by an LLM to autoregressively predict the DocTags sequence.", "level": 2}, {"type": "text", "bbox": [72.0, 298.831075, 541.5339365599999, 334.4173882], "content": "In this work, we outline how we close the gaps left by publicly available datasets and establish a training approach to achieve end-to-end, full-featured document conversion through a vision-language model, which we name SmolDocling . Our main contributions can be summarized as follows:", "level": 1}, {"type": "list", "bbox": [99.387, 342.095075, 540.0046143088001, 410.40938819999997], "content": "SmolDocling: An ultra-compact VLM for end-to-end document conversion, based on Hugging Face's most recent SmolVLM-256M [56], which is between 5 and 10 times smaller in parameters than comparable VLMs tuned on document understanding tasks. Hence, we reduce the computational complexity to roughly the same order of magnitude as typical representatives of ensemble methods. Uniquely, SmolDocling learns and produces a unified document representation that captures content, structure and spatial location of all elements in a page.", "level": 2}, {"type": "list", "bbox": [99.387, 411.636075, 541.0976111547999, 469.0413882], "content": "We augment existing document pre-training datasets with additional feature annotations, and create new datasets for tasks with insufficient coverage in open multi-modal training datasets, including code listings, equations, and chart understanding. Importantly, we construct full-page ground truth data that combines key annotation features such as layout, table structure, code listings, charts, and formulas in their natural context, and derive instruction datasets from all training datasets used.", "level": 2}, {"type": "list", "bbox": [99.387, 470.269075, 540.0018592584, 494.9463882], "content": "We introduce DocTags , a document markup format optimized to efficiently represent the full content and layout characteristics of a document, inspired by [55].", "level": 2}, {"type": "list", "bbox": [99.387, 496.174075, 539.9973633439998, 520.8513882], "content": "We evaluate the achieved prediction quality of SmolDocling on several tasks, including comparisons to popular community models.", "level": 2}, {"type": "section_header", "bbox": [72.0, 540.8875631999999, 164.712576, 546.434776], "content": "2 Related Work", "level": 1}, {"type": "section_header", "bbox": [72.0, 564.6671666, 273.66294919999996, 569.289813], "content": "2.1 Large Vision-Language Models (LVLMs)", "level": 1}, {"type": "text", "bbox": [71.522, 577.883075, 541.6807994399998, 722.5603882], "content": "With the advent of large language models (LLMs) [74, 66, 82, 32, 90, 4, 8, 25, 2] characterized by their powerful contextual understanding and adaptability, several approaches have emerged to extend them into large visionlanguage models. Some proprietary, closed-source LLMs have been adapted into multimodal large language models (MLLMs), such as GPT-4o [65], Gemini [80], and Claude 3.5 [5], demonstrating exceptional capabilities across various modalities, including vision. Open-source methods are actively advancing to match the capabilities of proprietary MLLMs. BLIP-2 [47] is one of the earliest works combining a vision encoder with a frozen LLM (OPT [102] or FlanT5 [14]) using a lightweight transformer (Q-former [101]). Building upon BLIP-2, MiniGPT-4 [105] integrated a frozen visual encoder with a frozen LLM (Vicuna [85]) using a Q-Former network and a single projection layer. LLaVA [52, 51] employs a similar architecture but uses a minimal adapter layer. LLaVA-OneVision [45] and LLaVA-NeXTInterleave [46] build upon LLaVA to support multi-image, higher resolution, and video understanding. For handling high resolutions and text-image compositions and comprehension, InternLM-XComposer introduces a family of models [99, 100, 20, 19] designed specifically for these tasks. Qwen-VL [8] introduces a position-aware adapter to address efficiency issues caused by the vision encoder generating long image feature sequences. Qwen2.5-VL [9]", "level": 1}]}]


