Beyond the Demo: The Hidden Complexities of Training and Validating VLMs for Document AI

The “Demo Trap”: From Notebook to Production

In the world of Artificial Intelligence, it is easy to fall into what we might call the “demo trap.” Today, thanks to the wide availability of open-source frameworks and online tutorials, creating a notebook that fine-tunes a Small Vision-Language Model (SVLM) for a specific task (such as extracting structured data from a set of documents) has become relatively straightforward.

However, moving from a simple notebook trained on a limited collection of documents to an enterprise production environment capable of handling the unpredictability of real-world documents at scale is an extremely complex process.

In practice, building the model accounts for only 20% of the overall effort. The remaining 80% is the real challenge and involves several critical activities:

• Creating and curating a sufficiently large multimodal dataset that accurately represents the task the model is expected to specialize in.
• Engineering the right training pipeline while avoiding complex technical issues.
• Thoroughly and rigorously evaluating the outputs generated by the model after fine-tuning.

It is also important to note that the newest and most powerful VLM is not always the best choice. The most suitable solution can vary significantly depending on the use case. In some scenarios, older but much lighter and faster models may be the more sensible option, helping optimize both resource usage and response times.

‍

The Need for a Multimodal Dataset

As mentioned earlier, the first major engineering challenge lies in the nature of the data.

Training a Vision-Language Model (VLM) is fundamentally different from training a purely text-based Large Language Model (LLM), as it requires a multimodal dataset. A VLM must learn to combine and interpret text, layout, spatial coordinates, and visual elements (such as stamps, logos, or signatures) simultaneously.

Creating, annotating, and managing this type of dataset is generally highly complex and time-consuming. In addition, training a model (even a relatively small one, such as a small VLM) on large volumes of high-resolution images requires substantial computational resources. If these resources are not managed properly, they can quickly translate into significant costs.

It is therefore essential, though far from simple, to find the right trade-off between several technical factors, including input image size, GPU memory consumption (VRAM), and the desired level of performance.

That said, fine-tuning is not always strictly necessary to specialize a model for a particular domain. If the target task is not especially challenging (and, more specifically in a Document AI context, if the documents are not highly complex or heavily structured) it may be possible to use the pre-trained base model directly.

What remains essential, however, is a multimodal dataset for validation. This dataset should share the same characteristics as the training data while remaining fully separate from it. It can then be used to thoroughly assess the model’s performance on the target task, whether the model has been fine-tuned or is used as-is.

The Complexities of Fine-Tuning

Once a suitable dataset has been created, the next step is to determine how to process it and feed it into the model so that, after fine-tuning, the model can return the required information in the most appropriate format.

In Document AI, for example, the model must return extracted data in a structured format that is ready to be integrated into enterprise systems such as ERP or CRM platforms.

Training a model to produce information in a strict output format (such as a predefined JSON schema) can be particularly difficult. These models naturally tend to generate free-form text or hallucinate keys and syntax, which can break downstream integrations.

Another major issue is catastrophic forgetting. If fine-tuning is not performed correctly, the model may become overly specialized in the new task and lose some of the general comprehension and reasoning abilities acquired during pre-training. As a result, it may become more fragile and less capable of generalizing to new or unexpected cases.

‍

The Challenges of Validating a Model After Fine-Tuning

Designing and training a model is already an extremely complex process, but measuring its actual success is even more difficult.

Although several techniques and metrics are commonly used to evaluate models on Document AI tasks, none of them is perfect and, in most cases, multiple metrics must be combined to obtain a reliable overview of the model’s true capabilities.

Initially, the most widely used metric was exact match, which returned a positive result only when the model’s prediction was identical to the ground truth.

This clearly creates a problem whenever the prediction is semantically equivalent to the ground truth, despite being syntactically different (for example, “10/05/2024” instead of “May 10, 2024”), as the model would incorrectly classify the prediction as wrong, potentially generating a large number of false negatives.

Later, since VLMs are still language models at their core, researchers began using popular NLP metrics such as ROUGE and BLEU. However, these metrics were designed to evaluate the fluency of translations or summaries and are therefore not useful in an information extraction context, where what matters is the precise retrieval of data rather than linguistic paraphrasing.

Current academic research focuses on metrics such as ANLS (Average Normalized Levenshtein Similarity). Although these metrics are much better suited to Document AI tasks, as they tolerate minor “noise” (such as small OCR errors), they still have limitations when it comes to evaluating the overall logical correctness of the extracted data.

Finally, a modern approach consists of using the LLM-as-a-judge technique to capture semantic nuances. This approach involves using a separate LLM (generally much larger and more powerful than the primary model) to evaluate the responses generated by the main VLM. Although this method is increasingly used and is both effective and generalizable across different tasks, it involves very high computational costs and execution times, which grow to almost unsustainable levels as the number of documents in the test dataset increases.

‍

‍

The Solution: A Heterogeneous Evaluation Approach

To navigate this maze of bottlenecks, the most effective approach is to adopt a heterogeneous evaluation strategy.

Rather than relying on a single metric, it is necessary to combine the right set of evaluation techniques in a balanced and appropriate way: logical and syntactic validation of the JSON, the use of ANLS to tolerate minor textual variations, and selective LLM-as-a-judge calls only for semantically ambiguous fields. Implementing an automated validation system of this level requires highly specialized software engineering expertise.

‍

Beyond the Model: Automating the Entire Process

At this point, it is important to clarify one key detail: the true goal of an IDP pipeline should never be limited to information extraction alone; the objective is to automate the entire document management process from start to finish.

From this broader perspective, the “model” itself (together with its fine-tuning and validation) represents only a small part of the overall pipeline, which instead consists of a series of complex and interconnected steps, ranging from file ingestion to the integration of the final structured data.

Note: The design of a complete IDP pipeline, the related infrastructure complexities, and an analysis of when it may make sense to rely on experts rather than attempt to build this type of pipeline “in-house” will be the central topic explored in detail in our next article.

‍

Conclusion

Document AI has the potential to transform business workflows, but technology alone is not enough. Without a pipeline specifically designed for the target use case, trained on high-quality data, and properly validated through heterogeneous metrics, even the most powerful and high-performing model will fail at its assigned task as soon as it moves beyond the “demo” environment.

Designing and maintaining a robust pipeline of this kind requires time and specialized expertise in both Machine Learning and Software Engineering.

For all these reasons, as well as a range of additional complexities not covered in this article because they depend on the specific use case, companies that want to successfully automate their document processes should rely on experts in Intelligent Document Processing.

Would you like to understand how to implement and integrate an IDP platform suited to your document workflows in a robust and scalable way? Contact us or book a demo to discover the enterprise solutions developed by the experts at myBiros.