NeoMME: an efficient Multimodal-native and Multilingual Encoder

In the rapidly evolving landscape of artificial intelligence, the standard approach to multimodal document retrieval has relied heavily on repurposing generative visual language models (VLMs). These architectures, while powerful, often carry significant baggage—specifically, the overhead of separate vision encoders and causal language decoders that are unnecessary for tasks like retrieval or classification. Today, we are introduced to NeoMME, a new family of multilingual, multimodal encoders designed to strip away that complexity, offering a streamlined, "native" approach to processing text and images.

Available in 260M and 800M parameter sizes, NeoMME represents a shift toward efficiency. By utilizing a single bidirectional Transformer to process both raw image patches and text tokens, the model eliminates the need for external vision towers or causal decoders, setting a new benchmark for performance and throughput in the visual document retrieval space.

Rethinking the Multimodal Architecture

The current industry standard for visual document retrieval often involves a "dual-tower" or VLM-based setup. In these configurations, a pretrained vision encoder extracts features, which are then projected into the input space of a causal language model. While effective for generative tasks, this architecture is fundamentally inefficient for retrieval, where the goal is to produce vector representations rather than generate text autoregressively.

NeoMME (pronounced "nee-oh-me") was developed to solve this by functioning as a foundation encoder from the ground up. It does not rely on existing pretrained components. Instead, it treats images and text as first-class citizens within a single computational path.

Key Architectural Innovations

• Unified Transformer Backbone: Both text and image inputs are processed through the same encoder, facilitating easier parallelization and serving.
• Dynamic Resolution: Images maintain their original aspect ratios, allowing the model to allocate more tokens to information-dense document pages compared to simpler images.
• Long-Context Capability: With a context window of 16,384 tokens, the model can comfortably process up to two 4K UHD images simultaneously.
• Modern Encoder Stack: The architecture incorporates state-of-the-art features, including grouped-query attention, 2D rotary position embeddings, gated attention, and squared-ReLU MLPs.

Training for Multimodal Mastery

NeoMME is trained from scratch using a discrete masked-diffusion objective. The training process is designed to force the model to learn deep, grounded representations of visual and textual data.

For text-only inputs, the model undergoes a variable corruption rate, learning to reconstruct missing tokens. When multimodal inputs are introduced, the corruption rate for text is increased, effectively forcing the model to ignore "language-only" shortcuts and instead rely on the visible image patches to reconstruct the missing information. This pretraining regimen covers a vast array of multilingual text, code, mathematics, and document images, totaling approximately 524 billion tokens. To maximize data efficiency, the developers utilized the NorMuon optimizer, allowing the model to achieve high performance with a significantly smaller training budget than predecessors like ModernBERT.

NeoMME-Retriever: Precision and Speed

To demonstrate the power of the NeoMME backbone, the team developed NeoMME-Retriever, a version fine-tuned for visual document retrieval. This approach mirrors the page-image methodology popularized by ColPali, which bypasses the need for error-prone OCR by treating document pages as images. This preserves critical visual cues like layout, charts, and font styles that traditional text extraction often flattens.

A Dual-Head Retrieval Design

NeoMME-Retriever features two distinct heads that are trained jointly: 1. The Dense Head: Performs mean pooling on the hidden states to create a compact, normalized vector. This is ideal for high-speed approximate nearest-neighbor (ANN) search. 2. The Late-Interaction Head: Projects each token and patch into a 128-dimensional vector. This fine-grained approach allows for superior matching between specific query tokens and document regions.

"One NeoMME-Retriever forward pass returns both representations, providing developers with the flexibility to use dense embeddings for initial filtering and late-interaction for high-precision reranking."

Performance on the ViDoRe Benchmark

The results speak for themselves. On the ViDoRe v3 benchmark, the NeoMME-Retriever-260M model achieves an nDCG@10 score of 0.523, outperforming all other models with fewer than 800M parameters. Notably, it performs within 0.002 of the much larger ColQwen2.5 while utilizing 14 times fewer parameters. The 800M variant reaches an nDCG@10 of 0.556, placing it firmly on the Pareto frontier of model size and retrieval quality.

Throughput Comparison

Efficiency is not just about accuracy; it is about the cost of indexing. When tested on an NVIDIA L40S GPU with 2048×2048 image inputs, the 260M model encodes approximately 51 pages per second. This is nearly double the throughput of ColModernVBERT, making it a highly attractive option for large-scale document processing tasks.

Solving the Storage Challenge

One of the primary drawbacks of late-interaction models is the storage footprint of the resulting embeddings. A single 2048×2048 page can generate over 2 MB of data. NeoMME addresses this with two clever compression techniques:

• Hierarchical Token Pooling: Clusters similar document vectors and replaces them with their mean, drastically reducing the number of stored vectors.
• Asymmetric Quantization: Compresses document embeddings to int8 or binary, while keeping query embeddings at higher precision.

By combining these methods, the team achieved a 255× reduction in storage—shrinking the footprint from 1.5 MB to just 6 kB per page—while retaining over 95% of the original retrieval quality.

Getting Started with NeoMME

NeoMME is fully integrated into the Hugging Face Transformers library, making it accessible for immediate deployment. Whether you are building a visual RAG system—where document images are retrieved and fed into a VLM for synthesis—or a standalone search engine, the model is ready for use.

Key Takeaways

• Efficiency: A single-tower architecture removes the overhead of separate vision/language models.
• Performance: State-of-the-art retrieval accuracy at compact model sizes.
• Scalability: Advanced compression techniques make high-resolution late-interaction indexing practical.
• Accessibility: Apache 2.0 licensing and full Hugging Face support ensure the model is ready for enterprise and research applications.

As the industry moves toward more efficient and specialized AI architectures, NeoMME stands out as a prime example of how thoughtful design can yield significant gains in both performance and operational cost. You can explore the model checkpoints, technical report, and visual RAG demo on the official NeoMME collection page on Hugging Face.