PRX Part 4: Our Data Strategy

Welcome back to our ongoing deep dive into the development of PRX. In our previous installments, we explored the intricacies of model architecture, the nuances of training design, and the high-octane 24-hour speedrun that put our systems to the test. Today, we are pulling back the curtain on the foundation that quietly supports everything we do: the data pipeline.

While often overshadowed by the flashier aspects of model engineering, the data pipeline is the true engine of PRX. Building it was a process of trial, error, and hard-won lessons. In short: we aggregate training data from a blend of public and internal sources, utilize a Vision-Language Model (VLM) to re-caption every image, and transform the final output into a streamable corpus optimized for training.

Guiding Principles: Breadth Over Perfection

Our primary objective for pre-training was to assemble a dataset defined by its diversity. At this stage, the model is learning the fundamental structure of the visual world—how objects are composed, how light interacts with surfaces, and the vast range of concepts that images can represent.

"Pre-training is for breadth; fine-tuning is for taste."

We believe that over-filtering for aesthetics during pre-training is a mistake. A model needs to see the "ordinary" to understand the structure of reality. By prioritizing a representative corpus over a smaller, highly curated one, we ensure the model gains a robust understanding of visual concepts that cannot be recovered later. We leave the pursuit of polished, high-aesthetic generations to the subsequent stages of fine-tuning and preference alignment.

A Pragmatic Approach to Sourcing

We built our pre-training data by combining internal assets with existing public datasets. Rather than reinventing the wheel, we leveraged existing quality filters and deduplication efforts where possible. This pragmatic strategy allowed us to move quickly, providing a solid, lightweight starting point for our 7B model.

The Philosophy of Long Captions

Our experience suggests that the most critical lever for pre-training quality is the use of long, descriptive captions. By ensuring that every image is described in exhaustive detail, we effectively turn "noise"—such as logos, advertisements, or text overlays—into controllable attributes. When the model learns to associate these elements with specific descriptive text, they become features the user can prompt for or prompt away, rather than unwanted artifacts.

Infrastructure: Lance and Mosaic

For our distributed training framework, we rely on Mosaic Streaming and Mosaic Data Shards (MDS). This setup provides a low-maintenance, high-performance environment that allows for seamless shuffling and training directly from object storage like S3 or GCS.

However, MDS datasets are inherently rigid. To address this, we integrated Lance, a columnar data format that excels at feature engineering and dataset exploration.

• Lance: Used for building, indexing, and querying billions of rows with cheap predicate pushdown.
• MDS: Used for streaming the final shards during the training loop.

Text Latents and Image Encoding

In previous iterations, we pre-computed text latents. With the transition to Qwen3-VL, we shifted to computing these latents on the fly. While this introduces a minor throughput cost (roughly 3–4%), it grants us the flexibility to swap text encoders without the massive overhead of rewriting terabytes of pre-computed data.

Regarding image storage, we opted for JPEG at quality 92. Our internal testing confirmed that this format is effectively indistinguishable from lossless PNGs in terms of model output, while offering significant storage and bandwidth advantages.

Building and Exploring with Lance

Interacting with hundreds of millions of rows requires more than just standard database queries. By storing our data in Lance, we can perform full-text searches and nearest-neighbor vector searches to profile our dataset effectively.

The Fragmentation Lesson

We learned the importance of fragmentation the hard way. Initially, we targeted 100,000 rows per fragment, which resulted in thousands of tiny files that caused query performance to crawl. By compacting these into fragments of approximately one million rows, we achieved the performance necessary for rapid iteration.

Qualitative Exploration

Before committing to a training run, we built a custom UI to browse our dataset. This allowed us to:

• Identify uninformative baseline captions.
• Spot non-photographic content like infographics or slides.
• Detect near-duplicate images.

This visual audit confirmed our decision to re-caption everything, ensuring a uniform standard of descriptive quality across the entire corpus.

Re-captioning: The VLM Advantage

Our benchmarking proved that long, dense captions significantly outperform short ones. We compared models trained on captions from Qwen2.5-VL-7B against those using shorter LLaVA-1.5-LLaMA3-8B captions; the former consistently achieved lower FID, CMMD, and DINO-MMD scores.

Choosing the Captioner

We shortlisted three candidates: Qwen2.5-VL-7B-Captioner-Relaxed, Qwen3-VL-8B, and Qwen3.5-9B. After rigorous testing, we selected Qwen3-VL-8B for its optimal balance of speed (20 img/s per H200) and high-quality output.

Our system prompt was designed to enforce a neutral, precise, and visually grounded tone, requiring the model to describe everything from lighting and composition to specific text transcriptions without emotional embellishment.

Mosaic Data Shards and Bucketing

The final stage of our pipeline involves converting our processed data into MDS. This format provides essential features for distributed training, including: 1. Deterministic, resumable shuffling. 2. Elastic mid-epoch checkpointing. 3. Weighted mixing of multiple data streams.

To handle varying image dimensions, we implemented aspect-ratio bucketing. By grouping images into specific resolution tiers and aspect ratios, we maintain a constant patch count, ensuring that per-image compute remains flat across the entire training process.

Filtering and Deduplication

Once our detailed captions were in place, we used Qwen3-8B in text-only mode to classify samples as "visual," "text," or "NSFW." Rather than rewriting the entire dataset, we implemented a skip-list feature in our MDS loader. This allows us to exclude samples at training time by referencing a sidecar file, providing a flexible mechanism for handling blacklists or user opt-outs without the need for costly data rewrites.

Similarly, we utilized perceptual hashes to perform deduplication. By identifying and skipping near-pixel-identical copies, we eliminated wasted compute and prevented the skewing of our training distribution.

Looking Ahead

The dataset we have described represents the pre-training corpus, where scale and breadth are paramount. However, the requirements for supervised fine-tuning and preference alignment are entirely different. We are currently developing more advanced curation tools, including a VLM-based tagging system, to assemble high-signal subsets for these later stages.

We invite the community to explore our work. PRX is released under the Apache 2.0 license, and you can find our model code on GitHub. We have also integrated PRX into the diffusers library and invite you to test our latest version, PRX Pixel, on our Hugging Face Space.

*

Key Metrics Summary (Final 100k Steps)

| Captioner | FID ↓ | CMMD ↓ | DINO-MMD ↓ | | :--- | :--- | :--- | :--- | | Qwen3.5-9B | 10.51 | 0.278 | 0.162 | | Qwen3-VL-8B | 10.98 | 0.351 | 0.182 | | Qwen2.5-VL-Relaxed | 13.95 | 0.306 | 0.185 | | Gemini 1.5 Flash | 13.46 | 0.316 | 0.234 | | Qwen2.5-VL-Base | 15.86 | 0.393 | 0.285 |