From code to pixels
Over the years, language models have become very capable at writing code. Code can also represent visual content, giving us a way to evaluate their visual generation and understanding directly through text inputs and outputs. SVG describes shapes, paths, and colors; CAD programs describe geometry. But these representations are still one step away from the image itself. In SVG and other domain-specific languages for visual content,1 the model writes an element, and a renderer turns it into many pixels.
This leads us to a question: can models generate and understand pixels directly?
There is already plenty of work on generating pixels directly, including PixelRNN, PixelCNN, Image Transformer, and iGPT.2 More specifically, our question is: what can existing language models already do if we simply ask them to output pixels through their text interface? And can they read those pixels, too?
That makes pixels a probe. We can use the same simple representation to study how models express and understand visual content. This was the starting point for PixelBench.
从代码到像素
经过多年的发展，语言模型已经很擅长写代码。其中，一些代码还可以用来表示视觉内容：SVG 可以描述形状、路径和颜色，CAD 程序可以描述几何结构。这让我们可以通过代码形式的输入和输出，评测语言模型的视觉生成和理解能力。不过，这些表示与图像本身的像素之间仍隔着一层。在 SVG 等用于表达视觉内容的领域专用语言1中，模型写下的是形状、路径等元素，再由渲染器将它们转换为像素，而不是直接陈述像素本身。
这引导我们提出一个问题：模型能不能逐个像素地理解和生成图像？
直接生成像素当然已经有很多研究。PixelRNN、PixelCNN、Image Transformer 和 iGPT 都探索过这件事。2 所以具体来说，我们这里的问题应该是：直接让现有的语言模型通过文本接口输出像素，它们已经能做到什么程度？反过来，它们能不能读懂这些像素？
这样，像素就成了一个 probe。我们可以用同一种很简单的表示，去观察模型如何表达和理解视觉内容。这也是 PixelBench 的出发点。
Making pixels practical to write: palette first, indices next
The next question was how to write those pixels.
We could give every pixel its own RGB values, but that would introduce a lot of repetition. We wanted a compact representation that reduced the extra burden of long sequences, so the evaluation would focus more directly on the model's ability to handle visual content.3 We therefore split the image into two parts: a palette of up to eight colors, followed by a two-dimensional grid of palette indices. Each cell only needs to specify which palette color it uses. The full RGB value is written once.
For example, the mushroom below uses five colors. Palette index 2 refers to #D62828, so every 2 in the grid becomes that color. Hover over a cell to see its index and color.
先让像素适合文本输出
先定义颜色，再用索引表示像素
那么像素应该被如何表达呢？最直接的办法，是给每个像素都写一组 RGB 数值。但这样会产生很多重复内容。我们希望把表示写得尽量紧凑，减少序列长度给评测带来的额外负担，让实验更多地反映模型处理视觉内容的能力。3所以，我们把一张图拆成两部分：先定义一个最多包含八种颜色的调色板，再输出一个颜色索引组成的二维网格。每个格子只需要说明自己使用调色板里的哪一种颜色，完整的 RGB 色值只写一次。
例如，下面这张蘑菇图用了五种颜色。调色板中索引 2 对应 #D62828，所以网格里的每一个 2 都会被画成这一颜色。把鼠标移到某个格子上，就能看到它的索引和对应的颜色。
2#D62828A 16 × 16 grid contains 256 color indices. The model can write it using its existing text vocabulary, and rendering only requires a palette lookup. There is no additional learned image tokenizer or decoder. For the generation tasks, we serialize the palette and grid as JSON.
With that representation in place, we could start testing models. We used a zero-shot setting, directly asking existing models to generate and recognize pixel images.
一个 16 × 16 的网格有 256 个颜色索引。模型通过已有的文本词表就能输出它，渲染时只需要查调色板，不需要额外学习一个图像 tokenizer 或 decoder。我们用 JSON 保存生成任务中的调色板和网格。有了这个表示，就可以开始测模型了。这部分采用 zero-shot 设置，直接向现成模型提出生成和识别要求。
What can the models draw?
We started with two generation tasks.
Text-to-Pixel (T2P) is straightforward: give a model an object name, such as rabbit, mushroom, or train, and ask it to draw a pixel image. There is no reference image. The model decides the shape, colors, and layout.
We tested 100 concepts across five categories at three resolutions: 16 × 16, 32 × 32, and 64 × 64. We evaluated whether SigLIP could retrieve the correct concept from 500 candidates, alongside scores from a VLM judge.
模型能画出什么？
我们先做了两种生成任务，Text-to-Pixel 和 Image-to-Pixel。
Text-to-Pixel（T2P） 很简单：给模型一个对象名称，比如兔子、蘑菇或者火车，让它画出对应的像素图。这里没有参考图，模型需要自己决定物体的形状、颜色和布局。我们选了五类共 100 个概念，在 16 × 16、32 × 32 和 64 × 64 三种分辨率下测试。评测时，一方面看 SigLIP 能不能从 500 个候选概念中找回正确的名称，另一方面用 VLM 对生成结果打分。
The models could already produce many recognizable images. Astra performed particularly well, with T2P Top-1 scores of 87%, 92%, and 94% at the three resolutions. Other models also produced images with clear silhouettes and color structure. Browsing the outputs makes it easier to see how different models put together the same rabbit or train.
Image-to-Pixel (I2P) adds another requirement. We give the model a natural image and ask it to reconstruct it as pixels, preserving the subject, pose, composition, and main colors of that particular image.
We selected one image from each of 100 ImageNet classes and tested the same three resolutions, without providing the class name. We measured whether each output could retrieve its source from 100 candidate images, as well as its similarity to the source in SigLIP and DINO feature space.
Here, we could also compare models with traditional image processing. Alongside downscaling and color quantization, we used a simpler version of the classical Pixelated Image Abstraction (PIA) method, which we call PIA-reduced.4 It first divides the source into regions of similar color. It then uses each region's mean color and area to cluster an eight-color palette, and samples the regions' palette assignments onto the output grid.
这些模型确实已经能生成不少可辨认的图像。Astra 的表现尤其好：三个分辨率下的 T2P Top-1 分别是 87%、92% 和 94%。其他模型也能画出有清楚轮廓和颜色结构的结果。直接浏览它们的输出，可以更直观地看到：同样是“兔子”或者“火车”，不同模型会怎样组织这张图。
Image-to-Pixel（I2P） 又多了一层要求。我们给模型一张自然图像，让它重建成像素图。这时它需要保留眼前这张图的主体、姿态、构图和主要颜色。
我们从 100 个 ImageNet 类别中各选一张图像，用同样的三种分辨率测试，输入中不提供类别名称。评测既看生成结果能否从 100 张候选图像中找回原图，也看它与原图在 SigLIP 和 DINO 特征空间中的相似度。
这里还可以和传统图像处理方法做比较。除了缩小图像、再进行颜色量化，我们使用了经典方法 Pixelated Image Abstraction（PIA）的一个简化版本，记作 PIA-reduced。4 它 先把原图分成颜色相近的区域，再根据区域的平均颜色和面积，聚类出八种调色板颜色，最后把区域的颜色分配采样到输出网格上。
Overall, Astra was one of the strongest models, leading the averages across resolutions for both similarity metrics. Sol had the highest average source-retrieval accuracy. Traditional methods were also competitive: at 64 × 64, PIA-reduced reached 78% source-retrieval accuracy, the best result for that metric in the main table.
When browsing these cases, we care about both whether the subject is recognizable and which details of the source survive. Looking at the scores and images together makes the models' strengths easier to understand.
整体来看，Astra 是表现最强的模型之一，在两种相似度指标的跨分辨率均值上都领先；Sol 的平均原图检索准确率最高。传统方法也很有竞争力。例如在 64 × 64 时，PIA-reduced 的原图检索准确率达到 78%，是主表中这一项的最好结果。
所以在浏览这些案例时，我们会同时关心两件事：图里还能不能看出是什么，以及它保留了原图的哪些细节。分数和案例放在一起，才能更容易看清模型各自擅长什么。
Do larger grids make better images?
After I2P, a natural question is whether giving the model more pixels helps it reconstruct the source more faithfully. Moving from 16² to 32² or 64² leaves more room for detail. But the results above already suggest that more room does not always produce a better image.
We therefore pushed the resolution further, testing Astra, Sol, Opus, and Gemini at ten sizes from 16² to 128² on both T2P and I2P. We used fixed subsets of 50 concepts and 50 source images to follow the same samples across resolutions.5
网格更大，就能画得更好吗？
看完 I2P，一个很自然的问题是：如果再给模型多一些像素，它能不能把原图重建得更好？从 16² 到 32²、64²，模型有了更多位置可以表达细节。但前面的结果已经说明，这些位置并不总能变成更好的图像。
所以我们把分辨率继续往上推，让 Astra、Sol、Opus 和 Gemini 在 16² 到 128² 之间的十种分辨率上做 T2P 和 I2P。这里分别使用固定的 50 个概念和 50 张原图，观察同一批样本随分辨率变化的表现。5
Scroll sideways to see all images →
The I2P pattern is fairly clear. Astra generally makes use of the larger grid: reconstruction similarity improves and remains strong at high resolutions. The other models tend to peak somewhere in the middle and then decline. For example, Sol's DINO similarity reaches about 0.151 at 48², then falls to 0.066 at 128². Astra rises from about 0.015 at 16² to 0.333 at 128².
T2P shows a similar problem. Astra stays fairly stable across these resolutions, but more pixels do not consistently help the other models. Sol fluctuates. Opus looks relatively steady under retrieval, while its VLM judge score drops more clearly. Gemini degrades particularly sharply at high resolutions. So this pattern is not just a feature of one evaluation metric.
The images make these differences easier to see. In the fig and altar examples, Astra preserves the subject and composition while adding texture, clearer boundaries, and architectural detail. Sol sometimes produces a larger grid that is still dominated by large blocks of color, leaving much of the added resolution unused. Opus can repeat local patterns or retain only a coarse structure. In some Gemini outputs, the subject and key structures begin to disappear.
This is what makes resolution scaling interesting. Resolution changes more than how much detail an image can hold. A small grid asks the model to choose what matters most; a large grid asks it to organize more local detail into a coherent whole. Going from 16² to 128² means writing 64 times as many pixel indices. More expressive space also means a longer spatial output. Being able to write those positions and being able to organize them are still two different things.
I2P 的趋势比较清楚。Astra 总体上能利用更大的网格，重建的相似度提高了，在高分辨率下也保持得比较好。其他模型则往往在中间的某个分辨率达到高点，之后开始下降。比如 Sol 的 DINO 相似度在 48² 时约为 0.151，到 128² 时降到了 0.066；Astra 则从 16² 的约 0.015 提高到 128² 的 0.333。
T2P 也有类似的问题。Astra 在这组分辨率上一直比较稳定，但其他模型并不是像素越多就画得越好。Sol 的结果会有波动；Opus 的检索分数看起来还算稳定，但 VLM 评分下降得更明显；Gemini 在高分辨率下的退化尤其明显。也就是说，这个现象不只是某一个评测指标带来的。
直接看图会更容易理解这些变化。在无花果和祭坛的例子里，Astra 能保留主体和构图，再逐渐补上纹理、边界和建筑细节。Sol 有时虽然输出了更大的网格，画面却还是由很大的色块组成，相当于没有真正用上新增的分辨率。Opus 会出现重复的局部图案，或者只保留粗略的结构；Gemini 的一些结果里，主体和关键结构则开始消失。
这也是 resolution scaling 有意思的地方。分辨率不只是决定一张图能有多少细节，也改变了模型要处理的问题：小网格要求它选出最重要的内容，大网格则要求它把更多局部细节组织成一张完整的图。从 16² 到 128²，需要写出的像素索引增加了 64 倍。更大的表达空间，也带来了更长的空间输出。能写出这些位置，和能把它们组织好，仍然是两件事。
Can they read what they draw?
After testing generation, we turned the question around: if a model receives only a palette and grid, can it recognize what they depict?
First, we needed good pixel inputs. The I2P experiments gave us two natural choices: Astra, which performed well at generation, and the traditional PIA-reduced method. We used their reconstructions as two pixel oracles: reference sources intended to provide high-quality inputs for the evaluation.
The model sees only the palette and index grid as text, without a rendered image. It must choose the correct concept from ten candidates.6
画得出来，也读得懂吗？
生成测完之后，我们把问题反过来：如果只给模型调色板和网格，它能不能认出里面画了什么？
这里需要先准备质量足够好的像素输入。前面的 I2P 实验给了我们两个自然的选择：一个是生成表现很好的 Astra，另一个是传统方法 PIA-reduced。我们分别使用它们的重建结果，作为两组 pixel oracle。这里的 oracle 指的是提供评测输入的参考理想来源。
测试时，模型看到的只有文本形式的调色板和索引网格，没有渲染后的图像。它需要从十个候选概念中选出答案。6
| Model | PIA reduced 0–60% | Astra 0–80% |
|---|---|---|
| GPT-6 Astra | 57.3% | 77.7% |
| GPT-5.6 Sol | 32.3% | 46.0% |
| Claude Opus 5 | 24.0% | 46.7% |
| Kimi K3 | 18.3% | 36.3% |
| Gemini 3.6 Flash | 16.3% | 27.3% |
| Seed-2.1 Pro | 13.7% | 18.3% |
| DeepSeek V4 Pro | 10.3% | 14.7% |
The differences between models were substantial. Astra reached 67.5% overall accuracy, compared with 39.2% for Sol and 35.3% for Opus. Pixel understanding also correlated strongly with generation: the Pearson correlations with T2P and I2P scores were 0.871 and 0.930, respectively.7
At least through this text interface, models that were better at drawing pixels also tended to be better at reading them.
That made us want to take the idea a step further. So far, we had been testing capabilities the models already had. What would happen if pixel generation became a training objective?
模型之间的差距在这里很明显。Astra 的总体准确率是 67.5%，Sol 是 39.2%，Opus 是 35.3%。而且，像素理解和像素生成的表现有很强的相关性：理解分数与 T2P、I2P 生成分数的 Pearson 相关系数分别是 0.871 和 0.930。7
至少在这个文本形式的像素接口下，比较会画的模型，往往也比较会读。
这个结果让我们想继续往前走一步。前面我们一直在测模型已经具备的能力，那么，如果把像素生成变成一个训练目标，会发生什么？
Training visual understanding through reconstruction
Image reconstruction has been explored as a training objective. Reconstruction Alignment (RecA), for example, trains unified multimodal models to reconstruct input images from their own visual representations, improving image generation and editing.8
We were interested in another question: can learning to output images help a vision-language model understand images better?
For a conventional VLM, taking an image as input and producing text is straightforward. Producing visual content is less convenient. The pixel representation gives it a way to do that through its existing text output: reconstruct the input as a palette and grid. The image itself can then provide a concrete training target tied to spatial locations.
We call this Pixel Reconstruction Tuning (PRT).
Building the reconstruction data
We selected images from PixMo-Cap, removed exact matches to evaluation images, and filtered out content poorly suited to small square reconstructions, including elongated images, text-heavy images, and multi-panel layouts.
We then used PIA-reduced to construct eight-color reconstructions at 16 × 16 and 32 × 32. We wanted the compressed images to remain recognizable, so we ranked them by SigLIP2 similarity to their sources, keeping the top 40% at 16 × 16 and the top 80% at 32 × 32.
Next, Gemini 3.8 Flash used the source image, reconstruction, and palette information to write an image description and explain what each color represented. The algorithm had already determined the palette and grid; the additional model wrote only the descriptions and explanations.
Each training example contains <CAPTION>, <PALETTE>, and <GRID> in that order: describe the image, explain the colors, then provide the reconstruction grid.
What happens when we add these examples?
We started post-training from the Molmo2-4B image-pretraining checkpoint at step 32,000. The control used 500,000 LLaVA-OneVision-1.5 instruction examples. The treatment used the same data plus 2,000 PRT examples, with 1,000 distinct images at each resolution. This adds 0.4% to the original number of examples.
Both conditions trained all modules for one pass over the data. We ran the main experiment with four matched seed pairs and evaluated 16 visual understanding benchmarks.9
用像素重建来训练视觉理解
图像重建作为训练目标，已经有一些相关工作。比如 Reconstruction Alignment（RecA）让统一多模态模型从自己的视觉表示中重建输入图像，从而改善图像生成和编辑能力。8
我们感兴趣的是另一个问题：让一个视觉语言模型学习输出图像，能不能帮助它更好地理解图像？
对常规 VLM 来说，输入图像、输出文本很自然，直接输出视觉内容却缺少一个方便的接口。前面的像素表示恰好提供了这种可能：模型可以沿用已有的文本输出方式，把输入图像重建成一个调色板和网格。这样，图像本身就可以提供一种具体的、与空间位置对应的监督。
我们把这个训练方法叫作 Pixel Reconstruction Tuning（PRT）。
重建数据怎么做？
我们从 PixMo-Cap 中选取图像，去掉与评测图像完全重复的样本，也筛掉不适合低分辨率方形重建的内容，比如长宽比过大、文字很多或者包含多个面板的图像。
接着，用 PIA-reduced 为每张候选图像构造 16 × 16 和 32 × 32 的八色重建结果。我们希望这些结果在压缩之后仍然能保留可辨认的内容，所以用 SigLIP2 衡量重建图与原图的相似度，分别保留两个分辨率下排名前 40% 和前 80% 的候选结果。
之后，再让 Gemini 3.8 Flash 根据原图、重建图和调色板信息，写出图像描述，以及每种颜色对应内容的解释。这里，调色板和网格已经由算法确定，额外的模型只负责写描述和解释。
最终，每条训练样本依次包含 <CAPTION>、<PALETTE> 和 <GRID>：先描述图像，再解释颜色，最后给出重建网格。
加入这些数据之后呢？
我们从 Molmo2-4B 在第 32,000 步的图像预训练 checkpoint 开始做 post-training。对照组使用 50 万条 LLaVA-OneVision-1.5 指令数据；实验组使用同样的数据，再加入 2,000 条 PRT 样本，其中两个分辨率各有 1,000 张不同的图像。这相当于额外增加了原有样本数的 0.4%。
两组都训练全部模块、遍历一次数据，主实验使用四组配对随机种子。在 16 个视觉理解 benchmark 上，我们得到以下结果：9
These evaluations test visual understanding with ordinary image inputs. Adding reconstruction data improved average performance, including gains of 1.4 points on BlindTest, 1.1 on VLM Bias, and 0.9 on CV-Bench.
This gives us a positive signal: visual output supervision that includes pixel reconstruction can help a VLM's visual understanding. We do not yet have a clear explanation of the mechanism. One idea worth testing is that reconstruction asks the model to handle colors, shapes, and spatial layout more explicitly, and that this information can also help with understanding.
这里测的是模型在常规图像输入下的视觉理解能力。加入重建数据后，平均表现有所提升，其中 BlindTest 提升了 1.4 分，VLM Bias 提升了 1.1 分，CV-Bench 提升了 0.9 分。
这个结果给了我们一个正面的信号：包含像素重建的视觉输出监督，可以帮助 VLM 的视觉理解。至于它为什么有效，我们还没有一个明确的机理解释。一个值得继续检验的想法是，重建要求模型更具体地处理颜色、形状和空间布局，而这些信息也可能对理解任务有帮助。
What should we explore next?
The training gains are still modest. A complete PRT example includes a description, color explanations, and a grid, and the treatment also adds images and training steps. To understand where the gains come from, we need to compare caption-only supervision, color explanations, and reconstruction grids on the same images and training budget. We also tested between 1,000 and 8,000 PRT examples. Average gains were positive at each amount, but did not increase monotonically with more data.
The representation has its own limits. Low resolution and a small palette make pixels practical to express as text, but lose detail, especially in images containing text or complex layouts. We started with this format because it offered a simple, workable interface for evaluation and training. A next question is which visual information to preserve, and at what level of detail to supervise it.
We think this may connect to a broader question about grounding. By grounding, we mean supervision tied to specific locations and content in an image. Beyond knowing that there is a bird, a model has to deal with where it is, its colors and outline, and what surrounds it.
Several studies have explored related ideas. Shikra expresses positions as numerical coordinates in ordinary text. In its CLEVR experiments, CoT with object center-point annotations outperformed ordinary CoT.10 VoCoT puts object descriptions and coordinates into multi-step reasoning, alongside visual features from the corresponding regions.11 Bootstrapping Grounded Chain-of-Thought adds verified boxes to key information in reasoning data, then uses that data to fine-tune models, improving understanding of charts, tables, and documents with few training examples.12
More recently, GroundingPI represents points and boxes as quantized coordinates in a shared vocabulary. It trains a grounding foundation model and uses it as a visual backbone for robotic manipulation and autonomous driving.13 Together with RecA, these studies make us want to compare more systematically what models gain when their training targets describe visual content more explicitly.
From this perspective, pixel reconstruction provides a relatively detailed form of spatial supervision. How does it relate to boxes, points, and regions? Which tasks need geometry, and which need color and appearance? What happens when we put these signals into a model's textual reasoning process, compared with asking it to reconstruct the image directly?
These are the comparisons we want to make next. Starting from this simple pixel interface, we would like to evaluate grounding signals more generally and study how to bring them into VLM training.
还有哪些问题值得继续做？
首先，一个还没有解决的问题是数据量。我们尝试把 PRT 数据从 1,000 条增加到 8,000 条，但更多数据没有带来持续的性能增益，收益大致维持在相近的水平。我们还不清楚，这是当前模型的规模限制了它能从重建中学到的东西，还是数据本身的质量、多样性或构造方式限制了收益。这需要更大规模的实验，也需要把模型规模、数据量和数据构成拆开，做更细的消融。
另一个局限是像素表示本身。如果希望这种训练保持 lightweight，我们就很难把输出分辨率设得很大。但低分辨率和有限的颜色又会丢掉很多细节，比如图像中的文字、细小的结构，以及物体之间精细的空间关系。
原生的视觉表示往往很 dense。我们用调色板减少了颜色的重复，网格本身仍然会随着图像面积增长。这也是为什么许多通过文本表达视觉信息的工作会选择 bounding box 或 point：它们用很短的序列，就能指向图像中的具体位置。不过，一个点或一个框也很难完整描述区域里的颜色、形状和内容。
但 Astra 又让这个问题变得很有意思。它为什么已经有这么强的 Pixel generation 能力？在我们测试的一些较高分辨率的例子里，它仍然能生成完整、可辨认的图像。这种能力来自什么样的训练，模型又是怎样在文本序列里组织这些视觉信息的？我们还没有答案，但它让我们更想知道，模型究竟能从这样的视觉表示中学到什么。
所以，我们更想继续讨论的是：什么样的 grounding 信号适合被放进模型的训练里？这里我们从更广泛的 CV 任务来理解 grounding，包括对图像中位置、几何结构和语义区域的具体刻画。Shikra 和 VoCoT 尝试把坐标放进模型的表达与推理过程；1011 Bootstrapping Grounded Chain-of-Thought 使用经过验证的框构造推理监督，GroundingPI 则把点和框作为视觉训练的目标。1213 这些工作和 PRT 都让我们看到，把视觉内容写进训练目标还有很多可以探索的空间。
除了像素重建、bounding box 和 point，我们还有什么样的表示可以用？我们希望它足够稀疏，不需要很长的序列；同时又能保留丰富的语义，并且精确地描述图像中的内容。比如，能不能只表达关键的轮廓、区域和它们之间的关系，就保留重建或理解任务真正需要的信息？
这种表示还需要容易构造。Pixel 一个很大的优势，是它的监督数据可以批量生成：给定一张图像，我们就能用 PIA 得到对应的调色板和网格，不需要为每张图额外做人工标注。如果模型能高效处理完整上下文，原则上甚至可以直接把原图逐像素写成文本。我们希望更紧凑的表示也能保留这个优点——可以通过简单、自动的转换，从现有图像中得到。
这就带出了一个具体的问题：存不存在这样一种视觉语言，既稀疏、精确，又保留足够的语义，而且可以方便地用于 reconstruction alignment？进一步说，这些信号应该作为独立的重建目标，还是进入模型的推理过程？不同的训练方式，会让模型学到什么？
在研究怎样把这些信号融入训练之前，我们想先做一件更基础的事：评测模型能否完成不同 CV 任务所要求的 vision grounding。比如，depth estimation 要求模型估计场景的深度，surface normal estimation 要求模型判断表面的朝向，semantic segmentation 则要求模型给图像中的各个区域赋予对应的语义类别。在我们看来，这些都属于 grounding。我们想知道，面对这些不同的需求，模型能不能完成对应的任务，又能做到什么程度。
这也是我们下一步想用 Grounding Bench 去系统评测的能力。先了解模型在这些任务上已经能做到什么、还缺少什么，再进一步探索哪些 grounding 信号对训练有用，以及怎样把它们融入训练。