3. Reading the Label, Not the Object: A Typographic-Distractor Control Dataset for CLIP-Based Robot Grasping

Planted June 24, 2026

Cover: "Care-O-bot grasping an object on the table" by Jiuguang Wang, CC BY-SA 2.0, via Wikimedia Commons.

1. Introduction

Robots that follow natural-language instructions to pick up objects increasingly rely on CLIP as their visual-language backbone. CROG [1], the model trained and evaluated on the OCID-VLG dataset for language-guided grasping in clutter, is a direct example: it fuses CLIP’s joint image-text embedding space with a grasp-synthesis head, so the robot grounds a phrase like “the red mug to the left of the bowl” directly into a graspable region of the scene.

CLIP has a well-documented weakness, however: typographic attacks. Goh et al. [2] showed that CLIP can be made to misclassify an object simply by writing a different object’s name on it — a photograph of an apple with a sticky note reading “iPod” gets classified as an iPod, because the model partially reads scene text rather than relying purely on visual appearance. A 2026 study has since shown this is not just a curiosity of static image classification: it propagates into the full sense-plan-act pipeline of household robot manipulation, with adversarial stickers causing simulated robots to physically grasp and deliver the wrong object to a target location [3].

This raises a direct question for the OCID-VLG / CROG setting: OCID-VLG’s scenes are real tabletop photographs with no text written on any object, so this specific vulnerability has never been exercised against the benchmark CROG is trained and evaluated on. Does that gap actually matter? This blog post describes a small, controlled dataset built to test exactly that.

Property under test: Does a CLIP-based encoder’s similarity score for a language query reflect genuine visual recognition of the referred object, or can it be overridden by incidental text present elsewhere in the scene?

H1: Text location drives grounding, not just text presence.

If CLIP partly relies on reading in-scene text, placing the target’s correct category name on a different object should cause CLIP to prefer that wrong object over the true target - a sharp drop relative to the no-text case. Conversely, placing the correct name on the target itself should reinforce the correct match.

H2: open question on mismatched text.

It is not obvious in advance whether writing an incorrect label on the target would hurt its score (because the semantics no longer match the query) or leave it roughly unchanged (because the mere presence of a salient labelled region could be a generic visual cue independent of correctness). This is tested as an open branch rather than assumed.

2. Core Motivation

Standard benchmark accuracy on OCID-VLG reports a single aggregate number — it cannot tell us whether a model’s grounding is driven by genuine object recognition or by incidental scene text, because no scene in the benchmark contains in-scene text in the first place. Average accuracy on a dataset that never exercises a failure mode says nothing about whether the model is vulnerable to it. This mirrors the broader shortcut-learning problem described by Geirhos et al. [4]: a model can rely on a feature that happens to correlate with the correct answer during training without that feature being the one we actually want it to use.

To isolate this single variable, an evaluation set is needed where the same scene appears under multiple controlled text conditions, so that any change in model behaviour can be attributed to the inserted text rather than to differences in scene content, lighting, clutter, or object pose. That is what this dataset provides.

3. Data Generation Pipeline

3.1 Why real OCID-VLG scenes, not synthetic ones

This dataset is built by minimally editing real OCID-VLG photographs rather than rendering synthetic tabletop scenes from scratch. If a model behaves differently on a fully synthetic scene, it is impossible to tell whether the difference is caused by the property under test or simply because the synthetic scene looks different from the real distribution the model was trained on. Since CROG is trained and evaluated on real OCID-VLG photos, the only way to isolate “does in-scene text change the prediction” as a single variable is to hold the rest of the scene fixed and change only the inserted text. Every one of the four variants below therefore starts from the exact same underlying photograph.

3.2 The four conditions

3.3 Automated generation

Source scenes are drawn from OCID-VLG [1], which provides 1763 cluttered tabletop scenes spanning 31 object categories with per-instance segmentation masks. The pipeline works as follows:

1. Loads each scene’s instance segmentation mask and referring-expression annotations from test_expressions.json
2. Filters to scenes containing at least two distinct object categories (needed to select a valid distractor that differs from the target)
3. Randomly samples 100 such scenes, at most one per unique image
4. For each scene, computes the pixel centroid of the target object’s mask and (for distractor_label) the centroid of a randomly chosen distractor object’s mask
5. Draws the relevant text label, centered on that centroid, on a small backing rectangle, never modifying the underlying image data outside that bounding box

This produces 100 source scenes × 4 variants = 400 images, all anchored to real, mask-verified object locations rather than hand picked coordinates.

The code to generate the dataset is linked at the end of the blog. The controlled dataset can be viewed here

3.4 Dataset composition

The 100 scenes span 28 distinct target object categories drawn from the OCID-VLG object vocabulary. The most frequent categories are shampoo (12 scenes), cereal/cereal_box (16 combined), food-related items (22 combined), and marker (7); rarer categories include flashlight, orange, banana, and food_can (1 scene each). All 100 scenes contain at least 2 object instances, and 93 out of 100 contain 3 or more distinct objects — ensuring the distractor condition is genuinely ambiguous rather than a trivial two-object choice.

Note: some category names appear in shortened form (e.g. “cereal” vs “cereal_box”, “food” vs “food_bag”) because OCID-VLG’s concept_map uses the bare referring word from the query rather than the full class label

4. Examples

Figure 1: Example of a cluttered scene with varied types of objects ranging from books to eatables where the primary information is the image of the scene (Visual and no texts).

Figure 2: Example of the same cluttered scene with a correct label to the target object (Here it is the shampoo)

Figure 3: Example of the same cluttered scene another object is labelled with the target object (Shampoo).

Figure 4: Example of the scene where a wrong label is given to the target object

Query provided is “Alverde shampoo”

5. Validation: Does the Dataset Actually Expose the Vulnerability?

The cleanest way to show the dataset is diagnostic rather than just plausible-looking is to run a CLIP model on it. Using the CLIP RN50 encoder, the OpenAI-pretrained checkpoint that CROG’s architecture is initialized from prior to fine-tuning on OCID-VLG, each scene’s target-object crop and distractor-object crop were scored against the query’s text embedding.

Metric: % of scenes where the target crop scores higher than the distractor crop against the language query.

Interpretation

H1 is supported, but not exactly as predicted.
The diagnostic comparison, moving the same correct label from the target to a distractor, produces an 82%-of-the-time failure rate. This is far larger than sampling noise could explain: with n = 100, the standard error on a ~50% proportion is roughly ±4–5 percentage points, well below the observed 35-point swing. The target_label condition did increase as hypothesized, but by more than expected: writing the correct word directly on the correct object gives CLIP a literal textual confirmation on top of the visual one.

H2’s open branch resolved toward “salience dominates.”
conflict_label did not drop below clean; it rose by 9 pp. A plausible mechanism: writing any legible, boxed label near an object (regardless of whether the text is correct) appears to make that region more visually salient to CLIP, generically boosting its similarity to almost any query. This explains why conflict_label (label present, wrong content) sits between clean (no label) and target_label (label present, correct content): it gets the salience boost without the lexical-match boost. This is a secondary finding the dataset surfaced rather than one designed in from the start.

A flat result across all four conditions would have indicated CLIP’s grounding does not depend on incidental scene text. The large, structured swing instead supports the existence of this shortcut.

6. Limitations & Future Work

Scope of the claim.
The tested checkpoint (RN50.pt) is the generic OpenAI-pretrained CLIP encoder that CROG’s architecture is initialized from prior to fine-tuning on OCID-VLG — not CROG’s own weights after that fine-tuning. The result therefore supports “this vulnerability is inherited from CLIP pretraining,” not “CROG specifically exhibits this after fine-tuning.” Whether CROG’s fine-tuning on OCID-VLG amplifies or mitigates the effect remains untested.

Disentangling label salience from lexical match.
The conflict_label result, a real but wrong category word still boosting the score, could mean either (a) CLIP is partially influenced by the wrong word’s semantics, or (b) it is reacting to the generic visual presence of a labelled box regardless of what the box says. A fifth condition, neutral_label, a nonsense word or a plain colored box with no legible text drawn on the target, would separate these two effects cleanly. If neutral_label also raises accuracy above clean, the effect is driven by label salience; if it stays near clean while conflict_label stays elevated, the effect is driven by lexical content specifically.

Links

• Entire OCID-VLG Dataset: OCID-VLG on Google Drive
• Generation code: generate_typographic_control_dataset.py
• Verification code: verify_typographic_attack.py

References

[1] Tziafas, G., Xu, Y., Goel, A., Kasaei, M., Li, Z., & Kasaei, H. (2023). Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter. Proceedings of the 7th Conference on Robot Learning (CoRL), PMLR 229:1450–1466.

[2] Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021). Multimodal Neurons in Artificial Neural Networks. Distill, 6(3). distill.pub/2021/multimodal-neurons

[3] Iranmanesh, A., & Liu, P. (2026). Not What You Asked For: Typographic Attacks in Household Robot Manipulation. arXiv preprint arXiv:2605.18593.

[4] Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673.