2. Reproducing GeoLanG: Geometry-Aware Language-Guided Grasping with Unified RGB-D Multimodal Learning

Planted April 7, 2026

Cover: "Robot arm picks up cylindrical objects in a factory setting" by Shixart1985, CC BY 2.0, via Wikimedia Commons (cropped).

Code: github.com/kart1802/HAGRID

Introduction

Humans naturally have a notion of an object’s geometric and material properties. Grasping an object comes naturally to us. However, for a robot, this is not a simple problem. Robotic manipulation involves multiple stages, starting from perception and going all the way to control. One of the key challenges lies in the perception stage where the robot needs to understand a scene and identify where it should interact with an object.

Traditional methods in robotic perception focus on predicting graspable regions using visual features derived from RGB and depth data [1][2]. These methods rely heavily on learning patterns like edges, surfaces and antipodal points (two opposite contact points - think of it as pinching an object between finger and thumb). While these methods are effective, they lack an understanding of task-specific preferences and object semantics. They rank the grasp candidates based on geometric feasibility and predicted grasp success, without considering the intended task. Hence, the highest-ranked grasp is not necessarily the most appropriate for task execution. This limitation is evidently seen if we consider an example of picking up a cup — a conventional model, unaware of the task, might rank a grasp on the body of the cup as optimal due to its geometric stability. For “Relocation” this works fine, but for “Pouring” it would result in an awkward or failed attempt, as the handle is the functionally correct contact point for that task. Conventional vision-based models are not equipped to capture such distinctions, as they do not incorporate semantic or linguistic information into their predictions.

In this reproduction blog post, we explore GeoLanG, an architecture designed to close this gap. Rather than relying purely on geometry, GeoLanG incorporates language — so instead of just asking “where can I grasp this object?”, the robot can now ask “where should I grasp this object, given what I need to do with it?”. By grounding task descriptions (like “pour” or “hand over”) in the visual scene, GeoLanG can identify grasp points that are not just physically feasible, but also contextually appropriate. It further makes use of depth information, helping the robot reason about where an object is in 3D space — even when parts of it are hidden from view. In the sections that follow, we will walk through the architecture that makes this possible, and how each component contributes to bridging geometry and intent. Apart from paper reproduction, we also conducted ablation studies which are covered in the later sections.

Related Works

Recent multimodal learning has enabled robots to ground language in visual data, yet moving from identification to physical grasping remains difficult due to occlusion and spatial complexity. Traditional multi-stage pipelines often suffer from cascading errors, and even direct image-text approaches like CROG [3] struggle with precise object boundaries in clutter. To optimize efficiency, DFormerV2 [4] replaced heavy, dedicated depth encoders with explicit geometric priors embedded directly into the attention mechanism.

The choice of visual backbone has also evolved to balance local precision with global context. While ResNets offer spatial detail but lack range, and Vision Transformers (ViTs) provide global context at a high computational cost of O(N²), models like CLIP-VMamba [5] utilize State-Space Modeling (SSM) to achieve global reach with linear O(N) complexity. This provides the hierarchical features necessary for fine-grained robotic perception without the “computational bloat” of previous architectures.

Building upon these advancements, GeoLanG [6] differentiates itself by unifying these isolated concepts into a single, end-to-end multi-task framework. Rather than treating depth processing and text-vision alignment as separate hurdles, GeoLanG explicitly injects depth directly into the attention mechanism as geometric priors without the overhead of a dedicated depth encoder. Furthermore, unlike prevailing methods that freeze visual backbones and discard valuable mid-level visual cues, GeoLanG dynamically aggregates multi-layer features. This unified approach allows the model to simultaneously maintain efficient global context and fine-grained spatial awareness, directly solving the boundary misalignments and occlusion failures that limit prior works.

Dataset

Figure 1: Example of a referring grasp instruction in a cluttered scene. In green, the bounding box is marked, with blue and red indicating the grippers and grasp depth respectively. [8]

OCID-VLG expands on the base dataset OCID [7], the Object Clutter Indoor Dataset. It provides structured and pixelwise RGB-D point-cloud data of objects in cluttered scenes. It concerns where an object is located with respect to other objects. OCID-VLG [8] reworks this dataset for robotics, introducing object masks, natural language instructions and most importantly grasp annotations, effectively combining Vision, Language and Grasping, respectively VLG. The dataset is capable of being used for bounding box prediction and grasp prediction, instead of needing OCID and OCID-grasp separately for this.

This design choice is crucial, since object detection and grasp synthesis are usually treated as two separate problems. The model is therefore not only required to identify what object is being referred to, but also how to act on it. All referring expressions follow the CLEVR data format: [prefix] ([LOC1] [ATT1]) [OBJ1] ((that is) [REL] the ([LOC2] [ATT2]) [OBJ2]) (Fig. 1) [9].

The sentences are built by combining object names, simple properties like color or shape, and spatial relations between objects. Some descriptions are straightforward (e.g. “the cereal box”), while others include extra details to resolve ambiguity (e.g. “the cereal box behind the bowl” or “the leftmost apple”). Before being fed into our model, the RGB and depth channels are resized to 416×416.

Model

In this section, we describe the GeoLanG network architecture and its components, as shown in Figure 2. Figure 2: Overview of the GeoLanG architecture. Taken from [6].

Pretrained Model

We use two pretrained components in our framework: CLIP-VMamba [5] vision encoder for visual feature extraction and CLIP-BERT [14] text encoder for language understanding. CLIP-VMamba captures rich multi-scale visual representations from the input image, while CLIP-BERT encodes the textual instruction into a meaningful sentence-level embedding. The visual backbone produces feature maps of sizes 13×13×1024, 13×13×1024, and 26×26×512. To match the multi-scale design of the architecture, these features are upsampled via bilinear interpolation: one 13×13×1024 feature map is upsampled to 26×26×1024, the 26×26×512 feature map is upsampled to 52×52×512, while the remaining 13×13×1024 feature map is kept unchanged. The text encoder from CLIP-BERT produces Byte Pair Encoding tokenized representations along with corresponding hidden state vectors of dimension $L \times 512$, where $L$ is the token limit of 20, and outputs both token-level text embeddings and a global sentence embedding of size 1×1024.

DGGM Module

The Depth-Guided Geometric Module (DGGM), first introduced in DFormerV2 [4], incorporates depth-aware geometric priors into the attention mechanism to enhance spatial reasoning. Multi-scale visual features from CLIP-VMamba are passed through DGGM, while preserving the original feature dimensionality.

The input image is partitioned into patches of: $H \times W$. In our case, each patch corresponds to a single pixel, and the associated depth value is directly taken at that location.

Given patch-wise depth values $D_{m,n}$, the depth and spatial relationships are defined as:

$$ \Delta D_{m,n,m^\prime,n^\prime} = |D_{m,n} - D_{m^\prime,n^\prime}| $$

$$ \Delta S_{m,n,m^\prime,n^\prime} = |m - m^\prime| + |n - n^\prime| $$ where $\Delta D_{m,n,m^\prime,n^\prime}$ denotes the pairwise depth difference between patches, and $\Delta S_{m,n,m^\prime,n^\prime}$ represents the Manhattan distance between their spatial coordinates.

These are fused to form the geometry prior:

$$ G = \lambda_1 \cdot \Delta D + \lambda_2 \cdot \Delta S $$ where $\lambda_1$ and $\lambda_2$ are learnable weights for the depth and spatial terms, respectively.

Given input features $X \in \mathbb{R}^{HW \times C}$: query, key, and value projections are computed, and attention is modulated as:

$$ \hat{X} = \left( \text{Softmax}(QK^\top) \odot \eta^{G} \right) V $$

where $\eta \in (0,1)$ is a decay factor that suppresses interactions between geometrically distant patches and emphasizes spatially closer ones.

The output retains the same spatial and channel dimensions as the input.

In Figures 4 and 5, we observe how a significant amount of information is removed from the embeddings in the attention matrices (with only the diagonal elements remaining prominent), and how the DGGM output vectors focus primarily on spatially close information.

Figure 3: DGGM uses depth and spatial position to guide attention through a geometry prior [6].

Figure 4: Attention matrices before and after applying the geometric prior on the 52×52×512 embedding. The geometric prior suppresses long-range interactions, resulting in a sparser attention map with values concentrated near the diagonal, reflecting stronger attention between spatially nearby regions.

Figure 5: Output embeddings of DGGM for the 52×52×512 vector without and with the geometric prior. The geometric prior suppresses background responses and emphasizes spatially closer regions, resulting in embeddings that focus more strongly on nearby objects.

Figure 6: ADCI pseudocode for adaptive grouping, weighting, and fusion of visual features [6].

ADCI Module

The Adaptive Dense Channel Integration module (ADCI) is the novel implementation from the authors (Fig. 6). Here they focus on fusing the depth embedding from the DGGM into a single representation. The module takes three visual feature maps (C1, C2, C3), projects them to the same channel size and upsamples each to a fixed grid of 26×26. A tiny gating MLP per scale assigns the importance of the features, in which these scores are softmaxed within groups, and group-weighted sums are built. Finally, these grouped features with the highest-level map are concatenated and passed through a 1×1 Conv/BN/ReLU stack to yield a single dense visual embedding.

Multi-Modal Fusion Network

While the original architecture introduces the concept of multi-modal integration, it does not explicitly define the internal mechanics of its fusion network. To address this ambiguity, we designed a custom, FPN-inspired multi-modal fusion module (Fig. 7). The ADCI output, a visual embedding of size 26×26×512, is processed through a point-wise convolution. Concurrently, the 1×1024 language embedding is pooled and then broadcast to the same 26x26 spatial grid, which gets fused to the visual embedding. After BatchNorm and ReLU normalization, a special CoordConv layer adds coordinate channels to inject spatial information. A final refinement convolution maps the resulting 26×26×514 representation back to 26×26×512.

Figure 7: Architecture of the proposed multi-modal fusion network.

Multi Task Decoder

Similarly, the original GeoLanG paper introduces the concept of a multi-task decoder without explicitly defining its internal structure. Therefore, we modeled it using a classic transformer decoder architecture following CROG [3] (Fig. 8), where the queries are visual features, while the keys and values are textual. Self-attention helps the visual tokens map out their own spatial layout. Next, cross-attention matches those visual regions with the language cues. Finally, a feed-forward network uses layer normalization and residual connections to stabilize training and maintain information flow.

Figure 8: The data flow within our modeled Transformer Decoder.

Segmentation and Grasp Projector

The GeoLanG framework adopts the formulation from CROG [3] to map shared multimodal features into task-specific outputs. The segmentation branch projects visual and sentence features into a shared space, enabling language-guided object localization. In addition, grasp prediction branches are introduced to estimate grasp quality, orientation (sin, cos), and width. Although grasp prediction primarily relies on geometric cues, the implementation conditions all branches on the sentence feature through a shared dynamic kernel.

The overall data flow is illustrated in Fig. 9.

Figure 9: The data flow within the segmentation and grasp projectors.

The loss function

For the segmentation branch, the predicted mask is supervised using binary cross-entropy with logits loss: $$ \mathcal{L}_{ins} = \mathrm{BCE}(M_{pred}, M_{gt}) $$

This loss is additionally weighted by the ground-truth mask so that pixels belonging to the target object receive higher importance during training: $$ \mathcal{L}_{ins} = \mathrm{BCE}(M_{pred}, M_{gt}) \cdot (1 + \alpha M_{gt}) $$

For the grasp branch, the predicted grasp quality, sine, cosine, and width maps are each supervised using Smooth L1 loss: $$ \mathcal{L}_{qua} = \mathrm{SmoothL1}(Q_{pred}, Q_{gt}), \quad \mathcal{L}_{sin} = \mathrm{SmoothL1}(S_{pred}, S_{gt}) $$

$$ \mathcal{L}_{cos} = \mathrm{SmoothL1}(C_{pred}, C_{gt}), \quad \mathcal{L}_{wid} = \mathrm{SmoothL1}(W_{pred}, W_{gt}) $$

The final training objective is given by: $$ \mathcal{L}_{total} = \mathcal{L}_{ins} + \mathcal{L}_{qua} + \mathcal{L}_{sin} + \mathcal{L}_{cos} + \mathcal{L}_{wid} $$

where $M_{pred}$ and $M_{gt}$ denote the predicted and ground-truth segmentation masks, respectively, and $\alpha$ is a weighting factor that increases the importance of foreground pixels. $Q_{pred}, S_{pred}, C_{pred}, W_{pred}$ represent the predicted grasp quality, sine, cosine, and width maps, while $Q_{gt}, S_{gt}, C_{gt}, W_{gt}$ denote their corresponding ground-truth maps. All predicted and ground truth maps are of size 1×104×104.

Experiments

Evaluation Metrics

For referring image segmentation (RIS), we use Intersection-over-Union (IoU) and Precision@X.

The IoU between the predicted mask and ground-truth mask is defined as: $$ \mathrm{IoU} = \frac{|M_{pred} \cap M_{gt}|}{|M_{pred} \cup M_{gt}|} $$

Precision@X measures the percentage of predictions whose IoU exceeds a threshold $X$: $$ \mathrm{Precision@X} = \frac{1}{N} \sum_{i=1}^{N} [\mathrm{IoU}_i \geq X] $$

For referring grasp synthesis (RGS), we use the Jacquard index (J@N).

A predicted grasp is considered correct if: $$ \mathrm{IoU}(G_{pred}, G_{gt}) > 0.25 \quad \text{and} \quad |\theta_{pred} - \theta_{gt}| < 30^\circ $$

The Jacquard index is then defined as: $$ J@N = \frac{1}{N} \sum_{i=1}^{N} [\text{grasp}_i \ \text{is correct}] $$

where $M_{pred}, M_{gt}$ denote predicted and ground-truth masks, and $G_{pred}, G_{gt}$ denote predicted and ground-truth grasp rectangles.

Comparing State-of-the-Art Approaches

Several recent models have attempted to process the standard language-to-grasping dataset (OCID-VLG). To measure true generalization, the table below outlines how previous methods compare to the proposed GeoLanG framework on the dataset’s “novel instance” split (testing specifically on unseen objects the models were not trained on):

Table 1: Comparison with prior methods on the OCID-VLG novel instance split.

Main Training Hyperparameters

Table 2: Main training hyperparameters used for GeoLanG.

Results

We reproduce the proposed framework utilizing pretrained CLIP-VMamba for visual encoding, CLIP-BERT for language understanding, the DGGM and ADCI modules for geometry-aware multi-scale feature fusion, and a transformer-based multi-task decoder for joint segmentation and grasp prediction. The segmentation and grasp projector is implemented to generate task-specific outputs, and the model is trained using a combination of binary cross-entropy loss for segmentation and Smooth L1 loss for grasp prediction.

Quantitative Results

We evaluate our model on the OCID-VLG test dataset for both referring image segmentation (RIS) and referring grasp synthesis (RGS). The results are reported using IoU and Precision@X for segmentation, and Jacquard index (J@N) for grasp prediction.

Table 3: Reported and reproduced RIS and RGS metrics on the OCID-VLG test set (bold indicates the better value).

Our reproduction closely matches the reported performance, with comparable IoU and precision across most thresholds. Notably, the reproduced model achieves higher performance at stricter thresholds (Pr@0.8, Pr@0.9), suggesting sharper predictions, while slightly underperforming in overall IoU and grasp metrics.

Figure 10: Comprehensive overview of training and validation metrics. Subplots (A) through (F) track the evolution of Mean IoU, multi-threshold precision, the train-validation generalization gap, and logarithmic loss convergence across 50 epochs, culminating in the final test set grasping and segmentation scores.

Qualitative Results

We present qualitative results demonstrating the model’s ability to localize target objects and predict corresponding grasp configurations under different language queries. For each example, the figure shows the RGB input and referring expression, together with the predicted instance segmentation mask, grasp rectangle, and grasp maps for quality, angle, and width. Figure 11 shows that, in both examples, the model accurately localizes the referred object and predicts a compact high-quality grasp region, indicating feasible grasp predictions.

Figure 11: Grasp predictions on two novel cluttered scenes for the commands “Get the glue” and “Grab the food bag”.

Ablations

We conduct ablation studies to evaluate the contribution of the DGGM and ADCI modules, which form the core components of the GeoLanG architecture. Other components, such as the multimodal cross-decoder and the task projectors, are standard architectural choices adopted from prior work [3] and are kept fixed to isolate the effects of the proposed modules. Specifically, we consider two variants:

1. Model without DGGM (only ADCI enabled):
   The motivation behind the experiment is to understand the effect of ADCI independently, comparing its contribution against DGGM, analyze how the absence of depth-based geometric priors affects the predictions, and determine whether DGGM or ADCI is responsible for the performance drop relative to the original paper.

2. Model without ADCI (only DGGM enabled):
   The motivation behind the experiment is to isolate the impact of DGGM, studying how incorporating depth-guided geometric information influences the predictions, and determine whether DGGM or ADCI is responsible for the performance drop relative to the original paper.

Quantitative Results (OCID-VLG Test Set)

Table 4: Ablation results showing the effect of DGGM and ADCI. The highest value is highlighted in bold.

These results suggest that while ADCI contributes strongly to performance, DGGM may be suppressing useful information or not being optimally tuned in the current setup.

To understand the impact of DGGM, we analyze how it modifies feature representations at different stages of the CLIP-VMamba backbone and how this propagates to the ADCI embedding (Fig. 12).

Across feature scales, DGGM suppresses surrounding activations and retains regions aligned with depth cues. This effect is most pronounced in the 52×52×512-channel feature, where the object region becomes clearly isolated. However, at deeper semantic levels such as 26×26×1024 or 13×13×1024, the suppression becomes increasingly aggressive, leading to loss of useful contextual and high-level information.

52×52×512 feature map

26×26×1024 feature map

13×13×1024 feature map

Figure 12: DGGM feature maps at three scales, shown before and after depth refinement. The 52×52×512 feature map shows a stronger suppression of responses from spatially distant regions, while the 26×26×1024 and 13×13×1024 feature maps are less visually interpretable.

ADCI embedding with DGGM off

ADCI embedding with DGGM on

Figure 13: ADCI embeddings with DGGM disabled and enabled, shown alongside the corresponding RGB image and overlay. With DGGM enabled, background activations are significantly suppressed, leading to more focused responses.

With DGGM enabled (Fig. 13), the ADCI embedding shows significantly reduced background activation, indicating strong filtering. However, this also reduces contextual richness, suggesting that the current decay setting (η = 0.75) may be overly aggressive.

It is also important to note that the current implementation uses a single shared decay factor (η = 0.75) across all feature channels and scales. This uniform decay may not be optimal, as different feature levels and channels may require different degrees of geometric suppression.

The original paper mentions that this formulation can be extended to multi-head attention by assigning different decay rates (η) per head, allowing more flexible modeling of spatial dependencies. Motivated by this, future experiments will explore varying η across channels (or feature groups) to reduce over-suppression and improve scale-aware behavior. These observations can be extended into exploration of tuning the decay factor, adjusting the relative weighting between the depth and spatial prior and selectively applying DGGM to earlier feature layers where spatial structure is more prominent.

Conclusion

In this work, we reproduced the GeoLanG framework and analyzed its ability to integrate geometric priors with language-guided perception for robotic grasping. Our implementation closely matches the reported performance, validating the effectiveness of integrating RGB-D features, depth-aware geometric priors, multi-scale feature fusion, and task-driven language understanding within a unified framework.

Through ablation studies, we observed that while ADCI consistently improves performance by enhancing multi-scale feature fusion, DGGM requires careful tuning, as overly strong geometric suppression can lead to loss of useful contextual information.

References

[1] I. Lenz et al., Deep Learning for Detecting Robotic Grasps, arXiv:1301.3592, 2013.

[2] D. Morrison et al., Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach, arXiv:1804.05172, 2018.

[3] G. Tziafas et al., Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter, arXiv:2311.05779, 2023.

[4] B.-W. Yin et al. DFormerV2: Geometry Self-Attention for RGBD Semantic Segmentation arXiv:2504.04701, 2025.

[5] Huang, W., Shen, Y. and Yang, Y., 2024. Clip-mamba: Clip pretrained mamba models with ood and hessian evaluation. arXiv preprint arXiv:2404.19394.

[6] R. Tang et al. GeoLanG: Geometry-Aware Language-Guided Grasping with Unified RGB-D Multimodal Learning arXiv:2602.04231, 2026.

[7] M. Suchi et al. OCID: Object Clutter Indoor Dataset, 2019.

[8] G. Tziafas and H. Kasaei, Towards Open-World Grasping with Large Vision-Language Models, 2024, https://arxiv.org/abs/2406.18722

[9] CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning, 2016, https://arxiv.org/abs/1612.06890

[10] G. Tziafas and H. Kasaei, Towards Open-World Grasping with Large Vision-Language Models, 2024, https://arxiv.org/abs/2406.18722

[11] H. Liu, D. Li, P. Wang, F. Yu, and X. Chang, Task-Oriented Grasp Prediction with Visual-Language Inputs, 2023, https://arxiv.org/abs/2302.14355

[12] M. Shridhar, L. Manuelli, and D. Fox, CLIPort: What and Where Pathways for Robotic Manipulation, 2021, https://arxiv.org/abs/2109.12098

[13] J. Xie, J. Liu, S. Huang, C. Wang, and F. Zhou, Listen, Perceive, Grasp: CLIP-Driven Attribute-Aware Network for Language-Conditioned Visual Segmentation and Grasping, 2024, https://ieeexplore.ieee.org/document/10786319

[14] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling, 2021, https://arxiv.org/abs/2102.06183