Refer to Anything with Vision-Language Prompts

We incorporate visual reference into text prompts in Omnimodal Referring Expression Segmentation (ORES), providing more flexible and intuitive interactions. ORES enables applications such as precisely localized image editing.

We propose a new multimodal model that can "Refer to Any Segmentation Mask Group" (RAS), which augments segmentation models with complex multimodal interactions and comprehension via a mask-centric design.

We curate two new datasets, MaskGroups-2M and MaskGroups-HQ, for large-scale training and high-quality finetuning/evaluation of RAS, respectively.