🎯 VCoder is an adapter for improving MLLMs at object-level perception tasks with the aid of auxiliary perception modalities as control inputs. 🎁 We also release the COST dataset to train and evaluate MLLMs at object-level perception tasks!

为了开发用于感知和推理的精确 MLLM 系统，文中使用 多功能视觉编码器 (VCoder) 作为多模态 LLM 的感知眼睛。 向 VCoder 提供感知模式，例如分割或深度图，从而提高 MLLM 的感知能力。 其次，利用 COCO 的图像和现成的视觉感知模型的输出来创建 COCO 分割文本 (COST) 数据集，用于在对象感知任务上训练和评估 MLLM。 第三，引入了评估 MLLM 在 COST 数据集上的物体感知能力的指标。 最后，提供了大量的实验证据，证明 VCoder 相对于现有的多模态 LLM（ …

Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as segmentation or depth maps, improving the MLLM's perception abilities.

Dec 21, 2023 · Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as segmentation or depth maps, improving the MLLM's perception abilities.

This document provides a brief intro to the usage of VCoder LLaVA-1.5. Our code is based on original LLaVA, please checkout their repo for more information.

VCoder: Versatile Vision Encoders for Multimodal Large Language Models, arXiv 2023 / CVPR 2024 - VCoder/docs/Data.md at main · SHI-Labs/VCoder

Jan 5, 2024 · 1、增强视觉感知能力：VCoder通过提供额外的视觉编码器，帮助MLLM更好地理解和分析图像内容。 2、处理特殊类型的图像：VCoder能够处理分割图和深度图等特殊类型的图像。 分割图可以帮助模型识别和理解图像中不同物体的边界和形状，而深度图则提供了物体距离相机远近的信息。 3、改善对象感知任务：VCoder通过提供额外的感知模态输入（如分割图或深度图）显著提高了MLLMs的对象感知能力。 这包括更准确地识别和计数图像中的对象。 实验结果. …

Nov 8, 2024 · 为了开发一个准确的多模态MLLM感知和推理系统，我们提出使用 V ersatile en Coder （VCoder）作为多模态MLLM的感知之眼。 我们将分割或深度图等感知模态输入 …

Jan 7, 2024 · Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as segmentation or depth maps, improving the MLLM's perception abilities.

Working towards developing an accurate MLLM system for percep-tion and reasoning, we propose using Versatile vision en-Coders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as seg-mentation or depth maps, improving the MLLM’s percep-tion abilities.