Let Multimodal Embedders Learn When to Augment Query via Adaptive Query
  Augmentation
Query augmentation makes queries more meaningful by appending further
information to the queries to find relevant documents. Current studies have
proposed Large Language Model (LLM)-based embedders, which learn representation
for embedding and generation for query augmentation in a multi-task manner by
leveraging the generative capabilities of LLM. During inference, these jointly
trained embedders have conducted query augmentation followed by embedding,
showing effective results. However, augmenting every query leads to substantial
embedding latency and query augmentation can be detrimental to performance for
some queries. Also, previous methods have not been explored in multimodal
environments. To tackle these problems, we propose M-Solomon, a universal
multimodal embedder that can adaptively determine when to augment queries. Our
approach first divides the queries of the training datasets into two groups at
the dataset level. One includes queries that require augmentation and the other
includes queries that do not. Then, we introduces a synthesis process that
generates appropriate augmentations for queries that require them by leveraging
a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation.
Through this step, M-Solomon can conduct query augmentation only when necessary
by learning to generate synthetic augmentations with the prefix /augment for
queries that demand them and to generate the simple string /embed for others.
Experimental results showed that M-Solomon not only surpassed the baseline
without augmentation by a large margin but also outperformed the baseline that
always used augmentation, providing much faster embedding latency.
690aafdad70e173c84528f82
Wongyu Kim
claimed_verified
2025-11-05T16:24:45.834Z
64d4615cf8082bf19b916492
/avatars/8e1b59565ec5e4b31090cf1b911781b9.svg
wongyukim
wongyukim
user
wongyukim
690aafdad70e173c84528f83
Hochang Lee
690aafdad70e173c84528f84
Sanghak Lee
claimed_verified
2025-11-10T09:32:32.180Z
676a420bae8a3fdaa04f3e68
/avatars/ea4c24134bb9d05b9e41980fc9f83fa4.svg
Sanghak Lee
SanghakLee
user
SanghakLee
690aafdad70e173c84528f85
Yoonsung Kim
690aafdad70e173c84528f86
Jaehyun Park
Let Multimodal Embedders Learn When to Augment Query via Adaptive Query
  Augmentation
Query augmentation makes queries more meaningful by appending further
information to the queries to find relevant documents. Current studies have
proposed Large Language Model (LLM)-based embedders, which learn representation
for embedding and generation for query augmentation in a multi-task manner by
leveraging the generative capabilities of LLM. During inference, these jointly
trained embedders have conducted query augmentation followed by embedding,
showing effective results. However, augmenting every query leads to substantial
embedding latency and query augmentation can be detrimental to performance for
some queries. Also, previous methods have not been explored in multimodal
environments. To tackle these problems, we propose M-Solomon, a universal
multimodal embedder that can adaptively determine when to augment queries. Our
approach first divides the queries of the training datasets into two groups at
the dataset level. One includes queries that require augmentation and the other
includes queries that do not. Then, we introduces a synthesis process that
generates appropriate augmentations for queries that require them by leveraging
a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation.
Through this step, M-Solomon can conduct query augmentation only when necessary
by learning to generate synthetic augmentations with the prefix /augment for
queries that demand them and to generate the simple string /embed for others.
Experimental results showed that M-Solomon not only surpassed the baseline
without augmentation by a large margin but also outperformed the baseline that
always used augmentation, providing much faster embedding latency.
M-Solomon, a multimodal embedder, adaptively augments queries using a Multimodal LLM, improving performance and reducing embedding latency compared to baselines.
Large Language Model (LLM) embedders query augmentation multimodal environments Multimodal LLM (MLLM) adaptive query augmentation