Can you beat an AI at Raven puzzles?
text
Can you beat an AI at Raven puzzles?
new_line
new_line
https://huggingface.co/spaces/HuggingFaceM4/ai_raven
HuggingFaceM4/ai_raven
space
resource
https://huggingface.co/spaces/HuggingFaceM4/ai_raven
new_line
new_line
The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
text
The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
How Far Are We from Intelligent Visual Deductive Reasoning? (2403.04732)
https://huggingface.co/papers/2403.04732
2403.04732
paper
resource
https://huggingface.co/papers/2403.04732
).
text
).
new_line
new_line
But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
text
But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
new_line
new_line
Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
text
Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
new_line
new_line
Such puzzles can be procedurally generated at scale. 
text
Such puzzles can be procedurally generated at scale. 
https://huggingface.co/datasets/HuggingFaceM4/RAVEN
HuggingFaceM4/RAVEN
dataset
resource
https://huggingface.co/datasets/HuggingFaceM4/RAVEN
 is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
text
 is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
new_line
new_line
We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
text
We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
new_line
new_line
Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
text
Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
Can you beat an AI at Raven puzzles?

https://huggingface.co/spaces/HuggingFaceM4/ai_raven

The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (https://huggingface.co/papers/2403.04732).

But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!

Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.

Such puzzles can be procedurally generated at scale. https://huggingface.co/datasets/HuggingFaceM4/RAVEN is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 

We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.

Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
❤️
samusenps osanseviero clefourrier
🔥
yizheapple
65f819cc01523ed703872006
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YurLUFRKxyg3R753mCF_V.jpeg
Kelvin Bruce 
Kelvin19
user