Pair a vision grounding model with a reasoning LLM with Cua
text
Pair a vision grounding model with a reasoning LLM with Cua
new_line
new_line
Cua just shipped v0.4 of the Cua Agent framework with Composite Agents - you can now pair a vision/grounding model with a reasoning LLM using a simple modelA+modelB syntax. Best clicks + best plans.
text
Cua just shipped v0.4 of the Cua Agent framework with Composite Agents - you can now pair a vision/grounding model with a reasoning LLM using a simple modelA+modelB syntax. Best clicks + best plans.
new_line
new_line
The problem: every GUI model speaks a different dialect.
text
The problem: every GUI model speaks a different dialect.
new_line
    •    some want pixel coordinates
text
    •    some want pixel coordinates
new_line
    •    others want percentages
text
    •    others want percentages
new_line
    •    a few spit out cursed tokens like <|loc095|>
text
    •    a few spit out cursed tokens like <|loc095|>
new_line
new_line
We built a universal interface that works the same across Anthropic, OpenAI, Hugging Face, etc.:
text
We built a universal interface that works the same across Anthropic, OpenAI, Hugging Face, etc.:
new_line
new_line
agent = ComputerAgent(
text
agent = ComputerAgent(
new_line
    model="anthropic/claude-3-5-sonnet-20241022",
text
    model="anthropic/claude-3-5-sonnet-20241022",
new_line
    tools=[computer]
text
    tools=[computer]
new_line
)
text
)
new_line
new_line
But here’s the fun part: you can combine models by specialization.
text
But here’s the fun part: you can combine models by specialization.
new_line
Grounding model (sees + clicks) + Planning model (reasons + decides) →
text
Grounding model (sees + clicks) + Planning model (reasons + decides) →
new_line
new_line
agent = ComputerAgent(
text
agent = ComputerAgent(
new_line
    model="huggingface-local/HelloKKMe/GTA1-7B+openai/gpt-4o",
text
    model="huggingface-local/HelloKKMe/GTA1-7B+openai/gpt-4o",
new_line
    tools=[computer] 
text
    tools=[computer] 
new_line
)
text
)
new_line
new_line
This gives GUI skills to models that were never built for computer use. One handles the eyes/hands, the other the brain. Think driver + navigator working together.
text
This gives GUI skills to models that were never built for computer use. One handles the eyes/hands, the other the brain. Think driver + navigator working together.
new_line
new_line
Two specialists beat one generalist. We’ve got a ready-to-run notebook demo - curious what combos you all will try.
text
Two specialists beat one generalist. We’ve got a ready-to-run notebook demo - curious what combos you all will try.
new_line
new_line
Github : 
text
Github : 
https://github.com/trycua/cua
https://github.com/trycua/cua
link
new_line
new_line
Blog : 
text
Blog : 
https://www.trycua.com/blog/composite-agents
https://www.trycua.com/blog/composite-agents
link
Pair a vision grounding model with a reasoning LLM with Cua

Cua just shipped v0.4 of the Cua Agent framework with Composite Agents - you can now pair a vision/grounding model with a reasoning LLM using a simple modelA+modelB syntax. Best clicks + best plans.

The problem: every GUI model speaks a different dialect.
    •    some want pixel coordinates
    •    others want percentages
    •    a few spit out cursed tokens like <|loc095|>

We built a universal interface that works the same across Anthropic, OpenAI, Hugging Face, etc.:

agent = ComputerAgent(
    model="anthropic/claude-3-5-sonnet-20241022",
    tools=[computer]
)

But here’s the fun part: you can combine models by specialization.
Grounding model (sees + clicks) + Planning model (reasons + decides) →

agent = ComputerAgent(
    model="huggingface-local/HelloKKMe/GTA1-7B+openai/gpt-4o",
    tools=[computer] 
)

This gives GUI skills to models that were never built for computer use. One handles the eyes/hands, the other the brain. Think driver + navigator working together.

Two specialists beat one generalist. We’ve got a ready-to-run notebook demo - curious what combos you all will try.

Github : https://github.com/trycua/cua

Blog : https://www.trycua.com/blog/composite-agents
🔥
dhruv3006 ThijsL202 thefloweri77 olivia2023 ShreeshaBhat1004
🚀
dhruv3006 John6666 ThijsL202