Get all your news in one place.

100’s of premium titles.
One app.

Start reading

Get all your news in one place.

100’s of premium titles. One news app.

Start reading

Windows Central

Technology

Kevin Okemwa

Anthropic's upgraded Claude AI model outperforms OpenAI-o1 in coding and can use a Windows 11 PC like humans — potentially backing NVIDIA CEO's claim about software development being dead in the water

Anthropic OpenAI Nvidia Amazon HP LG

Anthropic's upgraded Claude 3.5 Sonnet AI model.

What you need to know

Anthropic recently shipped an upgraded version of Claude 3.5 Sonnet alongside a new model dubbed Claude 3.5 Haiku, with enhanced coding capabilities and more.
The AI firm also unveiled computer use, a new capability that allows users to prompt Claude to use computers as people do.
The company admits shipping the capability to the public poses great risks, but it plans will use the avenue to observe how people are leveraging the tool. It has elaborate measures in place to prevent misuse, such as restricted access to the web during training.

The generative AI landscape is seemingly transitioning to the next phase beyond AI-generated images and text. Anthropic recently unveiled an upgraded version of Claude 3.5 Sonnet and a new model dubbed Claude 3.5 Haiku. According to the company, the upgraded version ships with enhanced coding capabilities and shares the same performance specs as Anthropic's Claude 3 Opus LLM.

More interestingly, the new capability dubbed “Computer Use”, which is available in open beta. Through the API, developers can "can direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text." This makes Claude 3.5 Sonnet the first AI model to provide computer use in public beta.

Anthropic admits that users could encounter several setbacks while interacting with the model, including errors and a not-so-seamless user experience. The company hopes to use feedback to enhance and improve the model's performance and efficiency.

Companies like Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company have joined the fold to simplify processes that often require dozens of steps. For instance, "Replit is using Claude 3.5 Sonnet's capabilities with computer use and UI navigation to develop a key feature that evaluates apps as they’re being built for their Replit Agent product."

The upgraded version of Claude 3.5 Sonnet is available on the Anthropic API, Amazon Bedrock, and Google Cloud’s Vertex AI. Anthropic is expected to ship Claude 3.5 Haiku later this month.

Per benchmarks shared, Anthropic's updated Claude 3.5 Sonnet shows a significant performance boost, especially in coding. For instance, the tool's performance on SWE-bench Verified has improved from 33.4% to 49.0%, which indicates it performs significantly better than publicly available models, including OpenAI Strawberry reasoning AI models while maintaining the same price and speed as its predecessor.

The model corrects its mistakes by making another attempt at a task when it "realizes" it has encountered an issue, swaying it away from the desired output. As you may know, OpenAI o1 and o1-mini are exceptionally great at coding and have passed OpenAI's research engineer hiring interview for coding at a 90-100% rate.

AI agents are here but proceed with caution

While the highlighted improvements are impressive, the updated Claude 3.5 Sonnet AI model completed less than half of the tasks assigned in an evaluation designed to establish its proficiency in modifying flight reservations. The model failed approximately a third of the time while attempting to initiate a return.

Anthropic highlights the model struggles with zooming and scrolling, making it easy to miss pop-up notifications because of how it processes screenshots. “Claude’s Computer Use remains slow and often error-prone,” the company added.

The company admits that releasing the model to the public poses significant risks but also outlines that the benefits of observing how the model is used outweigh the dangers.

According to Anthropic:

"We think it’s far better to give access to computers to today’s more limited, relatively safer models. This means we can begin to observe and learn from any potential issues that arise at this lower level, building up computer use and safety mitigations gradually and simultaneously."

In an attempt to prevent misuse and bad actors from leveraging the tool's sophisticated capabilities to cause harm, the new Claude 3.5 Sonnet isn't trained on users’ screenshots and prompts. It's also restricted from accessing the web during training. Anthropic developed the model with classifiers, swaying it away from high-risk actions like creating accounts and posting on social media.