![\"Diagram](\"https://dcbllm8dvghjo.cloudfront.net/media/blog/ad6ff0f35988c6c7c3eb0e197525e1934ea5fff2d0ac0ca1eafc019dcf3ae8a1-827w.webp\")
TL;DR - We rewrote Skyvern from scratch using Pi as inspiration, making it 2.3x faster and score 90.5% on Odysseys benchmark. Try it out the via Open Source or Cloud*.*
\n💡 Recap: What is Skyvern? It’s an open source framework that helps non-technical teams prompt to build browser automations. Skyvern is model agnostic, and can be run with closed models like Claude, or open ones like Deepseek. We have helped thousands of companies automate things like interacting with healthcare portals, fetching invoices or utility bills, filling out government forms, applying to jobs, and more.
\nFor those unfamiliar with agents, RAG stands for Retrieval Augmented Generation. It’s a way to pass information to an LLM so they make smarter decisions. For example, before asking questions to an LLM about a document, you could parse that document and feed it into an LLM. You can read more about it here.
\nRAG-based LLM Frameworks work by fetching information from a Knowledge base, transforming it into context, and passing it to an LLM to make decisions on
\nThe first version of Skyvern was built using RAG. We would load up a website, screenshot and annotate it, condense it into a smaller list of elements, and send it to an LLM to take actions on.
\n![\"Diagram](\"https://dcbllm8dvghjo.cloudfront.net/media/blog/2649dd5c355815ca96745ce5e55d8a7fc7ffacb04744b96b13378f80a3794e26-918w.webp\")
First version of Skyvern mimicked RAG based frameworks: It fetched information from the website, transformed it into actionable elements, and asked the LLM what action to take
\nThis worked great if you could identify every interactable element on a website with 100% accuracy. We thought we could eventually get there. We were wrong.
\nExcept.. the web is messy. Accurately identifying every element on a website proved to be a nightmare. What started off as a nice little utility to identify interactable elements essentially turned into building a new DOM interpreter.
\nParsing things like Shadow DOMs, iframes, select2 dropdowns, Kendo UI widgets, Canvas elements, …. this became an endless battle.
\nThe worst part was that agent performance did not improve with better LLMs. Instead, it was bottlenecked by this DOM parsing logic. If it failed to correctly identify an element, the entire agent run failed.
\nThis is a sign that we need to give the LLM more freedom.
\nRAG systems, if you aren’t careful, can assume that the pre-processing layer always happens ahead of any LLM calls, and don’t let the LLM decide when it is necessary. This was a good idea when LLMs weren’t good at following instructions, which meant that you couldn’t trust them to reliably pull the information when needed, but instruction following hasn’t been a big problem since the models released after Opus 4.5.
\nWe noticed 2 moves in the market that highlighted this change in thinking:
\nSo we asked ourselves: would the same approach work for browser agents? What if the agent started with the minimum context required, and gave it the ability to pull additional context as needed?
\nIn practice, it means giving the agent the following tools:
\n![\"Diagram](\"https://dcbllm8dvghjo.cloudfront.net/media/blog/c44ed2d10a248f0a1ce083ff13a6810458bfdd5b90fde66a133fe9e4766cf29d-1966w.webp\")
Our hypothesis was that we would see an increase in performance, in exchange for added cost and a slower processing speed. The new architecture should result more calls to the LLM to get the same task done.
\nTo test this hypothesis, we split the test up into two branches:
\nSkyvern 3.0 topped the Odysseys benchmark. It reported a 90.5% perfect rubric success rate, pretty much exactly as we expected. What surprised us though was that it was highly efficient in its runs, at only 65.4 average steps.
\n![\"Odysseys](\"https://dcbllm8dvghjo.cloudfront.net/media/blog/91ed1b57fd9186c4fa363fd87a15af9f0e3a39679c9329d8166ba8d935d392f2-2000w.webp\")
We ran the production A/B test on approx 100,000 runs and measured the impact
\n| Skyvern 2.0 | Skyvern 3.0 | Improvement | |
|---|---|---|---|
| Mean run time | 593 s | 262s | 2.3x |
| Total cost per run | $0.039 | $0.0302 | -22.5% |
The production A/B test results were surprising. The new architecture was.. faster and cheaper? We expected at least one of these dimensions to be a net negative
\nWhy was that the case? Digging into it, we found a few interesting things:
\nDespite the increased token usage, cached tokens carry a 70% discount, which explains the net reduction in cost.
\nThe answer is still no. They can still be improved across all 3 dimensions (cost, speed, accuracy).
\nOur team is now exploring the role of ”system one models” like Jev for browser agents. The current generation is still highly limited: 64K context window, no image processing, which causes unacceptable losses in accuracy, but we expect this to change over the next 6-12 months 🙂
\nRun it via our Skyvern Open Source or Skyvern Cloud versions and let us know what you think!
","headings":[{"level":2,"text":"RAG? What are you talking about?","id":"rag-what-are-you-talking-about"},{"level":2,"text":"Give the LLM freedom to do what it wants","id":"give-the-llm-freedom-to-do-what-it-wants"},{"level":2,"text":"The results were better than we expected","id":"the-results-were-better-than-we-expected"},{"level":3,"text":"Odysseys benchmark results","id":"odysseys-benchmark-results"},{"level":3,"text":"Production A/B Test results","id":"production-a-b-test-results"},{"level":2,"text":"So… are browser agents solved?","id":"so-are-browser-agents-solved"},{"level":2,"text":"Give Skyvern a try!","id":"give-skyvern-a-try"}]}}