Revolutionize Web Browsing with AI
· Introduction · Potential Use Cases · High-level Workflow · Architecture ∘ Starting Out ∘ Pick the Right Path ∘ IT IS A LOOP! ∘ Directory Structure ∘ Browser Controller Service ∘ Element Annotation Service · Conclusion
Introduction
Imagine you are keen on attending an AI event in your city this month, but you have specific criteria in mind, perhaps related to timing or the focus of the event. Normally, this would involve the following process:
- Launching a web search with terms like "AI events in [your city] this month."
- Sifting through search results to find a link that seems promising.
- Navigating the chosen website to determine its relevance, possibly needing to delve deeper through additional links.
- After much back and forth, finally pinpointing the event that fits your criteria and noting its details for your calendar.
If we break down the above process, it basically involves steps that can be categorized into the following:
- Control the browser, such as go to a URL, click on a link, go back, etc.
- Browse through the content of a page
- Make decisions based on the content of that page, such as determining which link is relevant to your query.
By utilizing the emerging Large Language Model (LLM) technology, now we are able to automate the whole process through a LLM powered AI Agent.
Enter the AI agent, it does exactly what you do as we described above.
- Browser Control: the AI uses tools like
Puppeteerto navigate the internet. Think of Puppeteer as the AI's hands, allowing it to open tabs, click on links, and navigate web pages with ease. - Content Browsing: Think of this as the AI's eyes.
Puppeteercan take screenshots of web pages, and feed them to the AI. - Decision-Making: This is where the AI's brain, powered by Large Language Models (LLM), comes into play. It assesses the screenshot of each page, analyzing the image, determining relevance and deciding on the next steps, mimicking human judgment.
In this article, we will explore and build an AI agent that utilizes the power of the gpt-4-vision-preview model from OpenAI. The model can analyze images and provide textual responses.
This agent will be able to interact with the user, control a web browser, and process data. We'll explore its structure and how it works.
This article is inspired by a youtube video GPT4V + Puppeteer = AI agent browse web like human?
