← all ideas

AI Agents and Web Traffic

The idea

The idea involves switching from using chat GPT's agent mode, which relies on raw HTML for context, to utilizing MCP tools for API communication. This shift is based on the realization that navigating web pages using agent mode is inefficient and contextually limited. MCP tools, as standardized communication methods for APIs, offer a more streamlined approach with less context required. This change is particularly relevant given the forecast in the transcript that 99% of web traffic could be AI bots within five years. To remain relevant and efficient, businesses need to adapt to AI’s capabilities, adopting technologies like MCP tools to better leverage AI for web interactions.

Why it works

• The author switched from using agent mode, which involves looking at the raw HTML and dumping the entire page into the context window, to MCP tools. MCP tools receive just the API documentation, significantly reducing the context needed for interaction.
• This shift allowed for more efficient and less context-intensive communication with APIs, which is crucial for handling the vast amounts of data involved in web traffic navigation.
• By focusing on API documentation, the new approach reduces the cognitive load on AI, making it more effective and adaptable to the dynamic nature of web interactions.
• The author suggests that by the time 99% of traffic is predicted to be AI, traditional human-like navigation would be impractical, emphasizing the need for AI to operate more like APIs.
• The author implies that as AI evolves, it will need to be more API-like in its interactions, streamlining processes and making them more efficient, which is a key reason why this approach works.

The playbook

The playbook

1. Switch to MCP tools: Begin by adopting the MCP (Anthropic Communication Protocol) tools, which are designed to simplify interactions with APIs. Unlike the raw HTML approach, these tools provide just the API documentation, significantly reducing the context needed for each interaction. This shift will make your AI more efficient and less dependent on detailed screen captures.

1. Implement MCP tools: Start by integrating MCP tools into your workflow. Focus on automating tasks that were previously handled manually, such as form submissions or button clicks. This automation will help in creating a more streamlined and efficient system, especially when dealing with multiple interactions.

1. Streamline communication: Once you have MCP tools in place, focus on refining how you communicate with these tools. Practice sending concise API requests and receiving clear responses. Understanding the API structure and documentation will help you tailor your interactions for maximum efficiency.

1. Test and iterate: After implementing MCP tools, test your system thoroughly. Ensure that the automated interactions are working as intended and that the responses are accurate and relevant to your needs. Use the feedback from these tests to make necessary adjustments and improvements.

1. Monitor and adapt: As your system grows and evolves, continue to monitor its performance. Keep an eye on any changes in user behavior or API documentation that might affect your system. Be prepared to adapt your approach as needed to maintain its effectiveness and relevance.

Where people get it wrong

Where people get it wrong:

• People often assume AI can perfectly mimic human behavior. Instead, AI should be designed to leverage its strengths, such as processing vast amounts of data and pattern recognition, without attempting to replicate human cognition. For instance, instead of using Playwright or Puppeteer for complex web interactions, consider using MCP tools that handle API documentation more efficiently, reducing unnecessary context.

• Some predict AI will navigate the web like humans. This overlooks the fundamental differences in how humans and AI process information. AI should be optimized for tasks where it can excel, such as automating repetitive tasks or conducting research, rather than mimicking human navigation. For example, instead of relying on agent modes, switch to more efficient communication methods like MCP tools.

• There is a misconception that AI will handle all web traffic. The transcript suggests that AI might manage significant portions of web traffic in the future, but it should not do so at the expense of human oversight. Instead, AI should augment human efforts, not replace them. For instance, use AI to gather and analyze data, freeing humans to focus on strategic decision-making and human-centric tasks.

Do this next

Do this next:

• Switch from using Playwright or Puppeteer for your agent mode to using MCP tools. MCP tools are standardized for communicating with APIs and significantly reduce the context needed compared to raw HTML pages.
• Start by reviewing the API documentation for any tools you use to ensure you're only receiving the necessary information, not the entire page.
• Begin testing the new MCP tools to see how they perform with your specific APIs and workflows. Make adjustments as needed to optimize efficiency.
• Collaborate with other developers or AI experts to share experiences and best practices with MCP tools, as they can provide valuable insights and optimizations.