Web Browsing and Scraping tools enable AI Agents to gather information from various online sources, including search engines, social media platforms, websites, and multimedia content. These tools provide capabilities for both broad web searches and targeted data extraction.

Available Tools

Tool NamePlatform/CategoryDescription
DuckDuckGo SearchSearch EngineBasic web search using DuckDuckGo
Tavily SearchSearch EngineAdvanced semantic search with AI-optimized results
LinkedIn Profile ScraperSocial MediaExtracts detailed information from LinkedIn profiles
Twitter Profile ScraperSocial MediaCollects data from Twitter/X user profiles
Twitter Search ScraperSocial MediaGathers tweets based on search terms
Website ScraperWebExtracts text content from web pages
Video Transcript ExtractorMultimediaRetrieves transcripts from video/audio content
Wikipedia SearchKnowledge BaseSearches and retrieves Wikipedia article content

Tool Details

Description: A basic web search tool that queries DuckDuckGo search engine to find relevant web pages and information.

System Tool ID: duckduckgo-search

Arguments:

NameRequiredTypeDescription
maxResultsOptional (default: 5)numberMaximum number of search results to return

Description: An advanced search tool that uses AI to provide more relevant and contextual search results.

System Tool ID: tavily-search

Arguments:

NameRequiredTypeDescription
maxResultsOptional (default: 5)numberMaximum number of search results to return

LinkedIn Profile Scraper

Description: Extracts comprehensive information from LinkedIn profiles including work experience, education, skills, and other public profile data.

System Tool ID: linkedin_scrape_profiles_by_urls

Arguments:

NameRequiredTypeDescription
profileUrlsRequiredstring[]Array of LinkedIn profile URLs to scrape

Twitter Profile Scraper

Description: Collects data from Twitter/X user profiles including tweets, profile information, and public metrics.

System Tool ID: twitter_scrape_by_handles

Arguments:

NameRequiredTypeDescription
twitterHandlesRequiredstring[]Array of Twitter handles to scrape
startOptional (default: ”)stringStart date in YYYY-MM-DD format
endOptional (default: ”)stringEnd date in YYYY-MM-DD format
sortOptional (default: ‘Top’)enum: [‘Latest’, ‘Top’]Sort order of tweets
maxItemsOptional (default: ‘10’)enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]Maximum number of items to return

Twitter Search Scraper

Description: Searches and extracts tweets based on specific search terms or keywords.

System Tool ID: twitter_scrape_by_search_terms

Arguments:

NameRequiredTypeDescription
searchTermsRequiredstring[]Array of search terms
startOptional (default: ”)stringStart date in YYYY-MM-DD format
endOptional (default: ”)stringEnd date in YYYY-MM-DD format
sortOptional (default: ‘Top’)enum: [‘Latest’, ‘Top’]Sort order of tweets
maxItemsOptional (default: ‘10’)enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]Maximum number of items to return

Website Scraper

Description: Extracts visible text content from any webpage URL, making it ideal for gathering information from articles, blog posts, and other web content.

System Tool ID: scrape_web_text

Arguments:

NameRequiredTypeDescription
urlRequiredstringURL of the webpage to scrape

Video Transcript Extractor

Description: Retrieves transcripts from various online video and audio content across different platforms.

System Tool ID: video_transcript

Arguments:

NameRequiredTypeDescription
video_urlRequiredstringURL of the video/audio content
languageOptionalstringLanguage code for the transcript (e.g., ‘en’, ‘ru’)

Description: Searches Wikipedia articles and retrieves relevant content and information.

System Tool ID: wikipedia-query-run

Arguments:

NameRequiredTypeDescription
topKResultsOptional (default: 3)numberNumber of top results to return
maxDocContentLengthOptional (default: 4000)numberMaximum content length per document

When using search and scraping tools, be mindful of rate limits and usage quotas that may apply to specific services.