Web Browsing and Scraping tools enable AI Agents to gather information from various online sources, including search engines, social media platforms, websites, and multimedia content. These tools provide capabilities for both broad web searches and targeted data extraction.
Available Tools
Tool Name
Platform/Category
Description
Apollo Company Enrichment
Business Data
Search for company details in Apollo database
DuckDuckGo Search
Search Engine
Basic web search using DuckDuckGo
Tavily Search
Search Engine
Advanced semantic search with AI-optimized results
LinkedIn Profile Scraper
Social Media
Extracts detailed information from LinkedIn profiles
Twitter Profile Scraper
Social Media
Collects data from Twitter/X user profiles
Twitter Search Scraper
Social Media
Gathers tweets based on search terms
Website Scraper
Web
Extracts text content from web pages
Video Transcript Extractor
Multimedia
Retrieves transcripts from video/audio content
Wikipedia Search
Knowledge Base
Searches and retrieves Wikipedia article content
Tool Details
Apollo Company Enrichment
Description:
Enriches company data by retrieving detailed information from Apollo’s comprehensive business database.System Tool ID:apollo_organization_enrichmentArguments:
Name
Required
Type
Description
domain
Required
string
Company domain to search for
DuckDuckGo Search
Description:
A basic web search tool that queries DuckDuckGo search engine to find relevant web pages and information.System Tool ID:duckduckgo-searchArguments:
Name
Required
Type
Description
maxResults
Optional (default: 5)
number
Maximum number of search results to return
Tavily Search
Description:
An advanced search tool that uses AI to provide more relevant and contextual search results.System Tool ID:tavily-searchArguments:
Name
Required
Type
Description
maxResults
Optional (default: 5)
number
Maximum number of search results to return
LinkedIn Profile Scraper
Description:
Extracts comprehensive information from LinkedIn profiles including work experience, education, skills, and other public profile data.System Tool ID:linkedin_scrape_profiles_by_urlsArguments:
Name
Required
Type
Description
profileUrls
Required
string[]
Array of LinkedIn profile URLs to scrape
Twitter Profile Scraper
Description:
Collects data from Twitter/X user profiles including tweets, profile information, and public metrics.System Tool ID:twitter_scrape_by_handlesArguments:
Name
Required
Type
Description
twitterHandles
Required
string[]
Array of Twitter handles to scrape
start
Optional (default: ”)
string
Start date in YYYY-MM-DD format
end
Optional (default: ”)
string
End date in YYYY-MM-DD format
sort
Optional (default: ‘Top’)
enum: [‘Latest’, ‘Top’]
Sort order of tweets
maxItems
Optional (default: ‘10’)
enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]
Maximum number of items to return
Twitter Search Scraper
Description:
Searches and extracts tweets based on specific search terms or keywords.System Tool ID:twitter_scrape_by_search_termsArguments:
Name
Required
Type
Description
searchTerms
Required
string[]
Array of search terms
start
Optional (default: ”)
string
Start date in YYYY-MM-DD format
end
Optional (default: ”)
string
End date in YYYY-MM-DD format
sort
Optional (default: ‘Top’)
enum: [‘Latest’, ‘Top’]
Sort order of tweets
maxItems
Optional (default: ‘10’)
enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]
Maximum number of items to return
Website Scraper
Description:
Extracts visible text content from any webpage URL, making it ideal for gathering information from articles, blog posts, and other web content.System Tool ID:scrape_web_textArguments:
Name
Required
Type
Description
url
Required
string
URL of the webpage to scrape
Video Transcript Extractor
Description:
Retrieves transcripts from various online video and audio content across different platforms.System Tool ID:video_transcriptArguments:
Name
Required
Type
Description
video_url
Required
string
URL of the video/audio content
language
Optional
string
Language code for the transcript (e.g., ‘en’, ‘ru’)
Wikipedia Search
Description:
Searches Wikipedia articles and retrieves relevant content and information.System Tool ID:wikipedia-query-runArguments:
Name
Required
Type
Description
topKResults
Optional (default: 3)
number
Number of top results to return
maxDocContentLength
Optional (default: 4000)
number
Maximum content length per document
When using search and scraping tools, be mindful of rate limits and usage
quotas that may apply to specific services.
{"docsConfig":{"$schema":"https://mintlify.com/docs.json","theme":"mint","name":"Invicta AI","colors":{"primary":"#0D9373","light":"#07C983","dark":"#0D9373"},"favicon":"/icon.svg","logo":{"light":"https://mintcdn.com/invictaai/rLU6J79e0_1ZhRdX/logo/light.svg?fit=max\u0026auto=format\u0026n=rLU6J79e0_1ZhRdX\u0026q=85\u0026s=7127842cec098877498aa67a3326b062","dark":"https://mintcdn.com/invictaai/rLU6J79e0_1ZhRdX/logo/dark.svg?fit=max\u0026auto=format\u0026n=rLU6J79e0_1ZhRdX\u0026q=85\u0026s=0839a3b56bf6b0f31965a76ad57dda04"},"navbar":{"links":[{"label":"Support","href":"mailto:support@invictai.io"}],"primary":{"type":"button","label":"Platform","href":"https://app.invictai.io/"}},"footer":{"socials":{"x":"https://x.com/InvictaAI","github":"https://github.com/Invicta-AI","linkedin":"https://www.linkedin.com/company/86028673"}},"navigation":{"global":{"anchors":[{"anchor":"Tutorials","href":"https://invictai.io/use-cases","icon":"youtube"},{"anchor":"Blog","href":"https://invictai.io/blog","icon":"newspaper"},{"anchor":"Community","href":"https://discord.com/invite/sDSeRXNBSd","icon":"discord"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"invictaai","hasAudioNarration":false,"actualSubdomain":"invictaai","customDomains":["info.invictai.io"],"gitSource":{"type":"github","owner":"axultan","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"entitlements":{},"buildId":"6ae921430330fbfc0d0ef5f5:success","clientVersion":"0.0.4205","searchFilterCounts":[{"language":"default","tag":"Integrations","count":54},{"language":"default","tag":"Documentation","count":40},{"language":"default","tag":"AI Agents","count":10},{"language":"default","tag":"Components","count":9},{"language":"default","tag":"Tool Docs","count":9},{"language":"default","tag":"Help","count":5},{"language":"default","tag":"FAQs","count":4},{"language":"default","tag":"How to Guides","count":4},{"language":"default","tag":"Platform Capabilities","count":4},{"language":"default","tag":"Trigger Docs","count":4}]},"assistantConfig":{},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"tools/data-\u0026-web-tools","slug":"/tools/data-\u0026-web-tools","pageMetadata":{"title":"Web Browsing \u0026 Scraping Tools","sidebarTitle":"Web Browsing \u0026 Scraping","icon":"globe","description":null,"href":"/tools/data-\u0026-web-tools"},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={code:`code`,p:`p`,strong:`strong`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,..._provideComponents(),...props.components},{Heading,Table,Tip}=_components;if(!Heading)_missingMdxReference(`Heading`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tip)_missingMdxReference(`Tip`,true);return _jsxs(_Fragment,{children:[_jsx(_components.p,{children:`Web Browsing and Scraping tools enable AI Agents to gather information from various online sources, including search engines, social media platforms, websites, and multimedia content. These tools provide capabilities for both broad web searches and targeted data extraction.`}),`\n`,_jsx(Heading,{level:`2`,id:`available-tools`,children:`Available Tools`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Tool Name`}),_jsx(_components.th,{children:`Platform/Category`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Apollo Company Enrichment`}),_jsx(_components.td,{children:`Business Data`}),_jsx(_components.td,{children:`Search for company details in Apollo database`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`DuckDuckGo Search`}),_jsx(_components.td,{children:`Search Engine`}),_jsx(_components.td,{children:`Basic web search using DuckDuckGo`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Tavily Search`}),_jsx(_components.td,{children:`Search Engine`}),_jsx(_components.td,{children:`Advanced semantic search with AI-optimized results`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`LinkedIn Profile Scraper`}),_jsx(_components.td,{children:`Social Media`}),_jsx(_components.td,{children:`Extracts detailed information from LinkedIn profiles`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Twitter Profile Scraper`}),_jsx(_components.td,{children:`Social Media`}),_jsx(_components.td,{children:`Collects data from Twitter/X user profiles`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Twitter Search Scraper`}),_jsx(_components.td,{children:`Social Media`}),_jsx(_components.td,{children:`Gathers tweets based on search terms`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Website Scraper`}),_jsx(_components.td,{children:`Web`}),_jsx(_components.td,{children:`Extracts text content from web pages`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Video Transcript Extractor`}),_jsx(_components.td,{children:`Multimedia`}),_jsx(_components.td,{children:`Retrieves transcripts from video/audio content`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Wikipedia Search`}),_jsx(_components.td,{children:`Knowledge Base`}),_jsx(_components.td,{children:`Searches and retrieves Wikipedia article content`})]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`tool-details`,children:`Tool Details`}),`\n`,_jsx(Heading,{level:`3`,id:`apollo-company-enrichment`,children:`Apollo Company Enrichment`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nEnriches company data by retrieving detailed information from Apollo’s comprehensive business database.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`apollo_organization_enrichment`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`domain`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Company domain to search for`})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`duckduckgo-search`,children:`DuckDuckGo Search`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nA basic web search tool that queries DuckDuckGo search engine to find relevant web pages and information.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`duckduckgo-search`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`maxResults`}),_jsx(_components.td,{children:`Optional (default: 5)`}),_jsx(_components.td,{children:`number`}),_jsx(_components.td,{children:`Maximum number of search results to return`})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`tavily-search`,children:`Tavily Search`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nAn advanced search tool that uses AI to provide more relevant and contextual search results.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`tavily-search`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`maxResults`}),_jsx(_components.td,{children:`Optional (default: 5)`}),_jsx(_components.td,{children:`number`}),_jsx(_components.td,{children:`Maximum number of search results to return`})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`linkedin-profile-scraper`,children:`LinkedIn Profile Scraper`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nExtracts comprehensive information from LinkedIn profiles including work experience, education, skills, and other public profile data.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`linkedin_scrape_profiles_by_urls`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`profileUrls`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string[]`}),_jsx(_components.td,{children:`Array of LinkedIn profile URLs to scrape`})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`twitter-profile-scraper`,children:`Twitter Profile Scraper`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nCollects data from Twitter/X user profiles including tweets, profile information, and public metrics.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`twitter_scrape_by_handles`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`twitterHandles`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string[]`}),_jsx(_components.td,{children:`Array of Twitter handles to scrape`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`start`}),_jsx(_components.td,{children:`Optional (default: ”)`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Start date in YYYY-MM-DD format`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`end`}),_jsx(_components.td,{children:`Optional (default: ”)`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`End date in YYYY-MM-DD format`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`sort`}),_jsx(_components.td,{children:`Optional (default: ‘Top’)`}),_jsx(_components.td,{children:`enum: [‘Latest’, ‘Top’]`}),_jsx(_components.td,{children:`Sort order of tweets`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`maxItems`}),_jsx(_components.td,{children:`Optional (default: ‘10’)`}),_jsx(_components.td,{children:`enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]`}),_jsx(_components.td,{children:`Maximum number of items to return`})]})]})]}),`\n`,_jsx(Heading,{level:`3`,id:`twitter-search-scraper`,children:`Twitter Search Scraper`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nSearches and extracts tweets based on specific search terms or keywords.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`twitter_scrape_by_search_terms`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`searchTerms`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string[]`}),_jsx(_components.td,{children:`Array of search terms`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`start`}),_jsx(_components.td,{children:`Optional (default: ”)`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Start date in YYYY-MM-DD format`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`end`}),_jsx(_components.td,{children:`Optional (default: ”)`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`End date in YYYY-MM-DD format`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`sort`}),_jsx(_components.td,{children:`Optional (default: ‘Top’)`}),_jsx(_components.td,{children:`enum: [‘Latest’, ‘Top’]`}),_jsx(_components.td,{children:`Sort order of tweets`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`maxItems`}),_jsx(_components.td,{children:`Optional (default: ‘10’)`}),_jsx(_components.td,{children:`enum: [‘5’, ‘10’, ‘25’, ‘50’, ‘100’]`}),_jsx(_components.td,{children:`Maximum number of items to return`})]})]})]}),`\n`,_jsx(Heading,{level:`3`,id:`website-scraper`,children:`Website Scraper`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nExtracts visible text content from any webpage URL, making it ideal for gathering information from articles, blog posts, and other web content.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`scrape_web_text`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`url`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`URL of the webpage to scrape`})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`video-transcript-extractor`,children:`Video Transcript Extractor`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nRetrieves transcripts from various online video and audio content across different platforms.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`video_transcript`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`video_url`}),_jsx(_components.td,{children:`Required`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`URL of the video/audio content`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`language`}),_jsx(_components.td,{children:`Optional`}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`Language code for the transcript (e.g., ‘en’, ‘ru’)`})]})]})]}),`\n`,_jsx(Heading,{level:`3`,id:`wikipedia-search`,children:`Wikipedia Search`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Description:`}),`\nSearches Wikipedia articles and retrieves relevant content and information.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`System Tool ID:`}),` `,_jsx(_components.code,{children:`wikipedia-query-run`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Arguments:`})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Name`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`topKResults`}),_jsx(_components.td,{children:`Optional (default: 3)`}),_jsx(_components.td,{children:`number`}),_jsx(_components.td,{children:`Number of top results to return`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`maxDocContentLength`}),_jsx(_components.td,{children:`Optional (default: 4000)`}),_jsx(_components.td,{children:`number`}),_jsx(_components.td,{children:`Maximum content length per document`})]})]})]}),`\n`,_jsx(Tip,{children:_jsx(_components.p,{children:`When using search and scraping tools, be mindful of rate limits and usage\nquotas that may apply to specific services.`})})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Web Browsing \u0026 Scraping Tools","sidebarTitle":"Web Browsing \u0026 Scraping","icon":"globe","description":null,"href":"/tools/data-\u0026-web-tools"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Available Tools","slug":"available-tools","depth":2,"children":[]},{"title":"Tool Details","slug":"tool-details","depth":2,"children":[{"title":"Apollo Company Enrichment","slug":"apollo-company-enrichment","depth":3,"children":[]},{"title":"DuckDuckGo Search","slug":"duckduckgo-search","depth":3,"children":[]},{"title":"Tavily Search","slug":"tavily-search","depth":3,"children":[]},{"title":"LinkedIn Profile Scraper","slug":"linkedin-profile-scraper","depth":3,"children":[]},{"title":"Twitter Profile Scraper","slug":"twitter-profile-scraper","depth":3,"children":[]},{"title":"Twitter Search Scraper","slug":"twitter-search-scraper","depth":3,"children":[]},{"title":"Website Scraper","slug":"website-scraper","depth":3,"children":[]},{"title":"Video Transcript Extractor","slug":"video-transcript-extractor","depth":3,"children":[]},{"title":"Wikipedia Search","slug":"wikipedia-search","depth":3,"children":[]}]}],"codeExamples":{}},"jsLinks":[],"subdomain":"invictaai","siteName":"Invicta AI","title":"Web Browsing \u0026 Scraping Tools - Invicta AI","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-01-07T02:48:30.798Z","originalFileLocation":"tools/data-\u0026-web-tools.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"application-name\" content=\"Invicta AI\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Web Browsing \u0026amp; Scraping Tools - Invicta AI\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Invicta AI\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://info.invictai.io/tools/data-\u0026amp;-web-tools\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://invictaai.mintlify.site/_mintlify/api/og/v2?division=Tool+Docs\u0026amp;title=Web+Browsing+%26+Scraping+Tools\u0026amp;theme=1a4a42097cea9166f283ae88\u0026amp;signature=9dca58dbe1839d0abfad61d8b7da99547dc3fb3a348e87b4bbf6c7cffcced71c\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Web Browsing \u0026amp; Scraping Tools - Invicta AI\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://invictaai.mintlify.site/_mintlify/api/og/v2?division=Tool+Docs\u0026amp;title=Web+Browsing+%26+Scraping+Tools\u0026amp;theme=1a4a42097cea9166f283ae88\u0026amp;signature=9dca58dbe1839d0abfad61d8b7da99547dc3fb3a348e87b4bbf6c7cffcced71c\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/invictaai/VTo60Gxpu_txrN-3/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Invicta AI\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#0D9373\">\u003clink data-mint-head rel=\"canonical\" href=\"https://info.invictai.io/tools/data-\u0026amp;-web-tools\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/tools/data-%26-web-tools.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://info.invictai.io/#organization\",\"name\":\"Invicta AI\",\"url\":\"https://info.invictai.io\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/invictaai/rLU6J79e0_1ZhRdX/logo/light.svg?fit=max\u0026auto=format\u0026n=rLU6J79e0_1ZhRdX\u0026q=85\u0026s=7127842cec098877498aa67a3326b062\"}},{\"@type\":\"WebSite\",\"@id\":\"https://info.invictai.io#website\",\"name\":\"Invicta AI\",\"url\":\"https://info.invictai.io\",\"publisher\":{\"@id\":\"https://info.invictai.io/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://info.invictai.io/tools/data-\u0026-web-tools#webpage\",\"url\":\"https://info.invictai.io/tools/data-\u0026-web-tools\",\"name\":\"Web Browsing \u0026 Scraping Tools\",\"dateModified\":\"2026-01-07T02:48:30.798Z\",\"isPartOf\":{\"@id\":\"https://info.invictai.io#website\"},\"breadcrumb\":{\"@id\":\"https://info.invictai.io/tools/data-\u0026-web-tools#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://info.invictai.io/tools/data-\u0026-web-tools#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Tool Docs\",\"item\":\"https://info.invictai.io/tools/email-\u0026-communication-tools\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Web Browsing \u0026 Scraping Tools\",\"item\":\"https://info.invictai.io/tools/data-\u0026-web-tools\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://info.invictai.io/tools/data-\u0026-web-tools#article\",\"headline\":\"Web Browsing \u0026 Scraping Tools\",\"name\":\"Web Browsing \u0026 Scraping Tools\",\"url\":\"https://info.invictai.io/tools/data-\u0026-web-tools\",\"mainEntityOfPage\":{\"@id\":\"https://info.invictai.io/tools/data-\u0026-web-tools#webpage\"},\"dateModified\":\"2026-01-07T02:48:30.798Z\",\"publisher\":{\"@id\":\"https://info.invictai.io/#organization\"},\"isPartOf\":{\"@id\":\"https://info.invictai.io#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Invicta AI\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Invicta AI\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/tools/data-%26-web-tools.md","siteName":"Invicta AI","nav":{"global":{"anchors":[{"anchor":"Tutorials","href":"https://invictai.io/use-cases","icon":"youtube"},{"anchor":"Blog","href":"https://invictai.io/blog","icon":"newspaper"},{"anchor":"Community","href":"https://discord.com/invite/sDSeRXNBSd","icon":"discord"}]},"tabs":[{"tab":"Documentation","groups":[{"group":"Basics","pages":[{"title":"Welcome to Invicta AI 👋","sidebarTitle":"Introduction","href":"/introduction"},{"icon":"microchip","title":null,"href":"/basics/credits"},{"title":"Available LLMs","icon":"brain","href":"/basics/available-llms"}]},{"group":"AI Agents","pages":[{"icon":"robot","title":null,"href":"/ai-agents/overview"},{"group":"Components","icon":"gear","pages":[{"group":"Knowledge Base","icon":"database","pages":[{"title":"Overview","href":"/ai-agents/knowledge-base"},{"title":"Long-Term Memory","href":"/ai-agents/memories"}]},{"group":"System Message","icon":"user-gear","pages":[{"title":"System Message","sidebarTitle":"Overview","href":"/ai-agents/system-message"},{"title":"Prompt Best Practices","href":"/ai-agents/prompt-best-practices"}]},{"icon":"wrench","title":null,"href":"/ai-agents/tools"},{"title":"AI Teammates","icon":"users","href":"/ai-agents/ai-teammates"},{"icon":"bell","title":null,"href":"/ai-agents/triggers"},{"icon":"code","title":null,"href":"/ai-agents/variables"},{"title":"LLM Settings","icon":"sliders","href":"/ai-agents/llm-settings"}]}]},{"group":"Platform Capabilities","pages":[{"icon":"table","title":null,"href":"/platform-capabilities/batch-mode"},{"icon":"clone","title":null,"href":"/platform-capabilities/cloning"},{"icon":"comment","title":null,"href":"/platform-capabilities/widget-and-embeds"},{"icon":"building","title":null,"href":"/platform-capabilities/organisations"}]},{"group":"Tool Docs","pages":[{"title":"Email \u0026 Communication Tools","sidebarTitle":"Email \u0026 Communication","icon":"envelope-open-text","description":null,"href":"/tools/email-\u0026-communication-tools"},{"title":"Web Browsing \u0026 Scraping Tools","sidebarTitle":"Web Browsing \u0026 Scraping","icon":"globe","href":"/tools/data-\u0026-web-tools"},{"title":"Calendar Tools","sidebarTitle":"Calendar","icon":"calendar","description":null,"href":"/tools/calendar-tools"},{"title":"Database Tools","icon":"database","href":"/tools/database-tools"},{"title":"Spreadsheet Tools","sidebarTitle":"Spreadsheet","icon":"table","href":"/tools/spreadsheet-tools"},{"title":"CRM Tools","sidebarTitle":"CRM","icon":"hubspot","href":"/tools/crm-tools"},{"title":"Image Generation Tools","sidebarTitle":"Image Generation","icon":"image","href":"/tools/image-generation"},{"title":"Payment Processing Tools","sidebarTitle":"Payment Processing","icon":"credit-card","href":"/tools/payment-processing"},{"title":"Data Processing \u0026 Miscellaneous Tools","sidebarTitle":"Data Processing","icon":"gear","href":"/tools/miscellaneous-tools"}]},{"group":"Trigger Docs","pages":[{"sidebarTitle":"Basic","icon":"clock","title":null,"href":"/triggers/basic-triggers"},{"sidebarTitle":"Email","icon":"envelope-open-text","title":null,"href":"/triggers/email-triggers"},{"sidebarTitle":"HubSpot","icon":"hubspot","title":null,"href":"/triggers/hubspot-triggers"},{"sidebarTitle":"Intercom","icon":"intercom","title":null,"href":"/triggers/intercom-triggers"}]},{"group":"How to Guides","pages":[{"sidebarTitle":"Integration Setup","title":"Setting Up Integrations","icon":"plug-circle-plus","href":"/how-to/integration-setup"},{"group":"Finding IDs","icon":"key","pages":[{"title":"Slack Channel ID","icon":"slack","href":"/how-to/finding-ids/slack-channelid"},{"title":"Intercom Admin ID","icon":"intercom","href":"/how-to/finding-ids/intercom-adminid"},{"title":"HubSpot Sender Actor ID","icon":"hubspot","href":"/how-to/finding-ids/hubspot-senderactorid"}]}]},{"group":"API Documentation","pages":[{"icon":"code","title":null,"href":"/invicta-ai-api/overview"}]},{"group":"Help","pages":[{"title":"Contact us","icon":"envelope","href":"/help/how-can-i-contact-the-invicta-ai-team"},{"group":"FAQs","icon":"life-ring","pages":[{"title":"How Secure is My Data?","href":"/help/how-secure-is-my-data"},{"title":"Can I try Invicta AI for free?","href":"/help/can-I-try-Invicta-AI-for-free"},{"title":"Does Invicta AI use my conversations or data to train AI?","href":"/help/does-invicta-ai-use-my-conversations-or-data-to-train-ai"},{"title":"Do Credits roll over?","href":"/help/do-credits-roll-over"}]}]}]},{"tab":"Changelog","pages":[{"title":"Changelog","href":"/changelog/overview"}]},{"tab":"Integrations","pages":[{"title":"Integrations","href":"/integrations/overview"}]}]},"basePath":"","title":"Web Browsing \u0026 Scraping Tools - Invicta AI","deploymentId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","buildId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","dynamics":[],"chunks":[]}