Scrape URL
Extract content from any webpage for analysis, monitoring, or data collection.
Scrape URL extracts content from webpages, turning unstructured web data into usable information for your workflows. Essential for competitor monitoring, content analysis, and automated data collection.
When to Use It
Use this node to:
- Monitor competitor pricing and product information
- Extract content for AI analysis and insights
- Collect data from news sites or blogs
- Scrape product details from e-commerce sites
- Monitor website changes and updates
Inputs
| Field | Type | Required | Description |
|---|---|---|---|
| URL | Text | Yes | Webpage URL you want to scrape |
| Extract Mode | Select | Yes | Choose what content to extract from the page |
| CSS Selectors | Mapper | No* | Specify elements to extract (*Required for custom mode) |
| Enable JavaScript | Switch | No | Render JavaScript-generated content (slower but more complete) |
How It Works
This node visits the specified webpage and extracts content based on your chosen extraction mode. It can handle both static HTML content and JavaScript-rendered pages.
Extract Modes
Entire Page Content:
- Gets all text and HTML from the webpage
- Perfect for AI analysis of full page content
- Includes headers, navigation, and footer content
- Most comprehensive option
Text Only (No HTML):
- Extracts only the readable text content
- Removes all HTML tags and styling
- Clean text perfect for content analysis
- Faster processing than full page extraction
Select Specific Elements:
- Target specific parts of the webpage using CSS selectors
- Extract only the data you need (prices, titles, descriptions)
- More efficient for structured data collection
- Requires basic knowledge of CSS selectors
CSS Selectors Guide
When using “Select Specific Elements”, you’ll map friendly names to CSS selectors:
| Element Type | Example Selector | Use Case |
|---|---|---|
| Product Price | .price, #price | E-commerce monitoring |
| Page Title | h1, .title | Content analysis |
| Article Text | .content, article p | Blog/news scraping |
| Product Description | .description | Product data collection |
| Reviews | .review-text | Sentiment analysis |
Selector Examples
JavaScript Rendering
When to Enable:
- Content is loaded dynamically with JavaScript
- Page shows “Loading…” initially
- Important data appears after page load
- Single-page applications (SPAs)
When to Keep Disabled:
- Static HTML pages
- Faster scraping is needed
- Content is visible in page source
Output
The node returns extracted content based on your selection:
Full Page/Text Mode:
- Content - All extracted text or HTML
- Page Title - Webpage title
- URL - The scraped page address
Custom Elements Mode:
- [Your Field Names] - Data mapped to your specified selectors
- URL - The scraped page address
Tips
Choosing Selectors:
- Use browser developer tools (F12) to find CSS selectors
- Test selectors on the webpage before using them
- Use specific selectors to avoid getting unwanted content
- Combine multiple selectors for comprehensive data extraction
Performance:
- Disable JavaScript rendering when possible for speed
- Use specific element extraction instead of full page when you only need certain data
- Consider rate limiting when scraping multiple pages
Reliability:
- Websites may change their structure, breaking your selectors
- Test your selectors periodically
- Have fallback selectors for critical data points
FAQ
How do I find the right CSS selector for an element?
How do I find the right CSS selector for an element?
Right-click the element on the webpage, select “Inspect” or “Inspect Element”, then right-click the highlighted HTML and choose “Copy selector” to get the CSS selector path.
What if the content I need loads after the page loads?
What if the content I need loads after the page loads?
Enable “JavaScript Rendering” to wait for dynamic content to load. This is essential for modern websites that use JavaScript to display content.
Can I scrape data from multiple pages at once?
Can I scrape data from multiple pages at once?
This node handles one URL at a time. To scrape multiple pages, use it inside a loop with a list of URLs or use multiple scrape nodes in your workflow.
What if my selectors stop working?
What if my selectors stop working?
Websites frequently change their HTML structure. Monitor your workflows and update selectors when they break. Consider using more general selectors that are less likely to change.
Is web scraping legal?
Is web scraping legal?
Web scraping is generally legal for publicly available data, but respect robots.txt files and website terms of service. Don’t overload servers with too many requests.
How do I scrape competitor pricing data?
How do I scrape competitor pricing data?
Use “Select Specific Elements” mode with CSS selectors targeting price elements. Create friendly names like “current_price” and “sale_price” for easy data analysis.
Why can't I scrape some websites or why do I get error codes?
Why can't I scrape some websites or why do I get error codes?
Some websites use advanced bot protection to prevent automated scraping. Sites like LinkedIn, Facebook, and many e-commerce platforms detect and block scraping attempts with measures like CAPTCHAs, rate limiting, IP blocking, and sophisticated bot detection. You’ll typically see 403 (Forbidden), 429 (Too Many Requests), or custom error pages. These sites require human interaction or special authentication that automated scraping can’t bypass.