> ## Documentation Index
> Fetch the complete documentation index at: https://docs.markifact.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape URL

> Extract content from any webpage for analysis, monitoring, or data collection.

**Scrape URL** extracts content from webpages, turning unstructured web data into usable information for your workflows. Essential for competitor monitoring, content analysis, and automated data collection.

***

## When to Use It

Use this node to:

* Monitor competitor pricing and product information
* Extract content for AI analysis and insights
* Collect data from news sites or blogs
* Scrape product details from e-commerce sites
* Monitor website changes and updates

***

## Inputs

| Field                 | Type   | Required | Description                                                    |
| --------------------- | ------ | -------- | -------------------------------------------------------------- |
| **URL**               | Text   | Yes      | Webpage URL you want to scrape                                 |
| **Extract Mode**      | Select | Yes      | Choose what content to extract from the page                   |
| **CSS Selectors**     | Mapper | No\*     | Specify elements to extract (\*Required for custom mode)       |
| **Enable JavaScript** | Switch | No       | Render JavaScript-generated content (slower but more complete) |

***

## How It Works

This node visits the specified webpage and extracts content based on your chosen extraction mode. It can handle both static HTML content and JavaScript-rendered pages.

### Extract Modes

**Entire Page Content:**

* Gets all text and HTML from the webpage
* Perfect for AI analysis of full page content
* Includes headers, navigation, and footer content
* Most comprehensive option

**Text Only (No HTML):**

* Extracts only the readable text content
* Removes all HTML tags and styling
* Clean text perfect for content analysis
* Faster processing than full page extraction

**Select Specific Elements:**

* Target specific parts of the webpage using CSS selectors
* Extract only the data you need (prices, titles, descriptions)
* More efficient for structured data collection
* Requires basic knowledge of CSS selectors

***

## CSS Selectors Guide

When using "Select Specific Elements", you'll map friendly names to CSS selectors:

| Element Type            | Example Selector        | Use Case                |
| ----------------------- | ----------------------- | ----------------------- |
| **Product Price**       | `.price`, `#price`      | E-commerce monitoring   |
| **Page Title**          | `h1`, `.title`          | Content analysis        |
| **Article Text**        | `.content`, `article p` | Blog/news scraping      |
| **Product Description** | `.description`          | Product data collection |
| **Reviews**             | `.review-text`          | Sentiment analysis      |

### Selector Examples

```
Friendly Name: product_price
CSS Selector: .price-current

Friendly Name: product_title  
CSS Selector: h1.product-name

Friendly Name: rating
CSS Selector: .rating-stars
```

***

## JavaScript Rendering

**When to Enable:**

* Content is loaded dynamically with JavaScript
* Page shows "Loading..." initially
* Important data appears after page load
* Single-page applications (SPAs)

**When to Keep Disabled:**

* Static HTML pages
* Faster scraping is needed
* Content is visible in page source

***

## Output

The node returns extracted content based on your selection:

**Full Page/Text Mode:**

* **Content** - All extracted text or HTML
* **Page Title** - Webpage title
* **URL** - The scraped page address

**Custom Elements Mode:**

* **\[Your Field Names]** - Data mapped to your specified selectors
* **URL** - The scraped page address

***

## Tips

**Choosing Selectors:**

* Use browser developer tools (F12) to find CSS selectors
* Test selectors on the webpage before using them
* Use specific selectors to avoid getting unwanted content
* Combine multiple selectors for comprehensive data extraction

**Performance:**

* Disable JavaScript rendering when possible for speed
* Use specific element extraction instead of full page when you only need certain data
* Consider rate limiting when scraping multiple pages

**Reliability:**

* Websites may change their structure, breaking your selectors
* Test your selectors periodically
* Have fallback selectors for critical data points

***

## FAQ

<Accordion title="How do I find the right CSS selector for an element?">
  Right-click the element on the webpage, select "Inspect" or "Inspect Element", then right-click the highlighted HTML and choose "Copy selector" to get the CSS selector path.
</Accordion>

<Accordion title="What if the content I need loads after the page loads?">
  Enable "JavaScript Rendering" to wait for dynamic content to load. This is essential for modern websites that use JavaScript to display content.
</Accordion>

<Accordion title="Can I scrape data from multiple pages at once?">
  This node handles one URL at a time. To scrape multiple pages, use it inside a loop with a list of URLs or use multiple scrape nodes in your workflow.
</Accordion>

<Accordion title="What if my selectors stop working?">
  Websites frequently change their HTML structure. Monitor your workflows and update selectors when they break. Consider using more general selectors that are less likely to change.
</Accordion>

<Accordion title="Is web scraping legal?">
  Web scraping is generally legal for publicly available data, but respect robots.txt files and website terms of service. Don't overload servers with too many requests.
</Accordion>

<Accordion title="How do I scrape competitor pricing data?">
  Use "Select Specific Elements" mode with CSS selectors targeting price elements. Create friendly names like "current\_price" and "sale\_price" for easy data analysis.
</Accordion>

<Accordion title="Why can't I scrape some websites or why do I get error codes?">
  Some websites use advanced bot protection to prevent automated scraping. Sites like LinkedIn, Facebook, and many e-commerce platforms detect and block scraping attempts with measures like CAPTCHAs, rate limiting, IP blocking, and sophisticated bot detection. You'll typically see 403 (Forbidden), 429 (Too Many Requests), or custom error pages. These sites require human interaction or special authentication that automated scraping can't bypass.
</Accordion>