Scraping Web Links
This guide explains how to configure an AI Employee to crawl and ingest content from specific web pages. Web scraping lets you use publicly available web content as a data source for your workflows, keeping your AI Employee's knowledge base current with information from websites, help centers, and documentation portals.
Prerequisites
- The specific URLs you wish to scrape and ingest
- Access to the Ema platform dashboard
Part 1: Creating the AI Employee
Step 1: Create a New AI Employee
Navigate to Ema's AI Employees page via the main dashboard. Click the Create New Employee button.
Step 2: Configure the AI Employee Profile
Select an AI Employee template (for example, Conversation AI Employee). Fill in the required fields: name, description, and type. Proceed to the configuration area once the basic profile is set up.
Part 2: Workflow Configuration and Web Scraping
Step 3: Access Shared Configurations
On the AI Employee configuration page, navigate to the Workflow Builder. Locate and click the Shared Configurations button in the top-left corner of the interface.
Step 4: Add a Web Scraping Data Source
In the Shared Configuration modal, click the Add button and select the option to Scrape Web Links.
Step 5: Configure Ingestion and Import Settings
Paste the specific web links you wish to crawl into the Base URL field. A URL is required to proceed.
Before starting the scrape, configure your Import Settings:
| Setting | Description | Default |
|---|---|---|
| Maximum pages | The total number of web pages to import from this URL. | 200 (minimum: 1) |
| Website navigation depth | How many folder levels deep the crawler should explore. A depth of 2 includes paths like /policies/hr/. | 2 (minimum: 1) |
If no custom values are provided, the system uses the defaults listed above. Ema clicks on all links on the page and scrapes downstream pages up to the configured depth. Ema also downloads any documents such as PDFs encountered during the scraping.
Step 6: Ingest Web Links
After configuring import settings, the system begins processing the URLs to extract and ingest the content.
Part 3: Verification and Status
Step 7: Verify Completion
Wait for the processing status to update. You will see a confirmation status indicating Completed once the scraping process finishes. Web link scraping can take a few minutes depending on the number of pages and depth configured.
Part 4: Synchronization Settings
Step 8: Configure Sync Options
Once the web links have been successfully scraped, you can manage their synchronization frequency to keep the AI Employee's knowledge up to date.
- Manual sync -- Click the three dots (...) next to the source in the root folder and select Sync Now to refresh data manually.
- Automatic sync -- Click the Auto Sync dropdown and select a sync interval: 1, 7, 15, or 30 days.
For frequently updated websites such as help centers or documentation portals, set automatic sync to 1 or 7 days. For relatively static content, 15 or 30 days is sufficient.