Getting Started Main Features Examples

How to Scrape Multiple URLs from Google

2018-11-27 17:30:42
1314 views

Abstract:In this article, we will tell you how to scrape multiple keyword data from Google using ScrapeStorm’s "Smart mode".

In this article, we will tell you how to scrape multiple keyword data from Google using ScrapeStorm’s “Smart mode“.

Introduction to the scraping tool

ScrapeStorm (www.scrapestorm.com) is a new generation of Web Scraping Tool based on artificial intelligence technology. It is the first scraper to support both Windows, Mac and Linux operating systems.

Introduction to the scraping object

Google LLC is an American multinational technology company that specializes in Internet-related services and products, which include online advertising technologies, search engine, cloud computing, software, and hardware.

Official Website: https://www.google.com/

Scraping fields

Title, title_link, Content, Key word

Function point directory

How to configure the extracted field

Preview of the scraped result

Export to Excel2007:

1. Download and install ScrapeStorm, then register and log in

(1) Open the ScrapeStorm official website, download and install the latest version.

(2) Click Register/Login to register a new account and then log in to ScrapeStorm.

Tips: You can use this web scraping software directly, you don’t need to register, but the tasks under the anonymous account will be lost when you switch to the registered user, so it is recommended that you use it after registration.

2. Create a task

(1) Copy the URL of Google

Click here to learn more about how to enter the URL correctly.

(2) Create a new smart mode task

To scrape multiple linked data, there are two ways to do this.

ⅰ: Copy multiple URLs directly to the creation box

Each URL is arranged separately, only one can be entered per line.

ⅱ: Read the URL form a file

Save the URL in a TXT file and import the file.

3. Configure the scraping rules

(1) Set the fields

Intelligent mode automatically recognizes the fields on the page. You can right-click the field to rename the name, add or delete fields, modify data, and so on.

Click here to learn how to how to configure the extracted field.

Add or remove fields as needed, and rename the fields. The results of the field settings are as follows:

4. Set up and start the scraping task

(1) Running and Anti-block settings

Once the rules are configured, we can start the acquisition task. Click “Start” and then jump out of the taskbar. The taskbar interface has a “More Settings” button, which we can click to set up to improve stability and success rate.

Click “Setting”, set waiting time based on web page open speed. The anti-block settings follow the system default settings. Then click “Save”.

(2) Start scraping data

Premium Plan and above users can use “Scheduled job and “Sync to Database”. If you want to download images, you can check “Download images while running”. Then click “Start”.

Click here to learn about scheduled job.

Click here to learn about sync to database.

Click here to learn about download images.

(3) Wait a moment, you will see the data being scraped.

5. Export and view data

(1) Click “Export” to download your data.

(2) Choose the format to export according to your needs.

ScrapeStorm provides a variety of export methods to export locally, such as excel, csv, html, txt or database. Professional Plan and above users can also post directly to wordpress.

Click here to learn more about how to view the extraction results and clear the extracted data.

Click here to learn more about how to export the result of extraction.