【Smart Mode】【Flowchart Mode】How to set IP Rotation & Delay | Web Scraping Tool | ScrapeStorm
Abstract：This tutorial will show you how to set IP Rotation & Delay. ScrapeStormFree Download
It contains IP Rotaion, automatic settings and manul settings. This part of the function is mainly used to avoid various website blocking problems that may be encountered.
In the editing task interface, click the “Start” button to open the running settings, and click “IP Rotation & Delay”.
1. IP Rotation
(1) IP Type
ⅰ. Bright Data Zone
Bright Data Zone is an external proxy, you need to buy ip on the official website.
How to Set up Bright Data Zone
ⅱ. Custom IP
If you need to use your own IP, please click “Set Up” and set it as required. (Note: the custom IP will be switched in sequence)
(2) Switch IP when the following condition is met:
IP switches based on time. For example, if you set the switching condition to “Interval: 3 minutes”, then the IP will be switched every 3 min and one IP will be consumed at the same time.
ⅱ. Text appears on the page
Switch according to the text. For example, if you set the switching condition to “Text appears on the page: error”, then when the corresponding text appears on the web page, the IP will be switched once and one IP will be consumed at the same time.
For general scraping tasks, just follow the default automatic settings.
If the webpage encountered is special, the automatic setting does not work. At this time, you can set the manual setting.
Some webpages open slowly and sometimes affect the scraping effect. You can set a delay to effectively improve the quality of scraping. The default delay of the system is 1 second, which can be modified according to your needs.
ⅱ. Detect CAPTCHA During the Scraping
Software generally detects captcha automatically. In special cases, you can manually set to detect captcha when encountering specific text.
ⅲ. Detect Login Prompt During the Scraping
Websites that need to log in to scrape data may be logged out during the running process, making it impossible to scrape data, or some websites will be prompted to log in after scraping a certain amount of data. Select this function. When you log out or need to log in, you will be prompted to log in.
ⅳ. Only Extract Visible Elements
Some websites mix invalid data with valid data. When scraping data, many invalid characters appear, and these invalid characters are hidden. In this case, we can choose this setting and only scrape visible elements.
Note: If the website has no setting to hide invalid characters, selecting this option will cause incomplete data or cannot be scraped.
ⅴ. Load Page Info one by one
Some websites need to scroll to a certain position before the content can be displayed, otherwise the data cannot be scraped, and this function can be selected at this time. However, it should be noted that when this function is checked, the scraping speed will be affected.
ⅵ. Switch Browser Regularly
Switching browser regularly can solve the blocking problem of some websites and achieve anti-blocking effect.
You can set the interval for switching browser versions. Set the interval to choose from 30 seconds to 10 minutes. The software will automatically switch various browser versions according to the interval.
ⅶ. Clear Cookies Regularly
Clearing cookies regularly can solve the blocking problem of some websites and achieve anti-blocking effect.
You can set the interval for clearing cookies. Set the interval to choose from 30 seconds to 10 minutes. The software will automatically clearing cookies according to the interval.