Download and Sign Up
Get a $5 Coupon For Free
Getting Started Main Features

【Smart Mode】【Flowchart Mode】How to use Webhook | Web Scraping Tool | ScrapeStorm

2023-01-07 11:21:54
8260 views

Abstract:This tutorial will show you how to use Webhook ScrapeStormFree Download

ScrapeStorm supports Webhook. By using this function, ScrapeStorm can publish the collected data to the user’s HTTP address in real time, and the user needs to develop the Webhook receiving end code.

This function can be set in Run settings.

After the Webhook function is enabled, the scraped data will be sent in JSON format. A notification will also be sent through webhook when the task has been finished.

The HTTP Header is "Content-Type: application/json; charset=utf-8".

The result of the md5 operation using Webhook Token and Timestamp can be verified against the “sign” signature of the JSON data to verify whether the data is legal.

Example of sending data:

{
    "task_id": 3920415,                 // Scraping task ID, find it by "View All Tasks"
    "task_name": "Scraping task name",  // Scraping task name
    "type": "data",                     // This is a webhook of scraped data
    "urls": [                           // The entries of the scraping task
        "http://www.*****.com/list",
        "http://www.*****.com/list2"
    ],
    "sign": "**********************",   // md5(webhook_token+timestamp)
    "timestamp": 1555326657,            // current timestamp
    "data_list": [                      // Scrape data list
        {
            "_id": "0000000000001",     // data ID
            "data": {
                "title": "Landscape",
                "url": "http://www.*****.com/scenery/"
            }                           // Scrape data field content
        }
    ]
}

Example of finish notification:

{
    "task_id": 3920415,                  // Scraping task ID, find it by "View All Tasks"
    "task_name": "Scraping task name",   // Scraping task name
    "type": "finish",                    // This is a webhook of finish notification
    "urls": [                            // The entries of the scraping task
        "http://www.*****.com/list",
        "http://www.*****.com/list2"
    ],
    "sign": "**********************",    // md5(webhook_token+timestamp)
    "timestamp": 1555326657              // current timestamp
}
Keyword extraction from web content python download file Generate URLs in batches Download videos in batches Download web page as word php crawler Data scraping with python python crawler Automatically organize data into excel Download images in batches
关闭