The Ubiquity and Danger of Web Scraping

Web scraping is a software method used to extract information from websites. It often includes transforming unstructured website data into a database for analysis, or repurposing stolen content for the scraper’s own online operations. Not only does web scraping pose a critical challenge to company branding, it can also threaten sales and conversions, lower SEO rankings or undermine the integrity of content that took considerable time and resources to produce.

Through analysis of top web scraping platforms and services, Distil Networks’ 2016 Economics of Web Scraping Report uncovers the ubiquity and danger of this practice. The following findings outline how the democratization of web scraping lets perpetrators effortlessly steal sensitive information on the web.

The Dangers of Web Scraping

Click through for more on the dangers of web scraping and how it can damage your organization, as identified by Distil Networks.

Stealing Original Content

Thirty-eight percent of web scraping customers employ the practice to scrape original content.

Content scraping is stealing original content from a legitimate website and posting it on another website without the knowledge or permission of the original content owner. Content scraping can come in the form of web mash-ups — using information from more than one source to create a new display of information, also known as web data integration. For example, a startup can build an aggregator jobs or classifieds site made up of data from multiple websites.

Research and Data Collection 

Twenty-six percent use web scraping for research.

Companies hire web scrapers to gather research on listening services that monitor consumer opinions about products and companies. Companies also use web scraping bots for mass data collection for various projects. For example, users can get marketing intelligence by using bots to identify key market developments from various sources on the web.

Contact Scraping 

Nineteen percent of companies use web scraping for contact scraping.

These companies wish to gain access to customers’ emails or other contact information for marketing purposes or for background reports. Bots help generate leads from business directories and social media sites like Twitter and LinkedIn.

Industry Leaders

The web scraping industry leaders are Screen-Scraper, Mozenda, Diffbot, and Scrapinghub. Additionally, a number of websites like Freelancer.com, Upwork and Guru.com host ads providing freelance and company web scraping services, as well as ads seeking web scraping services. The latest addition to the web scraping economy is Spinner Bot, a web scraping software that allows users to push requests across multiple proxies.

Top Victims

The top web scraping victims by industry in 2015 were real estate, digital publishing, e-commerce, directories and classifieds, and airlines and travel. Many of these industries are being targeted by an influx of startups that are scraping information from industry leaders in order to compete.

Real estate sites are the no. 1 web scraping victims. Real estate had the highest percentage of bad bots at 32 percent. From 2014 to 2015, the real estate industry saw a 300 percent increase in bad bot activity.

Speed

Some web pages can be scraped in less than an hour, particularly those with small amounts of content and little to no barriers to web scraping, like web application firewalls (WAFs), bot detection and mitigation, and CAPTCHAs.