Scrapy in the cloud – Using scrapy on AWS


I deal with a lot of data for my day job as well as for many of my side projects and a lot of that data comes from web scraping. Please note I’m not condoning the use of scraping bots and I’m well aware of the legal grey area that scraping currently sits in.

What is web scraping?

Web scraping is a computer software technique for extracting information from websites. For more detail on all the permutations of scraping check out Wikipedia here.

Tools of the scraping trade

Although there are hundreds of scraping utilities and tools out there I find myself always coming back to scrapy. For this tutorial series I’ll be concentrating on scrapy and scrapyd only.

Scrapy is a python application framework for building spiders. It’s an open source project which uses XPATH/CSS3 selectors to access ‘items’ from a page you want to extract.

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Scrapyd is a service that allows you to schedule and run deployed ‘spiders’ and control them via an API and web front end.

Scrapyd is a service for running Scrapy spiders. It allows you to deploy your Scrapy projects and control their spiders using a HTTP JSON API.

Why the cloud?

Sometimes it’s great to be able to spin up an EC2 instance and run a scrape quickly from a new IP address. Amazon provides an amazing infrastructure for developers and if you create a new AWS account there’s not much you can’t do with the Free Usage Tier provided by Amazon with an EC2 instance and perhaps some S3 storage and an RDS database, but more on that later.

In the next post I’ll walk through how to install scrapy and scrapyd on an AWS EC2 instance and be ready to start scraping.

About This Author

Big Data and Python geek. Writes an eclectic mix of news from the world of Big Data and Telecommunications interspersed with articles on Python, Hadoop, E-Commerce and my continual attempts to learn Russian!

1 Comment

You can post comments in this post.

  • Thanks for writing this up! It’s exactly what I need to get started using AWS for my hobbyist project.

    Samba 3 years ago Reply

Post A Reply