Posts Tagged Under: ec2

spark-logo-trademark

Spinning up a Spark Cluster on AWS EC2: Step-by-Step

Previously I walked through running Spark locally for development but one of the major challenges of learning to use distributed systems is understanding how the various components are installed and interact with each other in a production like environment.

You can use Vagrant or virtual machine images to run a cluster

Read More

scrapy_and_aws

Deploying scrapy on EC2

Welcome to part 3 of my guide to using AWS for scraping.

If you haven’t already make sure you check the first two parts, here and here. We’re going to continue using the same EC2 instance you created in part two.

Some assumptions before we begin

I’m going to assume a

Read More

scrapylogo

Installing Scrapy on Amazon Linux

I’ve recently moved all my AWS instances over to Amazon Linux and wanted to write a short update to installing Scrapy as the process is slightly different from Ubuntu.

Why Amazon Linux?

Amazon Linux is a distribution that evolved from Red Hat Enterprise Linux (RHEL) and CentOS. It is available for use

Read More

scrapy_and_aws

Installing scrapy and scrapyd on AWS EC2

See the updated version for installing scrapy 1.0 and above here.

This post will cover the basics of getting started with Amazon AWS, creating an account, creating an EC2 instance, installing scrapy and scrapyd and finally making sure you do it all for free!

Getting Started

Keeping It Free!

Before you go any

Read More