Welcome to picrawler’s documentation!

PiCrawler is a distributed web crawler using PiCloud.

Using PiCrawler, you can easily implement a distributed web crawler within a few lines of code.

>>> from picrawler import PiCloudConnection
>>>
>>> with PiCloudConnection() as conn:
...     response = conn.send(['http://en.wikipedia.org/wiki/Star_Wars',
...                           'http://en.wikipedia.org/wiki/Darth_Vader'])
...     print 'status code:', response[0].status_code
...     print 'content:', response[0].content[:15]
status code: 200
content: <!DOCTYPE html>

Contents:

Indices and tables

Project Versions

Table Of Contents

Next topic

Installation

This Page