Diving Into the Deluge of Data :: Lab 8 :: Twitter Word Clouds

Lab 8: Twitter Word Clouds

This lab uses the Twitter API as a source for sampling tweets real-time as well as grabbing tweets from a particular user. The focus is on constructing working software from existing libraries and doctumentation while also gaining exposure to JSON and text processing.

Here is a word cloud created from the text of the last 100 tweets by Williams College.

Step 0: Lab Preparation

  • You should begin by reading a blog post about the twitter streaming API.
  • To use the Twitter API, you must have a twitter account and be credentialed. The credentials take the form of a Consumer (API) key, a Consumer (API) secret, an Access Token, and an Access Token Secret. To gain credentials, perform the following steps:
    • Create a twitter account if you do not alrady have one.
    • Head to https://apps.twitter.com/ and log in.
    • Click Create New App.
    • Fill out the Application Details form with reasonable data (you can use http://www.cs.williams.edu as your website), agree to the Developer Agreement terms, and click Create your Twitter application.
    • Click on the Keys and Access Tokens tab.
    • Click on Create my access token in the Token Actions area.
    • In the lab you will need the consumer key, the consumer secret, the access token, and the access token secret.
  • Also read the Getting Started guide on the Tweepy API.

Step 1: Source Code

Step 2: Twitter Data

Make sure that you've reviewed this blog post on the twitter API. This tutorial shows how to use Tweepy to sample data from Twitter's public stream and filter it based on keywords.

The heart of the code lies in the StdOutListener class, which subclasses StreamListener. The streaming portion of Tweepy is event-based: when new data is available, it calls the on_data method with the new data. This data is always valid JSON and it always corresponds to some sort of Twitter event, like a status updated (i.e. a tweet), blocking users, following users, deleting accounts, etc. If the method returns False, then the listener stops. If it returns True or None then it continues listening. 

  class StdOutListener(StreamListener):

    def on_data(self, data):
        print data
        return True

The file twitter.py contains a sample program that you can run on the command line, which samples twitter events from users who speak english: 

  $ python3 twitter.py

Notice that JSON objects are streaming by the screen. You can exit this program by using CTRL-C.

Step 2a: Dumping Tweets Realtime

Alter the StdOutListener class so that it listens for and stores N tweets. After it's reached its limit, it dumps those tweets to stdout as a single valid JSON list. Here are some implementation notes:

Test your program from the command line: 

  $ python twitter.py 5 > tweets.json

Examining the file should reveal a JSON list with a dictionary inside... 

  [{"source": ...

Step 2b: Dumping Historical Tweets by User

Instead of sampling from the public twitter feed realtime, we can grab historical tweets from arbitrary twitter users. Here we will define a function timeline that grabs the last N tweets from a twitter user and dumps them to standard out.

Here are some implementation details:

Step 3: Word Counts

The file wordcounts.py contains the class WordCounts, which represents a dictionary of words and their associated counts. This class also features filtering — you can pass in a list of functions that operate on strings. If a function returns True then the string should not be included in the count list. 

  class WordCounts:

      """A class representing word counts.  Words are only included if they
         are not filtered."""

      def __init__(self, filters=[]):
          """filters is a list of functions, each of which operates on a single
          string and returns True if and only if the word should be
          filtered (i.e., not stored)"""

      def addwords(self, words):
          """Update the word count for each word in words.  Only include the
          word if it passes all the filters (i.e., all the filters return False)"""

      def cloudstr(self):
          """return a single string where each word in the dictionary
             is repeated N times (with spaces) where N is its associted count.
             In other words if {"Brent":2,"Courtney":3} then the string would be
             "Brent Brent Courtney Courtney Courtney"""

      def __str__(self):
          return ""

  >>> from wordcounts import WordCounts
  >>> def filter_dog(word): return word == "dog"
  ...
  >>> def filter_starts_with_hi(word): return word.startswith("hi")
  ...
  >>> counts = WordCounts(filters=[filter_dog, filter_starts_with_hi])
  >>> counts.addwords("Brent Brent dog hiBrent hiCourtney Courtney Courtney Courtney".split())
  >>> counts.addwords(["dog", "hidog", "Brent"])
  >>> print(counts)
  {'Brent': 3, 'Courtney': 3}
  >>> counts.cloudstr()
  'Brent Brent Brent Courtney Courtney Courtney'

Here are a few implementation notes:

Step 4: Word Clouds

In this section you will write code to create word clouds from JSON data representing status updates. Given a filename containing JSON data and a filename for an output image, your program should

The command line usage should be 

  $ python cloud.py tweets.json tweets.png

Step 4a: Word Clouds

Here are some details about creating word clouds:

Step 4b: Filters

You should define at least 4 different kinds of filters:

These filters will be passed to your WordCounts instance.

Step 5: Make some Clouds

You should create word clouds for some Twitter username and for a sample of 1000 tweets from the real time twitter feed. Add and commit these PNG files to your repo.

Step 6: Submission