Twitter Spark Streaming using PySpark

* Testing Environment * ( On Virtual Machine )

• Ubuntu - 12.04 ( 32 bit )
• 8GB Ram
• i7 - QuadCore

Contents in folder:

• tweepy_kafka_producer.py ( Publishes tweets to kafka topic )
• twitter_stream.py ( Process the tweets )
• tweet_task.csv ( Sample output file )
• get_intent.py ( classifies into topics )
• generate_Map.py ( Visualizes geo points on map )
• draw_Pies.py ( draws pie charts )
• generate_cloud.py ( draws word cloud )
• my_map.html ( locations plotted map )
• figure_1.png ( Word cloud )
• figure_2.png ( pie plot for top 17 locations )
• tweet_task_intent.csv ( Contains tweet , intent , topic )

Pipeline Structure

Data is ingested in realtime using tweepy ( Twitter API ) and is sent to the producer which publishes it to some user defined topic in kafka.

We then create a consumer, which subscribes to the topic and eventually gets the data.

Technologies Used

This application uses a number of open source technologies to work properly:

• Apache Spark 1.5.0 - Big Data Processing Engine
• Tweepy - Twitter Application Programming Interface
• Kafka - Messaging System
• Sublime Text Editor - Text Editor
• Python 2.7 - Programming Language
• Sentiment140 - API

Python Packages Used

• matplotlib , wordcloud ,scipy ,geocoder
• re , pylab , webbrowser , pygmaps
• json , nltk , collections , urllib2
• gensim , monkeylearn , random, csv

Kafka Commands

Starts Zookeeper Server:

$ bin/zookeeper-server-start.sh config/zookeeper.properties

Starts Kafka Server:

$ nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

Create Topic ( t ):

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic socialcops

Start Producer:

$ python tweepy_kafka_producer.py

Start Consumer ( another terminal ):

From inside Spark directory -

$ bin/spark-submit --master local[3] --jars external/kafka-assembly/target/scala-2.10/spark-streaming-kafka-assembly-1.5.0.jar SocialCops_Task_SoftwareChallenge/twitter_stream.py localhost:2181 socialcops

Analyse Intent and Topic ( saves to tweet_task_intent.csv )

$ python get_intent.py tweet_task.csv

Draw Pie Charts:

$ python draw_pies.py tweet_task.csv

Plot on map

$ python generate_Map.py tweet_task.csv

Draw WordCloud

$ python generate_cloud.py tweet_task.csv

License

Prakhar Mishra