Analyzing Tweet Sentiment in Real Time

200 Credit Example

The deployments built in this tutorial to read, process and display sentiment data represent an example of what you can do with the 200 free credits available to the Quix 'Free Plan'.

The breakdown for this usage is as follows:

Resource % of usage

Workspace

31

Topics

9

CPU

9

RAM

30

Storage

21

Aim

You will build a real-time streaming application with a machine learning model. The application will analyse tweets related to the Dogecoin cryptocurrency and visualize them on a positivity scale using natural language processing.

This guide will take you through the steps to start working with the Twitter API using Quix.
Dogecoin is analysed here but when your’e finished with the guide why not change the twitter search terms and analyse sentiment of other tweets

By the end you will have built and deployed:

  • a project to continuously stream tweets using the Twitter API

  • a model to analyze sentiment and write scores for visualization

Project Architecture

architecture

The solution has three main elements.

  • Read Twitter data

  • Analyze the sentiment

  • Display sentiment on dashboard

The Twitter Data project receives Twitter comments and places them into a topic on Kafka. The Sentiment Analysis project subscribes to and receives messages from the Kafka topic and processes them by assigning a score according what was said in each message. It writes the sentiment score back to another Kafka topic. The dashboard then reads the sentiment score and displays on a public URL for anyone to see.

Prerequisites

We assume that all you have is a Quix account that you haven’t started using yet.

If you don’t have a Quix account yet, go here and create one.

Overview

This walkthrough covers the following steps:

  1. Create a Twitter developer account

  2. Create a Workspace

  3. Create Topics

  4. Create new projects for the twitter API and sentiment analysis models

  5. Deploy both projects

  6. Visualize the results

Create a Twitter developer account

Let’s create a free Twitter developer account:

  1. Go to https://developer.twitter.com/en/portal/petition/use-case.

  2. Complete the signup process to create a Twitter developer account.

It may take an hour or two for your application to be approved if this is the first time you’re using the Twitter Developer Portal. If so, feel free to skip the next step and continue from the [Create a Workspace] step. You can return here to complete the final step later, once your Twitter application has been approved.
  1. Once your application is approved, you should see a page of credentials including your Bearer Token. Copy the Bearer Token and keep it in a safe place.

If you lose your Bearer Token, you can regenerate one via your App’s “Keys and tokens” page

Create a Workspace

Ok, you have set up a Twitter developer account. Now you will use Quix to easily create a model that imports live Tweets and analyses them to determine if the overall sentiment is positive or nagative.

So get to the Quix Portal and log in.

A workspace is a collection of all the components for a single application. Find more info here.
Table 1. Creating a workspace

new workspace

First, click the + sign to create a workspace.

naming new workspace

Use a name such as “Twitter Tutorial Workspace” and click CREATE.

This may take half a minute since a discrete, fully featured and highly scalable data platform is being created for you.

Once it is ready, open the workspace.

Create Topics

Once the workspace has been created, we will create two topics.

Topics are streams of data or parameters built on Kafka. Find more info here.
Table 2. Creating topics

create topic

To create a topic, go to the Topics Page (see menu on the left) and then click on the + CREATE TOPIC button.

naming new topic

Name the first topic "Tweets".

Click CREATE.

This may take half a minute. A discrete, fully featured and highly scalable data platform is being created for you.

select topic details

copy topic id

Copy the Topic id. To do so:

  1. Click the >> icon on the left of our “Tweets” row

  2. Then on the right hand side click the "Copy to Clipboard" icon

naming new topic 2

Click + CREATE TOPIC again, name the second topic "SentimentScores", and click CREATE.

data persistance on

Turn on data persistence (this will store the data streamed into you topic).

You can do it whilst the topic is being created.

select topic details 2

copy topic id 2

Again, copy the Topic id. To do so:

  1. Click the >> icon on the left of our “SentimentScores” row

  2. Then on the right hand side click the "Copy to Clipboard" icon

Keep the Topic IDs safe for later. You will need them in the project code.

Create and Deploy Projects

Now let’s create the two projects we are going to need.

A project is a set of code which can be deployed as one Docker image. Find more info here.
Table 3. Creating the two projects

new project

Go to the Projects Page using the menu on the left.

Then, click the + CREATE PROJECT button to create a new project.

naming new project 1

naming new project 2

You don’t have to name your two projects the same as this. But something similar will be good.

  1. ReadTweetStream: This will be reading data from the Twitter API.

  2. CalculateSentimentScore: This will be analysing tweet sentiment.

The examples are in Python, so choose that for now. If you don’t know Python don’t worry, we’ve done all the hard work for you and you don’t need to run anything on your local machine. Get started with these samples then adapt them to your preferred language later.

ReadTweetStream Project

Now open up the ReadTweetStream project. This is where we’ll add the code to stream tweets into the “Tweets” topic.

Code

  1. Your main code will be placed in your project’s main.py, which is the file that the deployment will invoke.

  2. Copy the username and password from main.py and keep it safe somewhere, you will need it later.

  3. You don’t have to do the coding this time, it’s done for you. So head over to our GitHub pages for our Twitter Streaming Code and copy it. Paste this into the main.py file, completely replacing the boilerplate code.

  4. After you have pasted the code in from GitHub, replace the following placeholders with the correct values:

  • USER_NAME

  • PASSWORD

The project’s username and password that you have just saved (step 2 above).

  • THE_TOPIC_ID_TO_WRITE_TO

The Topic Id from the “Tweets” topic that you created earlier. This project will write to that topic.

  1. Save all the changes in main.py

Environment Variables

In your project, click VARIABLES, and add the following variables in the window that appears.

bearer_token

Insert the value you obtained earlier from Twitter

twitter_search

(#dogecoin OR #Doge OR DOGE OR dogecoin)-is:retweet lang:en

Click SAVE to complete this process.

Dependencies

You can define Python library dependencies by adding them to the requirements.txt file.

Open requirements.txt, add the following lines to the bottom of this file, then SAVE it:

requests
pandas
Make sure you add these lines; do not replace the line that begins TwitterDoge_V1

Deploy the Project

Now the ReadTweetStream project can be deployed.

Find more info on deployments here.

commit history

Look at the section on the right of the project page. This is your commit history.

Your code is automatically committed to a Git repository. Find more info here.

naming new tag

The top row is the most recent commit. Click on the three dots next to it and then on "Create tag".

Give the tag a meaningful name such as 'TwitterDoge_V1'.

Click CREATE.

deploy button 2

With our work properly saved, we can now deploy the project. Click the DEPLOY button.

general deployment settings

In the General tab, select the Version Tag you created earlier.

Also change the Deployment Type to Service.

DEPLOY

Hit it!

Just click DEPLOY and the code will be built, deployed and the service will start running.

deployment running

It’s alive!

You should see the status change to Running.

contextual options deployment

deployment logs

Checking your deployment

Hover your mouse over the row to find some contextual options.

You can view the logs and deployment logs, download the docker image and delete the deployment.

Open the logs to watch the tweets streaming in!

CalculateSentimentScore Project

The ReadTweetStream project is ready and live, so let’s focus now on the CalculateSentimentScore project. Go to the Projects page by clicking its icon in the left menu and then open the CalculateSentimentScore project.

Code

  1. The username and password should be exactly the same as those you copied before. If you’ve lost them since, make sure to copy them again now and keep them safe for later.

  2. Again, same as before, you don’t have to do the coding! Get the Sentiment Analysis Code from our GitHub pages and copy it. Paste this into the main.py file, completely replacing the boilerplate code.

  3. After you have pasted the code in from GitHub, replace the following placeholders with the correct values:

  • USER_NAME

  • PASSWORD

The project’s username and password that you just saved (step 1 above).

  • TWEETS_TOPIC_ID

The Topic Id for the “Tweets” topic.

  • SENTIMENT_ANALYSIS_STATS_TOPICID

The Topic Id for the “SentimentScores” topic.

  1. SAVE all the changes in main.py.

Environment Variables

In your project, click VARIABLES, and add the following variable in the window that appears.

max_samples

50

The average score is a rolling window that averages the scores of the last max_samples tweets. You can use 50 as a reasonable default and experiment if necessary.

Dependencies

We’ll use the following libraries:

  1. transformers — natural language processing

  2. Beautiful Soup — parsing HTML

  3. Emoji — parsing emoji

Open requirements.txt and add the following lines to the bottom of this file and save it:

transformers[torch]
bs4
emoji

Deploy the project

Now as before, make sure you have saved all changes and Tag the latest version with a sensible tag.

deployment sentiment

Click DEPLOY and edit the deployment dialog:

  • Select the Version Tag you just created

  • Change the Deployment Type to Service

  • Set the "Memory in MB" to 1000

  • Click DEPLOY!

If the memory slider doesn’t go to 1000 you can type it into the input field on the right

Visualizing the Live Sentiment Scores

Using the left-hand side nav, navigate to Data. You should see the Sentiment Results stream in the list of streams. Hover over the row and click the Visualize button.

data visualize

You’ll be taken to the Visualize section where you can select the parameters (data points) that you want to visualize (as a waveform or as a table).

visualize params

You can also click the LIVE button and click + to zoom in on the stream and watch the data coming in real-time.

visualize waveform

Recap — What did you just do!

If you ended up with something that looks like the above visualization, then:

  • You created, coded and deployed a real-time, always-on solution to the Quix serverless compute environment. All for Free!

  • You created two services (two code projects). One to read data from another platform and one to analyse that data using natural language processing. All in real time.

  • You gained some experience of navigating the Quix portal and how to configure deployments. Including using and setting environment variables.

What’s Next

What else can you use Quix for?

If you haven’t already done so, why not check out out Currency Alerting tutorial, which uses the Twilio and CoinAPI services to track the Bitcoin price and send alerts when it reaches certain thresholds.

You can use what you’ve learned here to analyse Twitter sentiment of any topic, give it a try.