Sending TFL BikePoint availibility alerts to Slack

Aim

In this tutorial you will learn how to use Quix to create event driven notifications in real-time. In this example we’ll connect to the Transport for London BikePoint API and send availibility alerts to Slack using Slacks Webhooks.

By the end you will have:

  • Configured Slack to allow external services to send messages via WebHooks

  • Built and deployed a Quix Service that streams data received from the TFL BikePoint API

  • Received messages to your slack channel on the availibility of bikes around London

Project Architecture

architecture

The solution has 2 main elements:

  • A Quix Service to pull TFL BikePoint data

  • A Slack Alerting Quix service

A Quix Service receives data from the Transport For London BikePoint API and streams the data onto the Quix message broker. Another service listens to the bike data stream. When the number of available bikes at a station falls below the desired level an alert is sent to the Slack channel via the Slack Webhook.

Prerequisites

  1. A Quix account

  2. Access to Slack, you’ll need to be an admin

  3. A TFL Account and API keys

If you don’t have a Quix account yet, go here and create one.
Check out the projects README.md later on in the tutorial if you need help creating a Slack WebHook

Overview

This walkthrough covers the following steps:

  1. Creating a Slack App

  2. Setup Webhooks

  3. Create a service to pull TFL BikePoint data

  4. Send messages to Slack from Quix

  5. Deploy the projects and test

Create a service to pull TFL BikePoint data

Login to Quix and create a new Workspace. A Quix Workspace is a container to help you manage all the data, topics, models and services related to a single solution so we advise using a new, clean, one for this tutorial.

Create Topics

Once the workspace has been created. Create a topic.

Find out about Topics here.
Table 1. Create the topic

create topic

To create a topic, go to the Topics Page (see menu on the left) and then click on the + CREATE TOPIC button.

naming new topic

Name the first topic "BikeData".

Click CREATE.

This may take half a minute. A discrete, fully featured and highly scalable data platform is being created for you.

select topic details

copy topic id

Copy the Topic id. To do so:

  1. Click the >> icon on the left of our “BikeData” row

  2. Then on the right hand side click the "Copy to Clipboard" icon

  3. Paste it to a notepad or similar for later

Keep the Topic ID safe for later. You will need it in the project code.

Create the BikePoint project

The Quix Library contains various samples to allow users to quickly explore possible applications of Quix. In this tutorial we will direct you to select TFL BikePoint sample. This sample will connect to the TFL BikePoint API and pull data relating to the number of available bikes at each of Londons bicycle hire points.

Table 2. Create the first project

library menu

In the Quix Portal click the "Library" icon on the nav bar. Select Python as the language and find the TFL BikePoint API option.

select topic

Be sure to select the Topic, "bikedata", we created earlier. This is where the TFL BikeData will be sent to. In the Slack Alerting code we’ll create later this Topic will be read from and we’ll decide what to do with the data in real-time

save as project

Save this template code by clicking "Save as Project" in the top right

project name

Give your project a name like "SlackBikeProject" and click "Create"

tfl api keys

Select the tfl_api.py file and enter the TFL Primary and Secondary API keys.

The README.md in the solution gives directions on how to register for a TFL account and some guidance on getting the API keys.

Create a Slack App

Ensure you’re logged into the Slack web portal here then create a new app.

  1. Click "Create New App"

  2. Choose "From Scratch"

  3. Enter a name and choose your workspace

  4. Click "Create App"

Setup Webhooks

  1. On the left hand menu under "Features" select the "Incoming Webhooks" menu item

  2. Switch the slider to "On"

  3. Click the "Add New Webhook to Workspace" button

  4. Select the channel you want to give access to

  5. Click allow

You should now have a "Webhook URL" which is used to post messages to your chosen channel.

Send messages to Slack from Quix

In order to send Slack messages from Quix we will use another service.

Create the Slack Alerting project

Create another project using the library samples.

Follow these steps to use and then modify the "Hello World" Read example:

Table 3. Create the second project

create hello world

In the Quix Portal click the "Library" icon on the nav bar. Select the "Read" tab, select Python as the language and find the "Hello World" option. Make sure you select "bikedata" for the input Topic. This should be the default but if not ensure it’s selected.

save as project

Save this template code by clicking "Save as Project" in the top right

project name slackalerting

Give your project a name like "SlackAlerting" and click "Create"

tfl api keys

Now you have some basic Python to connect to the Topic and print a value from the data. It prints the numeric_value of 'ParameterA'. ParameterA doesn’t exist in the BikePoint data so lets modify it.

Modify the code

Lets modify the on_parameter_data_handler code to work with the TFL BikePoint data and send Slack alerts.

The curent "Hello World" on_parameter_data_handler looks like this:

def on_parameter_data_handler(data: ParameterData):
	# print first value of ParameterA parameter if it exists
	hello_world_value = data.timestamps[0].parameters['ParameterA'].numeric_value
	if hello_world_value is not None:
		print("ParameterA - " + str(data.timestamps[0]) + ": " + str(hello_world_value))

Replace it with this and then save your changes

def on_parameter_data_handler(data: ParameterData):
	available_bikes = data.timestamps[0].parameters['NbBikes'].numeric_value
	bike_location = data.timestamps[0].parameters['Name'].string_value

	if available_bikes <= 2:
		print("There aren't many bikes left at {}".format(bike_location))

The new on_parameter_data_handler now uses the data sent by the "SlackBikeProject" to the "BikeData" Topic. When data arrives the on_parameter_data_handler method is called with the ParameterData. This contains all of the numeric, string and any blob data sent to the stream.

We extract the number of available bikes and the bike location from the data and if the number of available bikes is 2 or less we print a message.

More changes

Now that we can detect and handle changes to the number bikes available at any given location we can add the code to send Slack alerts.

Table 4. Slack Alerts - At last

edit requirements

In the list of files in your "SlackAlerting" project you should have a "requirements.txt" file. Click it to open it in the editor pane. Notice that the file just contains "quixstreaming" followed by the version.

add requests

Add a line to the file and enter the word "requests". This is the requests library. We’ll use it to send HTTP POST messages to the Slack WebHook we created earlier.

Save the requirements.txt file and go back to main.py

At the top of main.py, under the other "import" statements, add an import for "requests"

Your imports should look like this:

from quixstreaming import *
from quixstreaming.models.parametersbufferconfiguration import ParametersBufferConfiguration
import sys
import signal
import threading
import requests

Now the moment you’ve been waiting for. Actually connecting Quix to Slack.

We’ll again change the on_parameter_data_handler method by replacing the print statement with the call to send a HTTP POST to Slacks WebHook.

So replace the print line with this:

slack_message = {"text": "There aren't many bikes left at {}".format(bike_location)}
requests.post("[placeholder:webhook_url]", json=slack_message)
Replace the placeholder, [placeholder:webhook_url], with your WebHook url obtained earlier

Your on_parameter_data_handler should look like this:

def on_parameter_data_handler(data: ParameterData):
	available_bikes = data.timestamps[0].parameters['NbBikes'].numeric_value
	bike_location = data.timestamps[0].parameters['Name'].string_value

	if available_bikes <= 2:
		slack_message = {"text": "There aren't many bikes left at {}".format(bike_location)}
		requests.post("[placeholder:webhook_url]", json=slack_message)

Deploy

Time to deploy both projects to the Quix Serverless environment.

Deploy the Projects

Find more info on deployments here.

We will deploy both projects, so start with either one then repeat the steps for the other one.

commit history

Look at the section on the right of the project page. This is your commit history.

Your code is automatically committed to a Git repository. Find more info here.

version tag

The top row is the most recent commit. Click on the three dots next to it and then on "Create tag".

Give the tag a meaningful name such as 'v0.1'.

Click CREATE.

TIP: Tags are useful for version controlling the projects you deploy to production

deploy button 2

With our work properly saved, we can now deploy the project. Click the DEPLOY button.

deploy dialog

In the General tab

Enter a name for your deployment e.g. "SendSlackAlerts"

Select the Version Tag you created earlier, v0.1

Change the Deployment Type to "Service"

This will ensure the service restarts if there is a failure

DEPLOY

Hit it!

Just click DEPLOY and the code will be built, deployed and the service will start running.

deployment running

It’s alive!

You should see the status change to Running.

contextual options deployment

deployment logs

Checking your deployment

Hover your mouse over the row to find some contextual options.

You can view the logs and deployment logs, download the docker image and delete the deployment.

After deploying the first project, repeat the steps above to deploy the second project

Test

You will start to see Slack messages arrive when any station has less than 2 available bikes

slack alerts

You will notice that the alerting code isn’t very sophisticated. You get an alert every time there are less than 2 bikes available.

Even if you have just had the same alert!

What’s Next

There are many ways you can use this code, try enhancing it so it only alerts you about each location once, or once every 5 minutes or perhaps only if the number of bikes drops below 2 rather than just every time it is 2.

Using Quix, coupled with something like Slack, allows for automatic real-time alerting. Quix allows you to react in real-time to events or anomolies found either in raw data or generated by ML models.