twitto.be - realtime twitter dashboard

Twitto is a realtime dashboard monitoring tweets about any topic. It presents statistics such as a graph of the most popular hashtags, most mentioned twitter users, number of tweets about the topic posted by minute, second. In addition, a twitter bot posts the list of top @mentions and top hashtags regularly on twitter.

Twitto is a side project, created to both showcase my know-how in data processing and dashboard design, as well to polish my skills in web development.

Twitto revealed itself to be an automated awareness and engagement machine, gathering interest from the public while running totally autonomously between 2013 and 2023.

(approximate figures, based on Google Analytics and twitter analytics)

Story

Back in 2012-2013, as most of my career had been revolving around corporate Business Intelligence, using Enterprise software like Business Objects, Oracle database and powerDesigner, I was looking at starting my own startup projects, innovating, and exploring leaner technologies.

I started to use twitter during a Startup week-end Brussels event, and was amazed by the incredible data source it was: besides accessing it via the website twitter.com or mobile app, all tweets and related information are available to any program, for any purpose.

I decided to use this as a data source, to power an interactive dashboard analyzing the tweets about a given topic (and/or geography).

While the demo website is focused on the Belgian entrepreneurship tweets, anyone can use twitto to monitor the topic(s) of their choice. The EU social media consultancy Vattel uses twitto to follow the trends by the “Eurobubble”: the crowd working inside or with the European institutions. Country variants twitto.es and twitto.gr have also been created.

Twitto is a non-commercial side-project, and its code is open source.

During the past 10 years, twitto has been one of my pet projects, and working on it on and off has been a great opportunity to experiment with both technologies and user behaviours.

How it started: a twitter influencers ranking - 2013

Twitto.be website was launched in June 2013, as a directory of Belgian influencers on twitter. It used the “Klout score”, which was a ranking associated to individuals based on their public social media activity. Klout scores were used as a ‘social proof’ by some social media pros to brag about how “influential” their were, but were also heavily decried at as meaningless measurement. From my observations, regardless of how flawed the concept of measuring “influence” is, Klout scores were a good indicator of the potential reach twitter users had (read also our articles series on the topic from October 2013: Finding twitter influencers - the recurring problem of defining influence).

Twitto grew step by step in terms of features (release early, release often), letting visitors filter the influencers through interactive charts. It was then possible to find twitter users related to a topic, in a specific location (down to the province level) and speaking a specific language. All this using the information publicly available, extracted using the twitter API (API stands for Application Program Interface and is a mechanism commonly used by organizations to give access to their data to third-party programs).

One year after the launch, the website referenced more than 100k twitter users, and got close to 8000 visitors. We wrote a blog article with interactive charts to highlight the results of the experiment: Belgians, twitter and Klout - One year of twitto.be.

Putting the project on hold for 10 months - 2016

At the end of 2015, the website’s code needed some maintenance: a memory leak was causing all server memory to get exhausted, then the website would just crash every other week. I didn’t have time to focus on it, so I just replaced the website by a page informing that the project was offline, pending redesign.

During this period, the website still received more than 1500 visits, due to the momentum from previous versions.

Twitto’s rebirth: redesign focused on content - 2017

Back to the project, instead of just solving the bugs affecting the stability of the previous version, twitto was rebuilt from the ground up.

Key aspects of the project:

• Get more privacy-friendly (before the GDPR era!): focus the monitoring on content rather than people,
• Switch to realtime : get all the tweets about a specific topic in real time, update realtime statistics and display them in a web page,
• Change server-side code’s technology: get rid of PHP (Laravel framework) in favor of node.js.

The new version of twitto got the following features:

• tweets statistics ring chart: number of tweets containing a link, replies, pictures and videos…
• top hashtags in a horizontal bar chart
• top @mentions in a horizontal bar chart
• geographical map of the tweets location, and tweets languages
• breakdown of languages used in the tweets
• timeline of the tweets posted in the last 30 minutes
• timeline of the tweets posted in the last 60 seconds

Twitter’s realtime API is amazing. You can literally publish a tweet on twitter.com containing a certain hashtag, and quasi-instantly see the twitto.be statistics getting updated on twitto.be.

This new version of twitto.be website got a little less interest than the previous - which was flattering influencers’ egos. However, a twitter bot tweeting top trending hashtags and mentions every two hours revealed itself being an engagement machine: getting likes, retweets and replies quite frequently.

Wider adoption

My colleague and friend Panos - who actually worked a bit with me on the first versions of twitto - launched two websites using twitto: twitto.es and twitto.gr - focused on Spain an Greece, respectively. Both the websites and twitter accounts got a way higher engagement than twitto.be: not only the countries have a bigger population, but twitter was heavily used at the time in these countries, especially Spain where heated debates were happening about the political situation.

Vattel.com, a social media consultancy, also hired me to create a variant of twitto monitoring the conversations about EU institutions - the consultancy’s target market.

Final touch: graph analysis feature - 2020

Besides multidimensional analysis which has been the main area of focus of my career, I find Social Network Analysis fascinating, as it is capable to give a clear perspective to otherwise chaotic-looking data.

Rather than just counting the hashtags, a graph visualization shows how each is used relatively to others. As the graph processing library Graphology looked like a serious tool raising in maturity, twitto was updated to include a hashtags graph, with community detection, node measures and a force-directed layout computed and updated in realtime. The result is not as polished as I would have liked - preventing overlaps while showing enough information could be fine-tuned -, the experiment was fun and results in a cooler version of twitto.

The New Twitter - end of the story? 2023

In February 2023, Twitter announced that the free access to their API would be closed within a week, and that a new paid offering would soon be announced. Nothing happened at the set deadline, and a few vague and clumsy messages followed, talking about a release in “a few more days”, then a roll-out “over the next few weeks”.

Twitter streaming API version 1.1, used by twitto, eventually got closed in March 2023, with a recommendation to use the API version 2 - released in 2018 -, and with no information yet about what the new paid solution would be. This is when I decided to shutdown twitto.be website: I do not want to make changes today to switch to API v2, then have to re-do all the work in a few months once twitter figures out what their new paid offering would be. Twitto.be is therefore put offline, and the open source project archived (still accessible on github if anyone wants to do something with it).

Closing thoughts

Beyond the technological challenge of processing data automatically in real time, twitto has provided interesting insights on how to catch the attention of social media users.

• despite the common advice not to build a product over another company’s offering, twitto has amazingly been able to run in autonomy, needing only a minimal involvement to keep it running over 10 years. This is very unique to twitter, which has been supporting its APIs very consistently. Other tools I created (multiGa, invEx, tagPlan) using Google and Facebook APIs required much frequent adaptations to changes by the data providers, making them less worth working on.
• ranking of people - “influencers” - gathers much more attention than ranking popular content: the early versions of twitto.be website ranking top influencers got about 4 times more visits than the next versions identifying top content.
• automated posting of trending tweets and people around a certain topic is a very efficient way to maintain an active presence on social media, and to get a community engaged.