Hello,
I recently received a free BuyVM box (thanks: @liam / @Francisco) and a free MySQL account (thanks: @Fliphost).
First thought: Now I do have enough resources to start my twitter archive. My first version was a local backup of my own tweets. Soon some of my friends started requests to add different twitter timelines. So I created a little frontend to list/search through the growing list of tweets. After some weeks the whole system was ready.
Currently the system is split into three parts:
- A lighttpd proxy running in front of a thin cluster
- A offside MySQL Database holding the jobs (timelines, searchterms) and the tweets
- A RabbitMQ server for command handling + ruby workers doing the jobs – running on a ipxcore vps (thanks: @Damian)
Every command issued by the web servers (excluding queries to the database) are forwarded through a RabbitMQ server (message driven) to ruby workers doing the Twitter API crawling – and storing the results into the MySQL server.
It is all about limiting the web frontend as much as possible. The only logic implemented in the web part are the queries every other “action” is handled through jobs sent by messages to the RabbitMQ server. On the other side there are workers listening on queues for new commands. The message routing / handlers within the RabbitMQ server do all the magic (filtering/delay/…).
Now I want to test the system as a whole. And therefore I ask for your support and your feedback.
For LET I have created a LEB related list of timelines (the system is supporting different lists to separate different user groups). So all major LEB hosts should already be added to the system. If anyone is missing -> feel free to add them to the system (add a request for a new timeline).
If Francisco and Fliphost agree I will post the IP of the web frontend. If not I will use “private messages” for everyone willing to test this tiny system.
Next steps:
* RSS Feed support for search terms
* Download of timelines (json)
* Customizing initializr template
If you have any suggestions or change request – just post them.
Kind regards.