In 2017 I built Twitark - https://github.com/pantchox/Twitark
Its a Node.js CLI tool that connects to Twitter Stream API to archive tweets and trends (all countries)
You can connect to Twitter stream API which is about %1 of whole twitter (was mentioned in their forums) and you can connect to Twitter Filter API with predefined hashtag monitoring or users monitoring and get all these related tweets.
Twitark creates daily rotated zip archives for each day structured in folders where each minute of tweets is set is a JSON file.
An archived day of Twitter stream API is about ~2GB per day.
I use this to analyze big data and trends with Twitter, also for past data.
So far it is working very well and I've been working on database adapters such as Postgres, Sqlite which are on different branch (not yet committed to master) so you can write directly to your database instead of zip files.
The second open source project I released is actually being used in Twitark.
It called Joebear (play of words of Jobber) https://github.com/pantchox/joebear
Its a Node.js serial queue manager, using old school Node.js event emitter.
The goal of Joebear is to receive a list of jobs and run one by one, if one fails there is ability to reset or stop at any given time.
In Twitark it is being used to monitor the Twitter Trends API.
I still hope more people will make use of my open source projects, I really invested time to write the documentation as well, but I don't have many stars on Github.
You can connect to Twitter stream API which is about %1 of whole twitter (was mentioned in their forums) and you can connect to Twitter Filter API with predefined hashtag monitoring or users monitoring and get all these related tweets.
Twitark creates daily rotated zip archives for each day structured in folders where each minute of tweets is set is a JSON file.
An archived day of Twitter stream API is about ~2GB per day.
I use this to analyze big data and trends with Twitter, also for past data. So far it is working very well and I've been working on database adapters such as Postgres, Sqlite which are on different branch (not yet committed to master) so you can write directly to your database instead of zip files.
The second open source project I released is actually being used in Twitark. It called Joebear (play of words of Jobber) https://github.com/pantchox/joebear
Its a Node.js serial queue manager, using old school Node.js event emitter. The goal of Joebear is to receive a list of jobs and run one by one, if one fails there is ability to reset or stop at any given time. In Twitark it is being used to monitor the Twitter Trends API.
I still hope more people will make use of my open source projects, I really invested time to write the documentation as well, but I don't have many stars on Github.