I have had some time off this week so decided to work on something a bit different. I have been working on a concept fuzzing framework for security testing. In the screenshot you can see some of the files produced by it - The bottom right is the configuration used to generate the file format (for this case Bitmap, although I have tested a few others like WAV)
Bottom left is a bitmap produced with no defects. The top shot is a bitmaps produced with some random changes - you can see the green bitmap is now corrupted due to a change somewhere in the format.
This week as been all about getting a second version stamped and out (nowhere near a finished version or even 1.0, but a 0.2 version)
I've been working on https://ghostream.com a framework for constructing stream processing systems. Getting a first draft of the website (http://imgur.com/ujsmjJz) and the documentation (although there is still much to do)
And after that, lots of bug fixes and environment clean ups (code coverage pipeline / static analysis etc.) Managed to knock down quite a few little bugs that way.
I've played with a variety of frameworks (Storm and IBM Infosphere Streams are probably the two I have worked with the most) - In my previous job I was in a team which designed and developed our own (and later integrated partly with IBM Streams) I took down many lessons from the initial creation, the later integration and the work following that. ghostream was born out of those lessons.
Mostly I found the follow problems with existing frameworks :
- Tied to an execution environment - Storm with ZooKeeper and Streams with their own custom one. This makes it very difficult to use either for small projects - and limits integration choices when scaling.
- Resource hogs - The IBM Streams environment is a huge resource hog, I never really got that far with Storm but the number of dependencies it required just to get something up and running provided a similar sense of dread.
- Easy to debug/optimization - Streams has the best tools for this at the moment but they are all heavily tied to the execution environment. I'm hoping with ghostream that structures can be built at the protocol level to provide a way to build tools on top - not tied to any particular environment.
I posted this on the last thread, but I am definitely still looking for volunteers.
I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.
ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.
There is a great episode of Black Mirror (All episodes of Black Mirror are great, but this one is more relevant - Season 1 Episode 3) titled The Entire History of Us, it turns the figurative mirror on a world where people do exactly this...
I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.
ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.
I am still amazed when I try to read a simple blog article and am greeted with a blank white page, it turns me right off the site and I go spend my time elsewhere.
I will happily shell out money for content that I like.
On my side projects I tend to use Vim for C++ which works fairly well after heavy customization with plugins.
At work I have tried to adopt Vim with Java but Java does not lend itself to simple developments (far too may 3rd party library imports). Eclipse is bulky and slow, doesn't customize well and constantly crashes.
I have heard emacs lends itself better to Java than Vim due to it's extensibility, but devoting the time to learning another application is not really something I want to do.
Vim is too basic for a complete IDE (and it should remain that way it is a text editor after all), Eclipse is too bulky, I want something that is just right.
I wish I could say Java was my target market, but unfortunately, I'm more focused on javascript. The app will of course work with Java, and it will be open-source so maybe somebody will inplement some killer Java features.
Have you tried SublimeText or LightTable? Did those not fit the bill as lighter than Eclipse but better than Vim?
I have tried neither of those two...I have heard good things both though, maybe I will make the time to give the a try.
A Javascript IDE would be great, I remember working on some Javascript a couple of years ago and being frustrated at even getting basic autocomplete to work in any tool (Vim was the closest but lacked any context). And +1 Open Source. I look forward to giving it a try.
I'm currently building ghostream (https://github.com/ghostream/ghostream) - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidifying and the new test framework is starting to shape up nicely.
Probably not what you are looking for, but I would love to see more real-time data curation - I would certainly be willing to hand over cash for some decently priced real time data feeds - stock tickers, sports events, raw traffic data, headlines, currency, flights - many of these are available in pull formats, but it would be great to get a push feed.
I have had some time off this week so decided to work on something a bit different. I have been working on a concept fuzzing framework for security testing. In the screenshot you can see some of the files produced by it - The bottom right is the configuration used to generate the file format (for this case Bitmap, although I have tested a few others like WAV)
Bottom left is a bitmap produced with no defects. The top shot is a bitmaps produced with some random changes - you can see the green bitmap is now corrupted due to a change somewhere in the format.