> * If you data is more fluid and changing all the time, and/or if it comes in JSON serializable format, consider switching to postgresql ^9.4, and use the JSONB columns to store this data. You can index/query those columns and performance wise its on par (or surpasses) MongoDB.
Would it be worthwhile to switch from MongoDB to Postgres (when MySQL is being used alongside Mongo)? I'd have to do lots of testing with it to know for sure though, I guess.
I use Mongo for timeseries-like data, and have >1 billion documents in Mongo currently. Querying super old data is pretty slow (and not needed generally), but querying recent data (recent being the last month) is "fast enough".
If your setup makes sense for you, its fine - stay with it. focus on real problem, not ones people on this thread think you have :)
I'm not one of those mongo bashers. I've used mongo before for the same purpose you do - time series statistical data along side a postgres db for everything else. And it worked fine for me. However, at that time postgresql was pre 9.4. Today I would have kept it all in postgres and simply archived older data if needed.
And to be fair, its not even a "problem" until you need to filter by one data source and sort by the other. We had to be creative when we got to those problems.
As for my choice of Postgres over mySql - that is for 2 reasons - first being that postgres is not owned by Oracle, and second that postgres is more mature, feature rich and reliable than mysql (i.e., better).
If you can reliably externalise the timestamps into Postgres for the documents, then yes. Just index and primarily query on the timestamps and you will be golden.
Also switching from mysql to Postgres should see performance gains (though unlikely to be a bottle neck).
> "When national security programs are hobby-level, you really have to worry that anybody else can do them."
It's not really surprising that bulk surveillance is simple, in my opinion. The act of collection/surveillance itself isn't the hard part. It's doing it on a much larger scale.
Wardriving a Starbucks as a hobby is slightly different than installing a specialized device into every downtown DC Starbucks, even if functionally both do the same thing (collect data and/or injection).
>It's not really surprising that bulk surveillance is simple, in my opinion. The act of collection/surveillance itself isn't the hard part. It's doing it on a much larger scale.
Sampled netflow at 100k connections/sec is straight-forward. You won't do it with BPF, but it's straight-forward with technologies like netmap or DPDK.
Using "the web" as an example TCP session requires a 3 phase handshake (SYN -> SYN ACK -> ACK), then the client can make a request (GET / HTTP/1.0) followed by the response.
The SYN/SYN+ACK/ACK packets are 40 byte each (IP + TCP headers), plus another 18 bytes (DST, SRC, length, CRC), plus another 20 bytes of 'overhead' (preamble, SFD, IPC).
Unfortunately, minimum payload on Ethernet is 46 bytes ('octets'), so the actual on-the wire SYN/SYN+ACK/ACK exchange is 84 bytes each (includes all overhead). Even 1G Ethernet has bandwidth sufficient for 1.488 million packets per second at this size.
Even if all the frames on a 1Gbps link are 1538 bytes (includes all overhead), you still have to deal with 81,274 packets per second.
The HTTP GET request is likely between 100 and 1500 bytes (depends on headers), and, if we figure that the response fits in a 1500 byte frame, (1460 bytes of actual content, plus headers and overhead), we then have a 4 phase close (FIN/ACK from each side). These, again are minimum 84 bytes on the wire.
So the reality is that given a web server that doesn't spend too much time sending actual data (maybe several packets for the page data), it's easy to get to 100K connections per second.
However, 100K connections / sec (remember that's sampled) is 8.640 billion connections per day, and now you have a classic "big data" problem.
What a world we live in. Defending the security of a nation = laid-back, no stress. Writing yet another CRUD backend for a mobile app = crazy hours and ulcers.
I mostly agree that the environment in government contracting is pretty laid back. We definitely have surge times, but generally we're focused more on getting things done correctly over fast. I think there's also a good understanding that people aren't as effective when they work over 40 hours/week for consecutive weeks.
Having been in government, this seems to be the opposite of what I've seen. Makes me think it is fully a byproduct of culture, independent of the work done.
I'm a full-time employee. Managers and government benefits are a pretty big part of it. Work hours are very flexible (some roll in at 6am, others 10am), lengthy vacationable time, etc. Like morga3sm said, we want to do things correctly.
It has its downsides of course: getting a TS security clearance can be stressful, but that goes away afterwards; not being able to tell your SO what you do other than high-level stuff; career advancement and progression can be a long process and you basically know where your salary is going to go when you first start (because it is standardized); and so on.
I find that incredible (in a good way!)
My idea of govt organizations was strict dress code, clock in at 8 AM on the dot, leave at 5 PM sharp, unknowable bureaucracy above your head.
Yep, that was my idea at first too! But in the summer I see people in flip flops and shorts :-) Execs still wear suits, but us lowly engineers not so much.
It depends on the contract. For the most part, the scrutiny of government auditors ensure that you actually can't work more than 40 hours in a single week. Sometimes the contract also ensures that the work cannot be done outside of a government-controlled facility.
So if you're the actual do-the-work peon instead of someone in any of the several tiers of useless middlemen, you work regular 8 hour days, get 10 or 11 paid holidays, and when you get home you are simply done with work until the morning of the next workday. It does not matter much if you do your job well or do it poorly, because there are so many other people working on it that you will never get recognition for a job done well, nor will you get any blame for doing it poorly, so long as it does what it says on the box.
The deadlines also vary by contract, but as every layer of middlemen adds its own padding, it wouldn't be remarkable to have two years to develop yet-another-CRUD-app with zero scalability, compatibility, or interoperability requirements.
No one doing the actual work is a direct government employee, because the uniform pay schedules are completely incompatible with industry norms. The government employees are simply there to make sure the contractors do what is required.
The net result is that you can get all your work done in just 2 hours and look busy for 6, or amble through it at a sloth's pace, gold-plating everything, achieving 100% test coverage, using ordinary software as a teaching lab for industry best practices, and such.
There are plenty of downsides, of course. The work is never glamorous, and you never really have much say in what you do or how you do it. You have zero job security, as you could be out of work if the wind changes direction in Washington, DC. The codebase will always be complete garbage when you are first introduced to it. The unimaginative and slow environment is not stimulating, so you absolutely must have a hobby or side-project that can engage you mentally outside of work.
> There are plenty of downsides, of course. The work is never glamorous, and you never really have much say in what you do or how you do it. You have zero job security, as you could be out of work if the wind changes direction in Washington, DC. The codebase will always be complete garbage when you are first introduced to it. The unimaginative and slow environment is not stimulating, so you absolutely must have a hobby or side-project that can engage you mentally outside of work.
These are the same downsides you see in the private sector.
> For the most part, the scrutiny of government auditors ensure that you actually can't work more than 40 hours in a single week.
Have yet to encounter a contracting environment where is is true. In fact, I was explicitly told that since I was salaried I was expected to put in whatever work it took to make deadlines, despite only putting 40 hours down on my timecard. Considering the facility I worked in had a DCAA office embedded in it I find it hard to believe the government did not know this was going on.
Oh, how delightful. Call the government ethics hotline and inform them that your employer is defrauding the government. You might get a nice bonus out of it, in theory.
In reality, whistleblowers always get screwed. You would have to extensively cover your own ass, and gather evidence that absolves you, specifically, of wrongdoing first.
If you are reporting time worked as anything other than the actual number that you were working, that is illegal. So alternately, start reporting the actual hours worked. If your employer makes an issue of it, tell them that you will need a signed, written copy of any order they give you that instructs you to violate the law.
If you get fired, you can retaliate by getting someone sent to jail.
I haven't worked there in almost four years, so that would be hard to do at this point. Even if I was, part of what sustained this is that we were a classified program, so only cleared auditors could access the full information.
Furthermore, our government customer didn't care as long as they got their deliveries on time and as cheaply as possible. I know in theory a program office can't lean on DCAA, but in practice (especially considering the clearance situation) I don't know if the PO would obstruct such an investigation. After all, from their POV getting what they want for less is not fraud.
Are you in SIGINT? If you're not in SIGINT, I agree with everything you've said, otherwise my experience has been pretty different likely because I'm in SIGINT. Most of us are employees and the work can be pretty satisfying and rewarding, especially when you can see the effects and/or results when you're just a small cog in the machine.
It's possible we're in different countries, though :-).
The federal government has exactly 10 federal holidays: New Year's Day, MLK Jr. Day, Washington's Birthday, Memorial Day, Independence Day, Labor Day, Columbus Day, Veterans Day, Thanksgiving Day, and Christmas Day. Government controlled sites are closed on those days. That means contractors can't work at such a site on those days. Rather than requiring their employees to work elsewhere, contractor companies often just concede that their employees can get a holiday too, and most of them even make it paid time off.
Some companies observe different holidays, such as by adding Christmas Eve and the day after Thanksgiving, and removing Columbus Day.
There is no such thing as public holidays in the US. Some days are bank holidays. Some days are school holidays. Some days are federal holidays. But there is no law that requires any employer to give all of its nonessential employees paid time off on any particular day. The only sort-of-exception is that an employer can't prevent someone from voting on Election Day, but that does not have to be paid leave.
Wal-Mart, for instance, has only 5 paid holidays: Easter, Memorial Day, Independence Day, Thanksgiving, Christmas. (Note that Easter is always on a Sunday, a time when many people already have a day off from work.) And if you get holiday bonus pay for working on those days, your hours are rumored to be cut afterward to compensate.
Some companies are also generous enough to allow 80 additional hours of paid time off every year, to be used as either sick leave or vacation days.
To me as a US English speaker this meant public holidays, since we normally call the other type of holiday "vacation". He probably gets at least 10-15 of those as well.
Depends on the company and facility situation. I worked for several years in such an environment, but the facility was open 24/7. You could be paged at any time for critical issues, but of course would always have to come in to find out what it actually was. Not being able to bring work home just led to the expectation that you would spend more time in the office.
Agreed. When I started learning to program around 12, I was very motivated by what I wanted to make, not follow some bland tutorial ala CodeCademy.
From my experiences in university so far (Comp Sci), this experience seems to be pretty rare. If I ever have kids, I hope to be able to guide them in a similar way I learned as a kid. If they want to mod Minecraft, sure, let's start with that and not a boring tutorial for TodoApp#5034.
What you say about Minecraft rings true. I've been trying to get my 12 year old brother into programming for a while, never really succeeded. Today I setup Forge and taught him how to make an item for Minecraft. That got him excited about programming (or at least what one can do with it) more than anything else.
It's a bit less polished, but far, far better when it comes to hacking. I found it significantly easier to create a mod for Minetest than to use a mod in Minecraft. Once my kids grasped the idea that they could add whatever items and behavior they could dream up, they really took a shine to it. The Lua coding used is very approachable.
That does actually seem like it might be better, I admit Forge and Java would probably seem quite daunting to someone new. I'll see what he thinks!
The only issue though, is that I think a lot of his motivation stems from potentially sharing what he has made - pretty much everyone his age has a copy of Minecraft.
There is a bit of a network problem with it; sure.
One advantage that may help is not needing a license; if you have siblings that have to share a Minecraft account they can play Minetest on as many machines as they have without bugging their parents to pay for a new account. Plus it can run on much older machines.
But yeah, it can be a tough sell just because people don't want to bother.
Indeed. I made a project a few years ago and maintained it for several years. At one point I just didn't have the time (or motivation, I suppose) to continue updates and essentially abandoned it without much official word, while putting in a good amount of work every few months. The guilt was pretty crushing, so I can understand where the author comes from.
Fortunately, someone offered to take it up recently so I feel much better now.
