c5 is such a bad instance type, m6a would be so much better and even cheaper,
I would love to see this on an m8a.2xlarge (7th and 8th generations don’t use SMT) and that is even cheaper and has up to 15 Gbps
Actually for this kind of workload 15Gbps is still mediocre. What you actually want is the `n` variant of the instance types, which have higher NIC capacity.
In the c6n and m6n and maybe the upper-end 5th gens you can get 100Gbps NICs, and if you look at the 8th gen instances like the c8gn family, you can even get instances with 600Gbps of bandwidth.
A Samsung 990 Pro reads at something like 50 Gbps and PCIe 4.0 x4 is quite a bit faster than that. You can get this speed with a queue depth that isn’t crazy, and you can have multiple NVMe operations in flight reading the same large Parquet file. Latency is in the tens of microseconds.
The consensus seems to be that S3 can read one object at somewhat under 1Gbps. You can probable scale that to the full speed of your NIC by reading multiple objects at once, but you may not be able to scale by reading one object in multiple overlapping ranges. Latency is in the milliseconds.
So, sure, an EC2 with a fast instance and massive multiple object parallelism can have 10x higher bandwidth than an NVMe device, but the amount of parallelism and latency tolerance needed is a couple orders of magnitude higher than NVMe. Meanwhile that NVMe device does not charge for read operations and costs a couple hundred dollars, once.
If you are so inclined, you can build an NVMEoF setup (at much much higher cost) that separates compute and storage and has excellent performance, but this is a nontrivial undertaking.
Off-Topic: Most stories about politics, or crime, or sports, or celebrities, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic.
Conservatives are champions of free speech unless it concerns Palestine and Israel. In this case any vague rule is good to supress it and impose full censorship.
it's a tale as old as time for people to believe the things they're in support of get censored while the things they dislike are over-represented. HN is genuinely relatively divisive on basically any issue, as humans we tend to be a bit blind to these things though.
The wildest one was the Tom Alexandrovich story. That guy literally committed sex crimes while in town for a tech conference and was let off - he works in our field, but it immediately got buried.
Palantir is openly named after a device of fantasy Satan. I don't think we need to wonder how evil they're being, it's always worse that you think they're being. That's what they're for.
The moderator dang has actually responded to several comments about their editorial stance re israel-palestine. I don't have time to find them but I've engaged with him personally on this several times over the last couple years.
Basically it boils down to he thinks we have talked about it enough and it annoys him how upset people are about it. And ultimately he considers admiring the technology used in war machines (and terrorism like the pager attack) to be clearly within the editorial remit of HN but that discussing their effects on the people they are used on is borderline so they police it more vigorously.
Ah, they are annoyed that people are upset about the genocide and keep talking about it. Very interesting. Well, that's their right, of course, annoyed that people are not closing their eyes to blatant human rights violations and genocide. I will keep calling it out for what it is: Israel is a genocider state, until it's stopped.
YSK that people get banned here for calling Israel out.
It will be claimed that it's over something else, to maintain plausible deniability. Or that you were starting a flame war.
I'm just saying: this is one of HN's most protected third-rail topics, so I'd advise you to have a backup account and a VPN handy if you want to keep stating the obvious and standing up for the voiceless.
Seems like a low risk test, it’s kinda a no-brainer. They used their free time and they had it running alongside the main program.
Have to remember if they didn’t have the comparison, I don’t think it would be as good a story.
“Nobody Ever Gets Credit for Fixing Problems That Never Happened”
“Nobody Ever Gets Credit for Fixing Problems That Never Happened”
While I agree...
The first few years at my current company, they kept telling me I did really good work, but I was too slow.
Then we had a security breach, and then an audit. My code was the only code that they found no vulnerabilities in, and most of the rest of the code was riddled with them. (TBF, there have been security vulnerabilities in my code, they just didn't find them in that audit. I'm not perfect.)
Since then, they have never complained about me being slow.
While that's not credit for fixing problems that didn't happen, I think it's the closest thing out there.
I have a slightly different interpretation of this, and it comes down to the horrible manager-ese phrase "risk appetite".
As a small scrappy startup shipping is the absolute priority - get something out in front of customers, because the existential risk to the company is that you run out of money before you've got either a sustainable income or investment. Security might get some lip service paid, but ultimately who cares if your code is insecure when there's no customers who's data is at risk yet.
Over time the company will (hopefully) grow its customer base. Maybe you get some big B2B contract to fulfil. That's the point at which things like security audits land, and people start having to really care about security, because now there's half a million people in your databases, and if they get compromised you're going to be front page news.
You need different kinds of people as a company grows. The scrappy "get it shipped" engineers of the early days are going to find it increasingly difficult to function in a larger process focused organisation because its no longer just a case of sitting down over lunch to hash out a new feature before hacking something together in the afternoon. Process means that's now a multi-week process involving three different departments, followed by a month of waiting until the necessary people are available to actually build it.
I don't really have any good answers to that. I suspect small skunkworks like teams are probably a good first step to being able to retain those early engineers into the later stages of a company, but I've never had the chance to try it.
I don't disagree about the company's needs changing, but they'd already recognized that need when they hired me. But they didn't actually recognize how well I was fulfilling those needs until the audit.
I think it’s mostly from using temporary accounts to hide transactions. When you can “open” an account digitally without limits, this seems like the inevitable. They should look at actively traded accounts. Fast search shows ~1M active out of 460M total.
