It depends on your use case, as always. Correctness is not always black and white (hence my favourite compilation flag, -funsafe-math-optimizations) and time complexity can be misleading (O(log N) with a large base is O(1) in practice), but a correct, theoretically optimal algorithm might still be leaving a lot of performance on the table. If you're a hyperscaler, a high-frequency trader, or perhaps a game programmer pushing the limits of the platform, small gains can accumulate meaningfully, as can the thousand cuts caused by small inefficiencies spread out over the codebase.
>It all seems very brittle, though. And that something has gone very wrong with our ecosystem of tools, languages, and processes when it becomes advisable to massage source until specific passes in a specific version LLVM don't mess things up for other passes.
I would say the main takeaway is actually to not do that, precisely because it's brittle, difficult to understand, and can backfire by making things harder for the compiler. As I point out at the end, the vast majority of developers will be better served by sticking to common idioms and making their intent as clear as possible using language facilities, rather than trying to outsmart the compiler or, conversely, relying excessively on the optimiser as a magic black box.
That being said, I do find it helpful to understand the broad strokes of the optimisation pipeline (things like "callees are optimised before callers", "maintaining invariants enables more simplifications", or "early simplifications are better") to make the most of it. Like with any other tool, mastery means letting it do its job most of the time but knowing when and where to step in if necessary.
You might want to read from the source file line oriented until you find the start position of the first line to copy. Then copy from the end as described in large chunks (I would do 16Mb chunks assuming ram is sufficient).
The sparse file suggestion a sibling made is really great. Otherwise, you could reverse the chunks as you write them, and end up with a fully reversed file, or just have to be careful about block boundaries and know that contents are forward, but blocks are backward. I wouldn't want to bulk read a 700g file with line processing, if it's possible to ignore that.
Full disclosure: I work for Veniam on connected vehicle technology, including LTE, Wi-Fi, and DSRC stacks.
Unfortunately there's also still no agreement on what 5G will actually be, what technologies it will entail, or how it will be deployed. Of course, since none of this is defined yet, there are also no guarantees on what performance will be achievable in practice.
The industry will need to solve this issue long before 5G is widespread. The solution will likely involve DSRC, C-V2X, or LTE-V/LTE Direct, which are tailored for the vehicular communication use case (involving both vehicle-to-vehicle [V2V] and vehicle-to-infrastructure [V2I]). It's unlikely a single technology will be able to handle all the challenges of a V2X scenario (highly mobile nodes, congestion, low SNR, requirements for sub-millisecond connection times), and so a second connection management layer will have to be built on top of multiple existing PHYs to enable simultaneous low-latency, high-throughput, and secure communications.
(Note that the linked article is actually about in-vehicle communication, however.)
Stupid question, but how do things like this resume from where they left off? Wouldn't the server need to be cooperating in this? Is that build into HTTP?
This is also how download accelerators worked (back in the late nineties, early naught's), by having different connections work at several ranges to maximize bandwidth usage.
Why? How could it be more useful? HTTP byte ranges are incredibly flexible, since you one request can specify many byte ranges at once (it's almost too flexible, since a dumb server can easily be overwhelmed by a maliciously complicated request)
It handles the basic case of fetching the remainder of an interrupted download, and can also support partial downloads e.g. for supporting a video stream with the user jumping to different places in the movie.
We are looking to fill several openings across Engineering, Sales, and Management: https://veniam.com/jobs/. We have opportunities for cloud, analytics, front-end, hardware design, and embedded software work.
Veniam is building the Internet of Moving Things by turning vehicles into WiFi hotspots and building networks of connected vehicles that expand wireless coverage and collect terabytes of urban data for smart city applications.
As a member of our team at Veniam, you will have the opportunity to work in fast moving teams of 3-5 people, work closely with our Customer Success teams and often directly with key customers worldwide. We are organized in changing mission-driven teams that deliver continuously and you will have the opportunity to work on something new every few weeks.
If you're interested drop me a line at hcabral@veniam.com!
This is the service supervision approach, which is indeed the best, avoiding race conditions and nasty polling. If you really must monitor processes which you cannot parent, at least under Linux you can do this using ptrace to sort-of-reparent the processes (as long as you have CAP_SYS_PTRACE or your ptrace scope is set to 0 in Yama), with waitpid to monitor all of them.
reply