Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Would agree with all of this.

The problem is that BART doesn't employ talented reverse engineers. There is now way they could afford them in that area.

Also: Talented reverse engineers are hard to find on a good day in any given area. I used to think that my mentors from the start of my tech career were just the baseline level of sophistication you'd expect of any decent software engineer. 20 years later I have seen how few people have the mindset to get into that sort of thing. (And it is a mindset much more than a level of intellect, IMO)



>Also: Talented reverse engineers are hard to find on a good day in any given area

Not true. Have you looked in Europe? It's full of them thanks to the tinker culture and the presence all big anti-malware companies (Esset, Avast, Avira, Bit-Defender, Kaspersky, etc). Plus every single cybersecurity consulting company out there in the world, big and small, has guys who can do reverse engineering for you. There's thousands of these companies, just look them up.

The reason you're probably not finding them is because you're probably not looking for them or that many juniors who don't have the "RE bug" don't stay in that gig professionally too long because, unless you're a zero-day finding God, cashing out six figure CVE bounties and giving Defcon talks, the pay is worse than a webdev doing CRUD since the demand for reverse engineering work is very low and not valued by customers as it's not the kind of job that impacts their sales or share price so they just look to offshore it to the lowest bidder.

I also left the field and moving to cloud engineering since the demand, pay, benefits, respect is so much better.


Not true. Have you looked in Europe?

I wonder to what extent that is still the case. When I graduated 15ish years ago it was definitely true that many of my classmates had those low level skills, and serval of them had written their Masters thesis on something related. However looking at people graduating from the same program today, I hardly see anybody doing that sort of work. Everything is either some sort of high level web stuff or machine learning.

Even the people I knew who had those skills haven't used them professionally in over decade since, as you say, the pay sucks.


>I wonder to what extent that is still the case.

It is. The university in my area has a famous master's program for cybersecurity and many grads there know reverse engineering to a degree. And there a re many such universities in Europe, plus all the Eastern European gray beards who work in the industry since they cut their teeth breaking DRM to pirate games in their youth.

I personally know at least 5 people who can do this work professionally at a consulting company.


I don't ride Bart anymore, but plenty of us would have done this work for free.. just to see a small chance at improving the system.

The dysfunction is broader than them being unable to afford engineering talent.


> plenty of us would have done this work for free

This is the interface to the train control system - a primary safety system - responsible for the lives of literally a million people on an average month. Are you prepared to spend months of your lives rigorously testing and phasing this system in? Are you prepared for the intense legal liability if something goes wrong?


If an individual engineer is put in a position where they can endanger lives by accident, there is something wrong with the management and systems in place around that person.

This is a team game. The people figuring out how the comms works and creating tests for those communications then hand their work off to the "rigor" team.


Or do like what the dutch railways are doing: run the old and new systems in parallel on real inputs and check if the outputs match. Then proceed to only use the output for the old system until you're really really sure there's no bugs.


I don't work in safety-critical hard real-time systems, but isn't this the standard approach when upgrading such a system?


I'd imagine so, I just knew about the dutch railways for certain. IIRC they are doing a mechanized translation from COBOL to C++. The idea being to first test that exhaustively, and then slowly manually translate the COBOL-y-C++ to more idiomatic C++.


>they are doing a mechanized translation from COBOL to C++.

That sounds interesting; do you mean that they are converting the current working system written in COBOL to C++ using a transpiler? Do you have any links to more information on this?


I do mean that. I can't find it now, but I read it some years ago in a news article. The project was slated to run for a loooong time so I've assumed it's still going on.

Sorry I can't be of more help wrt sources.


No problem, a search for "COBOL to C++ transpiler" actually brings up a couple of companies which is interesting. But how do they verify the resultant output? Mere testing cannot be enough for critical systems software.


Yes/no. The (formal) reason SFMTA went with the Thales/Alcatel system was the ability to overlay the new setup over the old. None of the other bidders met that requirement. Depending on the hardware side of things it's entirely possible you couldn't/wouldn't want to run the two systems in tandem with BART.

That said BART wanted to run AATCS trials on the mainline in Oakland. No clue if they actually managed to do that before GE pulled the plug on the project.


And test the crap out of it by just running trains empty at night when they are not running anyways!


Running trains is rather expensive and generally not something you'd do to test software. Especially given that in this case a software crash can result in a train crash.


>responsible for the lives of literally a million people on an average month

Even in 2019, BART never cracked more than 500K station exits per month, and now it is well below half that number[0]. And of those monthly rides, surely a majority of them were the same people taking multiple rides.

Additionally, even a fatal flaw in the software would most likely not result in every one on every train simultaneously dying or even being injured.

[0]https://mtc.ca.gov/tools-resources/data-tools/monthly-transp...


100,000s of thousands of people is still a lot of people. my bad

> Additionally, even a fatal flaw in the software would most likely not result in every one on every train simultaneously dying or even being injured.

I'm glad you're so willing to play fast and loose with safety, but I'm not. Not to mention the impact due to lack of trust when an accident happens! I'll leave train systems to engineers who are rigorously testing their implementations, not you, thanks :)


Sorry, I should have been clearer. "The work" here referred to assisting with the reverse engineering aspect of this.

I, and almost everyone else here, lack the domain expertise to be of any use in that.


  The problem is that BART doesn't employ talented reverse engineers. There is now way they could afford them in that area.
Well, no. Money is not the problem and BART doesn't need reverse engineers. They've already thrown away tens of millions of dollars on a replacement train control system (AATCS). Everyone's favorite defense contractor Raytheon was tasked with the project until they realized they couldn't make it work. Eventually Raytheon pawned the project off on Harmon, which was later bought by General Electric. GE shut the whole thing down because it was going nowhere.

I think most folks here are drastically underestimating the complexity involved in getting a train control system right (and how badly BART botched the original system). Considering that BART dates back to the 70s, DOS and Windows 98 were upgrades. Meanwhile Muni soldiered on with OS/2 for nearly a couple decades.


> I think most folks here are drastically underestimating the complexity involved in getting a train control system right

The phrase "butt-puckering" comes to mind. Ya some whizkid could come along and put something together but they'd be foolish to attempt, there is 1000 atm of pressure to not get one damn thing wrong. All the things currently wrong with the system are grandfathered. One single wrong thing in your new system and you are toasted.


Sure there are safety considerations (and plenty of dramatic safety events from BART), but I'd venture to guess even there it's rarely quite so black and white.

I was actually thinking more of day-to-day nuisance problems than anything else. Things like getting stuck at the station while the train's computer gizmos reboot, stop annunciators getting confused (or out of sync with each other), having to position the train manually within a station. Or even software failures as hardware wears out – shit like that almost always fails in a safe manner. Unfortunately even a safe failure can have massive consequences which raises the cost of failing fast and breaking thing often.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: