SW is weird in that way, we can basically build houses that nobody wants to live in. Thing is you probably were building the right thing, structurally speaking.

My rule over time has become "structure follows complexity" i.e. Anticipate complexity where possible but don't apply the engineering structure until the complexity really exists. If you built an MVP house, don't add the second bathroom until you've got customers complaining about deadlocks on the first.

It's tough because that basically runs counter to the Spirit of an Engineer, which is just to imagine the grandest well-thought-out + organized thing and march toward creating it.

The bonus of having two decades+ in building SW though is you start to see the patterns and realize that the same fundamental needs drive everything, so the intersection of building something great with customer needs becomes less of a mystery and more of a "how fast can we juice this thing". At that point I think scalability prep is like compound interest.

Very recently I’ve come to think of my architecture style as Jazz. Like the old trope about listening to the notes he isn’t playing. But also the improvisational nature and the flexibility to adapt to new information.

This is such a wonderfully concise way of describing something that I try to teach people.

I have come to use the phrase "no regrets". Leave things out, but do it in a way that you won't regret it, if you need to add it later.

Another phrase that I read somewhere else was, "Weeks of programming can save you hours of planning" (pretty sure I read it on HN, and I wish I had kept the link). The point being that when you decide to leave something out, or when you decide to intentionally do a half-assed job of something, you should still think about how someone in the future would go about efficiently fixing it.

If you do it right, there's missing functionality, but little or no technical debt. If you do it wrong, you miss out on functionality and also take on technical debt. And I think that's the subtlety that you speak of.

There’s a list of things I won’t try to add on later. It’s smaller than it used to be, but the shiv I brandish at you for suggesting we cut those corners is also much scarier.

Jim Highsmith has the best theory I’ve heard in the subject. His opinion is that the reason we keeps having the same unwinnable arguments is because we think in terms of solving problems but the real issue in front of us is resolving paradoxes. There are no neat answers that fit in a short essay box on the test everything depends, and everything depends on everything, and something is always changing.

What the PTSD reactions come from is other people making decisions on behalf of the rest of the team without consent. But like most things in software it’s a 2 dimensional problem not a 1 dimensional one. The trick is to delay irreversible decisions as long as possible and get all responsible parties involved.

The flip side of the coin is that reversible decisions should be made as cheaply as possible. For instance my “Kobayashi Maru” solution to the bike shedding problem is that the team should have budgeted three colors of paint into the bike shed budget, picked green for the first coat, and moved on immediately. That $200 is far cheaper than the time and importantly energy that would go into discussing this at a meeting with senior staff members present. If you start thinking about all of the $5000 conversations you have to solve $2000 problems it’ll drive you mad. I almost recommend you don’t look at it, for your own safety. Almost.

This is exactly right, but the irreversible decisions are often horizontal slices of software design that underlie other decisions - thing like authz, IPC, API design etc. If these questions can be identified and resolved up front, the cost of building vertical slices - features - goes way down. You're not tempted to reinvent the wheel, you can focus on the business logic.

What I'm saying is: it's worth spending time planning the foundations of a product. That includes the technology choices, but also the high level logical layout of the whole product. If you approach this foundational stuff like any other project, then if you're doing it right, you'll "get all responsible parties involved", like you should.

But defer the detailed planning of domain logic until you're actually ready to write it, and remain flexible with the plans you do make. "Plans are nothing; planning is everything".

Because like the GP said,

> the later you design something the more info at hand you have to make the design fit better

I find it remarkable how often engineers just leap in and start coding, and worry about the foundations later. Or never. I suspect this is one reason why rewrites are so common. If you start with bad foundations, it's often easier to blow up the whole building and start again.

To be fair, leaping in and starting coding allows you to discover the problem better. The problems start where you get stuck with your initial design and don't adjust/remake it once you explored some ideas, maybe found some new along the way and found out what doesn't fit.

That is of course problematic if you have deadline driven development because rewriting last 2 weeks of code might seem like total waste even if then-better design gonna save you much more even in near feature.

Not rewriting code when design flaws appear will lead to rewriting it eventually later and more expensive.

Of course, all of that depends on how much you know about domain and the problem. Exploring existing solutions and their flaws or having really detailed requirements might be much better approach that doesn't require starting to write code.

Some things you can cut corners and save time.

Some things, when you cut the corners off you also cut your fingers off.

We have all promised that we will “fix it later”. But whatever we do now is what’s done. Fixing it later is a promise whose resolution is often not in our control.

Or as Feynman said, the trick is not to fool yourself, and you’re the easiest person to fool.

Or a personal favorite, unattributed: I know the guy who made this plan [me] and he’s full of shit.

https://news.harvard.edu/gazette/story/2007/04/frank-h-westh...

Not sure what the actual source is.

I tried to bottle it in this document: https://pcblues.com/assets/pdf/approaching_software_projects...

Don’t put a hyperkinetic person in charge of something thinking that you’ll call their bluff.

I'm not sure if Spirit of an Engineer is a book, essay, or some prior set of principles I'm unaware of but if it's not (and even if it is), I tend to disagree here.

The spirit of of an engineer, in my opinion, is to solve problems and problems for people--that's what I and most engineers I've spoken with have been drawn to (ignoring those purely seeking a high paying profession here since were talking about "spirit"). When I was youthful I sought out technical complexity and grandiose solutions (I admittedly still admire it when it's done correctly). None the less, to me, most of it is wasteful and at the end of the day I want to build the most useful solution for people to make their lives easier.

Some problems require high complexity and glass cathedrals in terms of technical solutions, most really don't, at least most of the problems I've been exposed to.

And yes, engineers really do err on the side of designing for future problems and future users that often will never exist. This effect is worst on a rewrite of a system - look up the "second system effect". We tend to only truly get pragmatic tradeoffs the third and later times.

Make a bare bones version and see if it sticks, and go from there. Both the Progressive Enhancement philosophy and the Lean Startup methodology demand the minimal before building further. Ex: Will anyone click this survey link? Who knows but you'll learn real quick if you offer them $50. If they still don't, then it probably ain't gonna get clicked ever. If they do, then you found at least _something_ viable.

Good idea, you are welcome to help me hone my later chapters. For me the engineering spirit - especially in software - is being a creator/builder. At the core, someone who takes the Legos in front of them and creates with those pieces.

A couple people took my "grandest" comment as a nod toward scale/size/narcissism, which is reasonable, and I meant it more in terms of total architectural completeness - i.e. a system perfectly fulfilling all the requirements that necessitated its creation. A blade of grass is not "grand" in the primary sense, but it still is grand + magnificent by its own perfection in design.

The potential need to scale and add features will inform my design decisions, but they will not direct them.

[0] https://www.globaltimes.cn/content/974210.shtml

[0] https://www.globaltimes.cn/content/974210.shtml

https://youtu.be/XopSDJq6w8E

That's responsible, future-oriented development.

i think that's just delusions of grandeur. Not everybody would be (or should be) building the sistine chapel.

I’d vote for SWE if we are taking up acronym namespace

For purer technology / product businesses, how do you do this, fundamentally? How would Google have manually mocked up their early product? How would Facebook? Github? Tesla for that matter?

Sometimes you really do just unavoidably have to build the product out before testing the market, and if it doesn't work, just accept the sunk cost and throw it away - and sometimes fail completely as a result.

I don't see this as a fundamentally solvable inefficiency, just part of how tech product startups work, and the very tradeoff that must be made to allow for innovation to happen.

there are still smaller pieces you can MVP to a smaller audience before launching it to the world.

> Google have manually mocked up their early product? How would

Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.

> Facebook?

They had Myspace as an example so the idea wasn't exactly unproven.

> Github?

Kernel developers were already using the software, successfully, all (for large values of "all") they did was bring it to a wider market with a better UI.

> Tesla for that matter?

People don't get this, but Tesla's biggest achievement to date, isn't the cars themselves, but the factory that they're built in. There's no way to MVP an entire factory, but building a car in a bespoke, pre-assembly fashion is totally possible and totally doesn't scale.

If you're asking if electric cars were known to work, the first one came out in 1832. If you're asking about product-market fit, they keep selling cars they haven't made yet, just to gauge demand. Aka where's my Cybertruck!?

The hundreds of millions of USD as an interest free loan seemed more important than anything else.

> Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.

But that isn't a mock up, it's the real thing but on a smaller dataset. If you're going to do the real thing anyway, why not run it on all the data you can?

After all, the throttling factor to release is in the engine, not in the dataset. If you're going to write the full engine anyway, there's nothing to be saved by limiting it to a subset of the data.

Because more data requires more cleaning and standardization (with more edge cases). It also requires a bigger scale to obtain and process.

You are right that some businesses scale better doing high touch customer service like this. In the case of Pilot, you have the lead sales guy and accounting domain expert (CEO) asking SaaS customers to schedule enterprise service sales calls with him.

Which makes total sense. What he’s not automating is setting up Marketo or Hubspot and Drip or Constant Contact and committing to some CRM system that is both impersonal and adds friction when you’re going for quality over quantity.

He could hire BDRs and CSMs or outsource and spam and set up AI and knowledge bases, and possibly scale up faster.

But not only would that take away the personal touch and competitive advantage, he’d lose the opportunity to educate himself on what real customers really need.

Not to mention all the time spent evaluating tools and setting up automation and negotiating contracts that lock you in to a specific process that might not be what you want 2 years down the line when your business changes.

as an engineer this part I entirely agree with. Far too often CEO driven development leads to pivoting 1Month into 9 month projects, 9 months in a row. such that you have nothing deployed when you at least could have had 1 thing, but instead you have none.

This can be solved as much by discipline to stay on track as by choosing to reduce scope to 1 month ideas only.

Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.

Depends. No one at a unicorn says that, but I've seen plenty of startups that. The unicorn is the outlier, not the rule.

> I worked at one company that we couldn’t rack servers fast enough to keep up with our growth. Our competitor who was the market leader had to turn off the ability for new customers to signup to prevent their site from crashing under load. We ended up surpassing them.

Were they still in business at the end of all this? Because it turned out you made a bet, and it came through. The odds are against this, though.

> Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.

A million in what time period, and what are the usage patterns? Because from what I've seen, as long as the backend isn't in Python, Ruby, PHP or similar, a single hosted server easily vertically scaled to handle more than 2m million users, because it a) didn't need to handle 2m concurrent connections, and b) usage patterns were infrequent enough (users interacted only every few minutes at most).

You should make architectural choices that CAN be scaled within reason, on demand and ideally incrementally. That doesn’t mean you need to build your app on Kubernetes and automate welcome emails, surveys, metrics, A/B testing etc. But you probably shouldn’t build on top of fundamentally non-scalable systems either.

This is provided you’re in an environment that can reasonably scale to 1M users or whatever, which isn’t true for all domains. Say B2B where that last B is only 10000 people world wide.

Unless you built something absolutely atrocious, scaling it vs getting it ‘right’ or ‘right enough’, is trivial. You couldn’t sign up to Facebook if you weren’t a college student.

Chatgpt overloads daily so what? When your app becomes necessary to enough people, bring in your k8s evangelists and shove into x cloud provider. But by god it doesn’t take 30 people to write a marketable mvp.

Chatgpt doesn’t have competitors. Customers can’t go elsewhere. Twitter had the same protection when they launched, there wasn’t a short form social network so they could have full day outages. Friendster would be the opposite, when they closed signups, customers just went to MySpace.

Scale goes beyond the ability to support traffic. It also means efficiently handle data/traffic. Heap is a good example, great product but they haven’t figured out to provide that level of analytics without significant costs which are passed on to the customer.

Everywhere? Sounds like you're cargo culting.

There's a use-case for it, and that use-case is when your load is so high you need the orchestration.

Right now I'm using cluster of 2 nodes (master, worker). No reliability, but convenience is good and we have very simple path to more nodes when we would need it.

I'm not sure why you are getting downvoted. It would be great to hear of some of the bad experiences of those that disagree. Were they running their own cluster instead of a managed one? Or doing k8s with a team where nobody had any experience? Or run into some helm-chart spaghetti from hell?

But ... no one claimed it didn't, did they?

The claim was that the unnecessary complexity (of which Kubernetes was just one) from day 1 consumes valuable time that could be spent on making the product better.

Kubernetes might start consume valuable time if you would start to bring niceties. Like GitOps, CI, and so on. Those things are good but not necessarily justify time spent from the start. So you can start with simple deployment yaml which you can write with 10 minutes and apply it manually whenever you need to update it.

Kubernetes almost out of the box brings to you:

ingress. You don't need to set up your own nginx reverse router. This task alone could consume days of setting up and maintaining. I work with organization which spends 2-3 hours to update their nginx if they need new route. It must be done in off-hours because they break it every single time they touch it and then they need to fix it. Kubernetes ingress is a breeze of fresh air after this nightmare.

cert-manager. Again, I work with organization which failed at setting up certbot. Their certs break all the time. cert-manager just works.

zero-downtime rollout restart. This is absolutely nice thing which works from the box and hard to achieve manually.

kubernetes-dashboard. There're other dashboards as well. I think that lens is very popular although I didn't use it. It allows for people who know very little about kubernetes to interact with it productively. For example in my team we have developers who use kube-dashboard to view logs, to exec into pod and check things, to update deployment, change envs. They just push buttons in GUI, they have no idea about any yamls or kubectl. And it works.

And those things in my experience so far do not bring any unnecessary complexities. Quite the opposite. They solve lots of inevitable complexities for very little cost.

There is however a pitfall that this approach seems to lead to unless you actively prevent it. Roughly the process is 1. You hire someone to take care of things manually. 2. As you grow, your devs are worrying about other things and so instead of automating you hire more and more people to handle the toil that should have been automated along the way. 3. Because these jobs are handled by one group of people and developers are working on customer facing features, the communication and prioritization necessary to fix the issues rarely come up.

Ignoring history is a guaranteed path towards repeating it.

Not that all spikes are low quality, or even that one on HN ones are. I reckon for some products they are.

Edit: (The load was easily handled by a single Heroku dyno. I turned it up to two dynos overnight, just in case, but it turned out to be unnecessary.)