Some of this I agree with, even if the overall is to me asking for "magic" (I wouldn't mind trying to get there though).
For me the one I really want is "serverless" SQL databases. On every cloud platform at the moment, whatever cloud SQL thing they're offering is really obviously MySQL or Postgres on some VMs under the hood. The provisioning time is the same, the way it works is the same, the outage windows are the same.
But why is any of that still the case?
We've had the concept of shared hosting of SQL DBs for years, which is definitely more efficient resource wise, why have we failed to sufficiently hide the abstraction behind appropriate resource scheduling?
Basically, why isn't there just a Postgres-protocol socket I connect to as a user that will make tables on demand for me? No upfront sizing or provisioning, just bill me for what I'm using at some rate which covers your costs and colocate/shared host/whatever it onto the type of server hardware which can respond to that demand.
This feels like a problem Google in particular should be able to address somehow.
> Basically, why isn't there just a Postgres-protocol socket I connect to as a user that will make tables on demand for me? No upfront sizing or provisioning, just bill me for what I'm using at some rate which covers your costs and colocate/shared host/whatever it onto the type of server hardware which can respond to that demand.
How isn’t RDS Aurora Serverless from AWS, available in both MySQL-and Postgres-compatible flavors, exactly what you are looking for?
Yes and no. Aurora Serverless isn't really "serverless", it's PG or MySQL binary compatible wire protocol with AWS custom storage engine underneath.
Provisioning of the compute part of the database is still at the level of "ACU" which essentially map to the equivalent underlying EC2s. The scale up/down of serverless V1 is clunky and when we tested, there was visible pauses in handling transactions in progress when a scaling event occurred.
There is a "V2" of serverless in beta that is much closer to "seamless" scaling, I assume using things like Nitro and Firecracker under the covers to provision compute in a much more granular way.
All cloud providers offer exactly what you are describing, just not for SQL DBs.
In my experience shared servers for MySQL (and I assume Postgres) never work because I always have to set at least some custom configurations very early into the development cycle.
I know this perspective is not relevant for most companies: But for side-projects and a-like the cold-start time for Aurora is like 15 - 30 seconds, and the first query always times out.
Having it always on (if just 1 "compute") will cost you 30 USD a month. I'm hoping for Aurora to eventually be closer to DynamoDB pricing and startup (I'm fine with 5 - 10 second cold start as long as it doesn't time out the first request everytime.)
For me the one I really want is "serverless" SQL databases. On every cloud platform at the moment, whatever cloud SQL thing they're offering is really obviously MySQL or Postgres on some VMs under the hood. The provisioning time is the same, the way it works is the same, the outage windows are the same.
But why is any of that still the case?
We've had the concept of shared hosting of SQL DBs for years, which is definitely more efficient resource wise, why have we failed to sufficiently hide the abstraction behind appropriate resource scheduling?
Basically, why isn't there just a Postgres-protocol socket I connect to as a user that will make tables on demand for me? No upfront sizing or provisioning, just bill me for what I'm using at some rate which covers your costs and colocate/shared host/whatever it onto the type of server hardware which can respond to that demand.
This feels like a problem Google in particular should be able to address somehow.