Right now I wouldn't say prime now's inventory integration is any better, but I'm sure it eventually will be. Both instacart and prime now seem to have trouble determining if an item is actually in stock.
Graphcool is awesome. I think you guys made the right choice moving to a code-first, open-source platform.
Obviously you still need to make money. Besides hosting and automatic backups, how will you monetize? What are your plans for Graphcool Enterprise on-premise?
It was awesome to see how Graphcool enabled so many frontend developers to get started with backend development.
And it has been even more exciting to see these same developers move to a combination of Prisma and GraphQL bindings with relative ease.
The GraphQL ecosystem is maturing to a point where the tools are now easy enough to be used by small dev shops and powerful enough to be used by large enterprises. There was an episode on GraphQL Radio (https://graphqlradio.com/) diving into how IBM build their own library to standardise schema composition. This is a fairly advanced concept, but with GraphQL binding this is now part of day-to-day development for thousands of developers.
To address the money part - Enterprise and on-premise is certainly a thing. We get a lot of requests for custom setups. If you are interested to explore this option, please get in touch!
Distributed storage is still a big issue for sure. There are some options, but none are ideal. One option is to map to host and use NFS to share across hosts. Another option is to use something like Convoy or Flocker, which come with their own complexities and limitations. Hopefully more progress is made on this front.
As for the wordpress app and other issues mentioned, it's actually very simple:
This isn't a "production" config, but that wouldn't look that much different. The real beauty is that I found this compose file with a simple search and very easily made minor tweaks (e.g. not publicly exposing the mysql ports).
You might run into permissions issues if you use host mounted volumes, but I have not. Normally I prefer to use named volumes (docker-compose v2) and regularly backup the volumes up to S3 using Convoy or a simple sidecar container with a mysqldump script.
This is interesting. I'd been considering mounting drives for persistence of stateful data from containers.
Let's say I want to run a Wordpress hosting service. In my ideal world, I deploy an "immutable" container for each customer, i.e. everyone gets an identical container with Wordpress, Nginx, MySQL etc. So what to do with state info, like configs and the MySQL data files? I'm thinking of mounting a drive at the same point inside each container e.g. /mnt/data/ and /mnt/config/ or similar.
This way the containers can all be identical at time of deployment, and I can manage the volumes that attach to those mount points using some dedicated tool/process.
This is all still on the drawing-board... but what you've said here seems to suggest this approach should work. Or have I optimistically misinterpreted what you've said? :)
Yes that's a pretty good approach. Just organize the configs in a directory structure on your host and mount them as volumes (along with any other necessary volumes for e.g. uploaded media). There are more advanced methods like using Consul/etcd, but only go that route if you're ready to invest a lot of time and need the benefits.
In your example -- assuming 20 different blogs/customers -- you'd be running 20 separate instances of MySQL (plus 20 nginx instances plus 20 php-fpm instances plus ...)?
Now, let me first say that I haven't come anywhere close to even touching containers and most of what I know about them came from this HN thread so please forgive me if I'm missing something...
I, personally, would rather only have a single MySQL instance -- or, in reality, say, a few of them (for redundancy) -- and just give each customer their own separate database on this single instance.
With regard to containerization, why is all of this duplication (and waste of resources?) seemingly preferred?
In my scenario, I want to provide a package for easy download and deployment. Each customer will indeed run their own mysql db, if they choose to self-host the containerised software.
I plan to offer a paid hosting service, where I'll rent bare metal in a data centre, onto which I'll install management and orchestration tools of my choosing.
An identical container for any environment is my ideal, since this will make maintenance, testing, development etc simpler. Consequently each customer hosted in my data centre will, in effect, get their own mysql instance.
This way the identical software in each container will be dumb, and expect an identical situation wherever it's installed.
Now, in reality, I may do something clever under the hood with all those mysql instances, I just haven't worked out what yet :)
Actually it will probably be Postgres, but I'll use whatever db is most suited.
So yes, some duplication and wasted disk space, but that's a trade off for simplified development, testing, support, debugging, backups, etc.
In this case, a single mysql instance with individual databases may indeed be the best approach. It'd be very easy to launch a mysql container and have each wordpress container talk to it. I use Rancher for orchestration, and it automatically assigns an addressable hostname to each container/service, so I'd just pass that to each wordpress container. Or you could expose a port and use a normal IP + port.
The duplication is preferred because you can take that stack and launch it on any machine with Docker with one command. Database and all. Usually that's great, but it'd be very inefficient in this case.
Docker gives you the building blocks, but that means you have more pieces to arrange and manage. Take a look at Docker Compose if you haven't already, since the Docker CLI only gets you so far when you're creating apps that consist of multiple containers.
I think the best approach for your cert issue is to abstract that into a separate service (nginx is an option, but I'd recommend the Rancher approach below). Yes, that means you have to add another container, but that's just another block in your docker-compose.yml file. Embrace the approach of separating your components into containers and organizing them as a stack. You can easily link containers together, share data volumes, and start/stop individual containers or the stack as a whole.
The problems that you're having are pretty easy to fix with some tooling. Rancher (http://rancher.com/) greatly simplifies the cert issue by allowing you to import certs and provide them to the Rancher loadbalancer service (which you can add to any stack). There's also a LetsEncrypt community catalog template that automatically retrieves and imports certificates to Rancher. There are other open source orchestrators like DCOS, but Rancher is probably the simplest to use, and it's the only one I'm very familiar with. There are SaaS options that you can look into, but I don't have experience with them.
As for the container crashes, it's trivial to automatically restart them. Just pass the --restart=always flag to the Docker run command. You can also add the flag to a docker-compose.yml file.
> wait. aren't these things supposed to give us less pieces to arrange and manage?
No, not fewer pieces. You'll have more pieces, but you can combine the pieces and control them individually or as a group. You can think of Legos, since you have many pieces but they all fit together in the same way.
Docker compose lets you group these containers together, and that is what ultimately makes it feel like you have fewer pieces to manage. With that, you can start/stop a stack of containers (e.g. django, nginx, postgres, redis) with a single command, but still inspect and manage each component separately. This is something you might normally do with bash scripts, but with Docker you can take that same app to an orchestration platform and run it on any host. Run it on your laptop, run it on a linux server, run it on a SaaS provider like Docker Cloud, run it on a private cloud with an orchestration platform.
> yes, of course the solution is more tools. what exactly was the problem again?
Docker is just the foundation. Nothing more. I'm fine with learning more tools because I feel that the foundation is solid. The problem is being able to ship and manage your apps. That is much, much easier for me now and I'm very glad I invested the time.
Seriously?
If something crashes continuously in production then the solution is "just pass the --restart=always flag.
I really wonder if you guys are really using docker in prod.
I would never use something like that to manage important transactions.
It's not one or the other. Restarting might save you some downtime if you're running a single IRC container. That doesn't mean you shouldn't find and resolve the root issue. Normally I just rollback to the previous version container if I have a recurring issue.
Anyway, don't use containers if you don't want to. I'm glad I invested the time since I understand them very well and use them to my advantage. But I did have to learn a lot and experiment with a bunch of tools, and maybe that's not worth it to you.
As someone said, this is the normal behavior in the Erlang/Elixir world, and it seems to have worked extremely well for the telecom industry.
That said, my reply was to someone running an IRC server, presumably on a single server, so don't stretch my advice to a production app handling millions of transactions. Obviously the core issue is that the app crashes, and it's still up to him to fix that. This is almost certainly a problem with his app/config, not Docker itself (though it ain't perfect). If it's something that happens every 6 months, then auto restarting will probably save him a lot of problems. If your transactions are so precious, don't pass the flag- it's up to you.
I never used Erlang, so I have no idea if it follows the same approach(although I find it quite strange).
But for sure I can't afford to deploy anything like that in prod.
You lose one transaction in the middle and several millions go lost.
I'd rather lose an hand than try to explain my clients that it is fine, docker just restarted by itself as expected..
Thanks, I'll have to look into Rancher. I'm already using Docker-compose.
Also good to know about automatic restarts. I don't know how I missed that, I had to read a lot of docs to get where I am.
Still, the amount of tooling that exists and the knowledge needed to pick the right ones for a given situation goes to show that this isn't as simple as packing a shipping container and letting someone ship it...
You're right, it's not quite that easy yet, but it probably will be eventually. Docker just provides the building blocks. SaaS providers like Docker Cloud will get better and continue to abstract complexity away until it really is that easy.
You don't need to use Rancher if you're just running one app. If that's all you need to do, then it could be as simple as running docker-compose on a linux server and mounting the certs into the nginx container as a host volume (https://github.com/jwilder/nginx-proxy). This is a fine approach until you want to split your containers across several hosts (redundancy or scaling) and you have several apps to worry about.
Hey chewchew you seem to be knowledgeable on docker-compose.
What's the easiest way to get a .yml on a cloud server somewhere and let it assemble it assemble the containers for you?
Also, do you know where containers set environment variables? The official Postgres makes available something like PG_PORT_3542 and u can just refer to it from another container. Where I can't seem to do the same with the Redis one...
The absolute easiest is probably Docker Cloud. I've checked it out but haven't really used it much. It's still relatively immature but if you just want to deploy and forget something simple, this is probably the way to go.
If you want to use your own Linux host, then the simplest way would probably just be to SSH into the box, git pull, and run "docker-compose build && docker-compose up".
Setting environment variables is pretty easy, but I don't think you need them in this case. If you're trying to make a redis or postgres container available to another container (your app), then you can do so easily with links in docker-compose. Something like:
You can then access redis from the myapp service using the hostname "redis" and the default port "6379". So, "telnet redis 6379" would work from the myapp container (assuming telnet is actually installed). The redis port isn't even publicly exposed- it's only available to myapp.
If you need to define environment variables, you can do so with an environment dict as shown above. There are a few other ways to define env vars:
It's up to you to make your containers bloated or keep them slim. You can use the alpine versions of the official Dockerhub images. Python on Alpine is 30 MB (vs 267 MB for the debian one). https://hub.docker.com/r/library/python/tags/
If your app is a compiled go binary (so, it runs anywhere) why do you need a container?
The whole point of containerisation is to group installed dependencies (as opposed to installable dependencies like with a regular deb or rpm package) and configuration into a 'black box'.
If your binary is already a single-file distribution, why lump it in with the crapfest that is docker?
I don't think that's the whole point. If all you care about is packaging your code in a container, then you don't need Docker. That was solved long ago. Docker simply adds a nice API on top which allows you to package, ship, and manage your apps in the same exact way.
By using something like Docker Compose, you can explicitly define each container and the relationships between the containers. Then, you can start/stop groups of containers (an application stack) while still retaining some component isolation and the ability to upgrade and scale containers independently. All of this is defined in a relatively simple YAML file, which can be committed to VCS and tweaked. I can't tell you how awesome it is to find repos on Github that have use docker-compose. Even complex apps with many pieces can often be launched with a single command. It's easy to take these stacks and tweak them to your needs, adding/removing/changing components as necessary.
Since Docker provides a standard way for managing any container, orchestrators like Swarm, Rancher (my preference for small-medium clusters), and DC/OS can take that functionality and scale it across many hosts. You can get a birds-eye view of your Docker cluster (all apps) or drill down into individual apps and their components. Each container is a uniform piece and can be controlled, scaled, updated, audited in the same way. Throw a UI in front of it and now you can manage applications that you know nothing about. That's great for developers that manage just a few apps, and it's incredible for enterprises with thousands of them. If you don't want to think about infrastructure at all, then you can use a SaaS Docker provider. Obviously there are pros and cons to each approach, and there are some remaining challenges.
Docker isn't perfect, especially in regards to storage and networking. Distributed storage isn't simple, but a lot of progress has been made with volumes and volume drivers. It's not as easy as it needs to be, but the general direction seems to be good and with the proper tooling I think this will be less of an issue.
If "you" here is one person and "your app" is one app, then yes, why do you need a container. If "you" is 200 developers, and "your app" is 75 different applications written in Java, Java plus native libs, .NET, python, the other python, R, scala, NodeJS, and various C libraries, then docker containers, and more importantly images, are about as ideal as it gets. We're running Mesos so we don't have to use docker to get containers, but packaging up the products as images is a significant advantage.
Alpine containers are nice if you're just looking at size. But they break down once you `docker exec` into them to try to debug something:
$ docker exec -ti mycontainer /bin/bash
stat /bin/bash: no such file or directory
$ docker exec -ti mycontainer /bin/sh
/ # curl https://localhost:5000/
/bin/sh: curl: not found
/ # strace $command
/bin/sh: strace: not found
Thats not really broken. It is just a minimal image. If you want more pacakages then install them via the Dockerfile before you fire up the image. I like starting with a bare container. For example I might take an alpine image, install openvpn and use it to serve an always on vpn connection to other containers. The VPN container doesnt need a shell or anything else really. The only thing I want to see when I attach to the container is what openvpn is spitting out.
Well yeah, "break down" was probably too much of a term. I did not mean to diss Alpine; I just meant to point out that the downsides of minimal images should be carefully considered.
