So the problem is 1) port mapping and 2) backing up data volumes?
There are simple solutions to each
1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for details see https://docs.docker.com/compose/how-tos/multiple-compose-fil...
2) this is literally as simple as running a small bash script that creates a backup, tars and gzips the file and then uploads it to an S3 repository. Sure, maybe not out of the box, but generally pretty simple. Likewise i'd always have a script that does the inverse (downloads the latest backup, loads it into the db and restores the files).
For 1, docker has a handy special file to make this even easier! `compose.override.yaml` loads automatically on top of the main `compose.yaml`. We keep dev configs in there so Devs only need to run `docker compose up` locally without any environment variables, but on prod (which is automated) we set the compose file to be `compose.yaml:compose.prod.yaml`.
I do it the other way round, so in production you just have to use `docker compose up`. I did not understand the `compose.yaml:compose.prod.yaml` syntax or never saw it. What does this do?
Oh if you set the environment variable COMPOSE_FILE, you can specify a colon-separated list of compose files, which get merged together. This lets us have our core services in our main compose.yaml, since they're shared between dev and production, and then have extra services that are prod-only, or toggle certain prod-only environment variables, inside the compose.prod.yaml . And extra services/settings which we only want in the dev environment go in compose.override.yaml .
Eg
COMPOSE_FILE="compose.yaml:compose.prod.yaml" docker compose up
Yup, I was surprised how weak the arguments were. I mean, surely there are better reasons why docker compose fails to scale? I have been pretty happy with it right now, and haven't felt the need to try something like k8s or docker swarm.
It is a skill issue, but not a problem. The developer who just wants to write unrelated code doesn't need Docker to write the code that goes into Docker.
For a lot of developers who have their primary focus set on writing code and documentation, trying to remember Docker or Kube on top of the rest of your stack is burdensome. Deployment is an end-user task. Obviously the development environment fits the target use-case already.
So please understand that there are lots of people who are building the things that go inside your containers who don't want, need, or see the value of these containers. For you they might offer value, but for someone like me they bring cognitive load and troubleshooting.
Too much abstraction can be just as bad as not enough. Like Akin's Laws of Spacecraft Design, but for software.
(Not parent) The quickest way I could do it is if I just used my existing wireguard setup to directly access the docker container IP. In that case it's just one compose up away even without mapping/exposing ports.
Sounds like your container has some kind of side-car that makes it directly addressable over Wireguard without needing to address the host IP. Does that mean you'd need to modify the docker-compose in some way before `docker-compose up`?
How do you know which port Paperless is using for HTTP?
When you want to load up Paperless in a web browser, are you typing in a service name, or the container IP address? If it's a service name, how are you doing DNS? Do you have TLS?
I can already directly access the docker network because it's in the allowedips setting of wireguard. But for convenience, yes, when I do get a docker-compose.yml file I change 127.0.0.1 IPs to that of the wireguard interface IP. (I can leave 0.0.0.0 IPs as is because this host does not have a public IP) This way I am exposing the ports, but I am exposing them in a way that they are not world-accessible but still accessible to me conveniently.
For services open to the public internet I just add a subdomain + a reverse proxy entry into an existing caddy instance and point to the docker IP.
> Sounds like your container has some kind of side-car that makes it directly addressable over Wireguard
Not necessary. You can access the deployed docker container without exposing any ports or having any reverse proxy (what you've likely thought about with sidecar, which is a k8s concept, not docker) or anything else by using the ipadress of the started container and the ports the started service used. This is usually only possible from localhost, but wireguard can be configured as what's essentially a bastion host and exit node, this would let connecting clients also address containers that were started on that server, without opening any ports.
You can technically also do that without wireguard even, as long as you configure the docker host to route relevant traffic into the docker ethernet and define the docker subnet as a static route that points to the docker host, but that's another story
It took me a moment to put this together too, so to be clearer - the wireguard endpoint is in docker, so you're adding the docker bridge to your vpn. So DNS is handled by docker, just as containers can already address each other by name - you're on the container network with them.
I don't actually do this. I either access the services by IP or add a private IP to dns. (I think this is not widely supported but cloudflare does support it.)
Your explanation is interesting though. Would that actually work?
You can solve anything with enough scripts, but that in no way invalidates his point that the abstraction level seems a bit off.
I'm not sure that his solution is really an improvement, however, because then you have to bake in the semantics for a fixed set of product types, and when that list gets extended, you have to bake-in a new one.
Not only that he wants to re-invent messaging and daemons, essentially.
There are pre-existing paradigms for service brokers, message passing, etc., they all try to solve this problem, and they all share this same fault, they become in-extensible...
Which is the core programming tradeoff, flexibility vs robust constrained behavior.
On the extreme end of flexibility you have AI goop you barely understand, but arguably it is far more flexible than anything you ever wrote, even if you can't make any guarantees about what it does, on the other hand you have these declarative things that work fantastic if that's what you wanted, but probably not worth getting into computers for...
This is a really good conceptual model, the tradeoff between flexibility and constrained declarative frameworks. The goals is to make self-hosting applications extremely easy and extremely reliable. With that as a goal, being highly constrained seems like the way to go.
This feels like quite a shallow reading. Being able to manage system-wide things (DNS, and a centralised Caddy, centralised database) from individual compose units in such a way that you're not going through a ton of setup each service. This much might just need Caddy to work better with other instances of itself but it isn't a solved problem.
I'll grant you that backing up volumes isn't hard.
I think 2) actually has the most merit. If you have a large number of e.g. temp files a container creates, retaining that file history may not be particularly valuable.
Some sort of "hey back this data up, it's important to my function standard would be nice, and not just for container apps.
Trying to figure out where your app hid its configuration and save files is a perennial user frustration.
I assume you're saying this as a container image author, not as someone who is deploying containers. It'd be great if every other container image author standardized on something like this. We just need someone to create the standard and some tools to make the standard a well-paved path.
As someone who is self-hosting with a docker on my own server, I don't see the negatives mentioned in this article as being a problem at all. Quite the opposite, it gives you the freedom to do your docker setup how you want it.
It took me some time initially to figure things out and I had to learn some things. But now it's a breeze. Once I had reverse proxy and automatic certificate renewal in place it has been working ever since then without me having to do anything. Adding a new service like Immisch or Jellyfin takes me an hour or less. Which can be quicker, but I adjust every docker compose to my setup and make it more secure. E.g. I create a new non-root user for each service. Basically I have the whole setup figured out; I have notes and checklists for the new services I add. I don't need to figure out things anymore and in 95% of the cases things just work.
Updating existing services takes minutes: just increment the version in the compose file and rebuild.
For my setup, I use macvlan as opposed to the default docker bridge network. So, the `ports: - "8000:5000"` docker compose config is being ignored. Instead, each docker container gets its own unique IP and MAC address I assign to it. It can then use any port it wants. Thanks to this, on my home network, I can access any container I want by IP. And any container can access any other container. I then add some restrictions for security purposes on the host machine using nftables.
For reverse proxy, I use Nginx Proxy Manager which is a simplified GUI around Nginx. I'm slowly moving to just using Nginx.
For the database, I run multiple instances of Postgres and MySQL and I don't see any issues. They take an insignificant amount of resources compared to the applications. If the application is not in use, its database instance is not being used as well.
I've always understood docker-compose to be a development or personal tool, not for production. Is that not usually the case?
Also aside, "docker compose" (V2) is different from "docker-compose" (V1) [0], it was rewritten and integrated into docker as a plugin. It should still be able to handle most old compose files, but there were some changes.
For many people self-hosting implies a personal server; it's not really "development" but it's not "production" either. In that context many people find k8s or other PaaS to be too heavyweight so Docker Compose is pretty popular. For more production-oriented self-hosting there are various newer tools like Kamal but it will take a while for them to catch up.
I've managed to keep my personal server to just docker, no compose. None of the services I run (Jellyfin, a Minecraft server, prowlarr, a Samba server [this is so much easier to configure for basic use cases via Docker than the usual way], pihole) need to talk to one another, so I initialize each separately with shell scripts. Run the script for a new service once, and Docker takes care of restarts on system reboot or if the container crashes, don't even have to interact with my base OS's init system or really care about anything on it. When I want to upgrade a service, I destroy the container, edit the script to specify the newer version I want, and run the script again. Easy.
On my personal home server I abuse Docker Compose so I don't have to use a huge command line to spin up/spin down containers but I can just say "docker compose up -d" and be done with it.
> For many people self-hosting implies a personal server; it's not really "development" but it's not "production" either.
There's Docker swarm mode for that. It supports clustering too.
It's nuts how people look at a developer tool designed to quickly launch a preconfigured set of containers and think it's reasonable to use it to launch production services.
It's even more baffling how anyone looks at a container orchestration tool and complains it doesn't backup the database they just rolled out.
> In that context many people find k8s or other PaaS to be too heavyweight so Docker Compose is pretty popular.
...and proceed to put pressure to shitify it by arguing it should do database backups, something that even Kubernetes stays clear from.
The blogger doesn't even seem to have done any research whatsoever on reverse proxies. If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about. He would also have researchdd what it means to support TLS and how this is not a container orchestration responsibility.
Quite bluntly, this blog post reads as if it was written by someone who researched nothing on the topic and decided instead to jump to
I'm curious how that last sentence was going to end.
Let's say I agree with you and that TLS termination is not a container orchestration responsibility. Where does the responsibility of container orchestration start and TLS termination end? Many applications need to create URLs that point to themselves so they have to have a notion of the domain they are being served under. There has to be a mapping between whatever load-balancer or reverse proxy you're using and the internal address of the application container. You'll likely need service discovery inside the orchestration system, so you could put TLS termination inside it as well and leverage the same mechanisms for routing traffic. It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
In the end we all build systems to do things to make people's lives better. I happen to think that separating out backups and managing ports as an exercise for the deployment team raises the barrier to people that could be hosting their own services.
I could be totally wrong. This may be a terrible idea. But I think it'll be interesting to try.
> If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about
I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes cluster, and later on a stack of Odroids using k3s. This was years ago, so it may have changed a lot since then, but it seemed at the time that I needed an advanced degree in container orchestration studies to properly configure it. It felt like Kubernetes was designed to solve problems you only get above 100 nodes, then k3s tried to bang that into a shape small enough to fit in a home lab, but couldn't reduce the cognitive load on the operator because it was using the same conceptual primitives and APIs. Traefik, reasonably, can't hide that level of complexity, and so was extremely hard to configure.
I'm impressed at both what Kubernetes and k3s have done. I think no home lab should run it unless you have an express goal to learn how to run Kubernetes. If Traefik is as it was years ago, deeply tied to that level of complexity, then I think small deployments can do better. Maybe Caddy is a superior solution, but I haven't tried to deploy it myself.
If you want an HTTPS ingress controller that's simple, opinionated, but still flexible enough to handle most use cases, I've enjoyed this one:
https://github.com/SteveLTN/https-portal
> Let's say I agree with you and that TLS termination is not a container orchestration responsibility.
It isn't. It's not a problem, either. That's my point: your comments were in the "not even wrong" field.
> (...) It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
No. My point is that you should invest some time into learning the basics of deploying a service, review your requirements, and them take a moment to realize that they are all solved problems, specially in containerized applications.
> I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes (...)
I recommend you read up on Traefik. None of your scenarios you mentioned are relevant to the discussion.
The whole point of bringing up Traefik is that it's main selling point is that it provides support fo route configuration through container tags. It's the flagship feature of Traefik. That's the main reason why people use it.
Your non sequitur on Traefik and Kubernetes also suggests you're talking about things that haven't really clicked with you. Traefik can indeed be used as an ingress controller in Kubernetes, but once deployed you do not interact with it. You just define Kubernetes services, and that's it. You do interact directly with Traefik if you use it as an ingress controller in Docker swarm mode or even docker-compose, which makes your remark even more baffling.
> I'm impressed at both what Kubernetes and k3s have done. (...) If Traefik is as it was years ago,(...)
Kubernetes represents the interface, as well as the reference implementation. k3s is just another Kubernetes distribution. Traefik is a reverse proxy/load balancer used as an ingress controller in container orchestration systems such as Kubernetes or Docker swarm. The "level of complexity" is tagging a container.
Frankly, your comment sounds like you tried to play buzzword bingo without having a clue whether the buzzwords would fit together. If anything, you just validated my previous comment.
My advise: invest some time reading on the topic to go through the basics before you feel you need to write a blog post about it.
it's perfectly fine if everything can run on a single server instance. which is probably the majority of things.
i've run production instances with 2PB of data being scraped per month by 100+ CPUs and 256GB of RAM using docker compose. some of the machines (smaller instances) have run flawlessly with zero reboots for years on end. both on cloud and on-prem.
+1 to this. Smaller datapoint from my side I guess, but anyway, docker is the core of my self hosting setup and it is one of the things that I don't have to fiddle with, it just works.
That's fine. Some people also manually launch containers individually with Docker as their production system.
They'd be wasting their time and making their lives needlessly harder, though.
They already have tools that do all the legwork for them. Why not do the research and opt to force a square peg into a round hole?
Research Docker swarm mode, reverse proxies like Traefik, and Let's Encrypt. Kubernetes can be even easier with implementations like microk8s, which are a snap away from any Ubuntu installation. Look at what you're doing and figure out the right tool for the job. Don't just whine about how you're doing something wrong and it's the tool that needs to change to fix your own mistakes.
Not sure what axe you have to grind, but the person you replied to didn't whine about anything as far as I can see. I've also been hosting a lot of stuff on docker-compose and it is perfectly fine for a lot of things. No one said it's perfect or does everything.
>They'd be wasting their time and making their lives needlessly harder, though.
Using Kubernetes where it's not needed is just that - wasting your time and making your life harder.
Before you say that I just need to research more: I know Docker swarm mode, I run my personal server on Kubernetes using Traefik and Let's Encrypt, I professionally work with Kubernetes (both as an admin and working on Kubernetes security, which is tough to get right), most services in my dayjob run on Kubernetes, and I was the person who introduced CI/CD pipelines there some years ago.
I still claim that there are production usecases that are better served by docker-compose.
> Using Kubernetes where it's not needed is just that - wasting your time and making your life harder.
I think this is a very ignorant and misguided take.
Kubernetes is a container orchestration system, just like Docker swarm mode or even Docker compose. If you need to deploy sets of containerized apps into your own VMs, you can pick up any Kubernetes implementation. You don't even need Helm or anything. Plain old kustomize scripts will do. Some aren't even longer than a docker-compose.yml.
More to the point, Kubernetes is an interface. One that you can use in local deployment and in major cloud providers.
You should really check your notes because your comments contrast with the realities of actually running a service.
> Before you say that I just need to research more: (...)
All your appeals to authority are falsified by your claims.
I, on the other hand, actually use Kubernetes both professionally and in personal projects, as well as Docker swarm mode, and can tell you I'm no uncertain terms that none of your points have any traction in reality.
> I still claim that there are production usecases that are better served by docker-compose.
I'm sorry, but your comments simply sound deeply uninformed and misguided.
I mean, it makes absolutely no sense to comment on using docker compose in production when Docker swarm mode is far more capable and Docker swarm mode stacks already share most of the schema with docker compose. You literally have virtually nothing to do to adapt a docker-compose script to launch a stack.
Most people dont need rolling release, 24/7 availability, auto scaling, etc.. on their home server, so managing k8s just add way more complexity. My main reason to not use is because I would need to host a artifact service somewhere else which is PITA. Some k8s runtime support local building but is not as easy as compose in my experience.
> Most people dont need rolling release, 24/7 availability, auto scaling, etc..
That's perfectly fine. That's not the reason why you are better off running your apps in Kubernetes though.
You are better off running your apps in Kubernetes because it handles everything you ever need to effortlessly run containerized apps. You don't even need to install tooling or deployment tools or anything at all. You can have a single kustomize script that defines your services and deployments, have in place an ingress and Let's Encrypt, and you're done. You don't need to bother with anything else. Just run kubectl apply and go grab a coffee.
Hey that is not true, at least last time I tried I spent considerable time trying to integrate a bare metal load balancer from a 9 stars github repo plugin because apparently exposing the port is not recommended. Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
One last point, the runtime ram and cpu overhead is far from minimal.
> Hey that is not true, at least last time I tried I spent considerable time trying to integrate a bare metal load balancer from a 9 stars github repo plugin because apparently exposing the port is not recommended.
You sound like you tried to put together an ad-hoc ingress controller without knowing what an ingress controller is and why abusing NodePort is a mistake in the context of a cluster.
> Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
You should look at the problem to see if an approach makes sense. If you're using vanilla Kubernetes on a one-box deployment then you're abusing a large-scale high-scalable cluster management system by shoving it into a single box. It can work, but most of the resources will be wasted on managing your cluster of one node.
There are plenty of Kubernetes distributions that are designed to handle well small clusters, and even one-box systems. Minikube, microk8s, and even k3s come to mind. I'm partial towards microk8s because installing it is just a matter of installing a single package from the distro's official repository.
I've used it for both.... but for production only in extremely constrained/limited environments. Its every bit as stable as anything else... you can version the compose you deploy so rollbacks are easy, etc etc.
far more useful for development IMO, but when push comes to shove and you need a few things running together on a machine in the middle of nowhere with limited/unreliable internet access.... you can't do much better... a few volume mounts for durability and you're really in decent shape.
logs, networking, system health etc etc.... just all a few docker and docker-compose commands away
Its not going to be amazingly professional and 100% best practice, but you can set up docker-composes and/or design containers to pull everything they need on first run.
That plus a decent backup system would work for a small array of servers with fail-safes.
Though I would die inside if a production user-focused app under any level of proper load was set up like that.
You'll need more than a backup system. At least some sort of a load balancer to switch between different groups of running docker containers (so that upgrades, backups, etc... can happen without service being interrupted).
not every single system requires 100% uptime... some folks still use release windows for specific applications/hardware. I'd argue that most systems out there can support a release window.... and the overlap between "docker-compose works for us" and "we have release windows" is probably quite high.
We needed to do inference on remote machines stationed at random points across North America.... the machines had work to do depending on external factors that were rigidly scheduled. really easy to make upgrades, and the machines were _very_ beefy so docker-compose gave us everything we needed.... and since we shipped the data off to a central point regularly (and the data had a short shelf life of usability) we could wipe the machines and do it all again with almost no consequences.
I needed to coordinate a few small services and it did the trick beautifully
My experience has been that the vast majority of systems could tolerate a few minutes offline per month for upgrades. Many could tolerate a couple hours per month. No or negligible actual business harm done, enormous cost savings and higher development velocity from not adding the complexity needed for ultra-high uptime and zero-downtime upgrades.
What's vital is being able to roll back a recent update, recover from backups, and deploy from scratch, all quickly. Those are usually (not always) far easier to achieve than ultra-high-uptime architecture (which also needs those things, but makes them all more complicated) and can be simple enough that they can be operated purely over ssh with a handful of ordinary shell commands documented in runbooks, but skipping them is how you cheap out in a bad way and end up with a system down for multiple days, or one that you're afraid to modify or update.
Early this year I consulted for a company that used docker-compose as their software deployment mechanism. They created new compute, but compose gave them the sidecars you’d want from k8s without all the other overhead that came with.
To go even further, podman supports actual sidecars (one or many forming a pod) and even supports actual Kubernetes deployment manifests without needing a kube-apiserver.
According to docker: "Compose has traditionally been focused on development and testing workflows, but with each release we're making progress on more production-oriented features."
Why not? I can't imagine you'd deploy an application without some form of service management (even if it's just throwing it in tmux) and unless you've gone out of your way to use a non systemd distro systemd is builtin and works for both user and root containers.
Most places (though not all) that I've seen using docker or docker-compose are throwing them in systemd units anyway.
I was more questioning why a .container file would work for system services versus application services, since basically all those same problems occur for system services too.
Either way this type of argument just comes down to "should I cluster or not" but to think out loud for a bit that's just basic HA planning: Simplest solution is keepalived/etc for stateful services, standard load balancing for stateless. Don't want load balanced services running all the time? Socket activation. Don't have a separate machine? Auto-restart the service and you can't cluster anyway. The only thing you'd really have to script is migrating application data over if you're not already using a shared storage solution, but I'm not sure there's any easier solutions in Kubernetes
Not having to install and manage Kubernetes? Unless you're paying someone else to run it for you (in which case this entire conversation is sort of moot as that's way out of scope for comparison) that stuff is all still running somewhere and you have to configure it. e.g. even in small-scale setups like k3s you have to set up shared storage or kube-vip yourself for true high availability. It's not some magic bullet for getting out of all operational planning.
Also even in separate components it's not really "all of that". Assume an example setup where the application is a container with a volume on an NFS share for state: on a given node we'd need to install podman and keepalived, a .container and .volume file, and the keepalived conf. An average keepalived conf file is probably 20ish lines long (including notification settings for failover, so drop like 8 lines if you don't care about that or monitor externally) and looking at an application I have deployed a similar .container file is 24 lines (including whitespace and normal systemd unit file boilerplate) and the NFS .volume file is 5 lines. So ballpark 50 lines of config, if you or others wanted to compare configuration complexity.
Also, fun fact, you could still even use that Kubernetes manifest. Podman accepts .kube files to manage resources against itself or a Kubernetes cluster; I've been recommending it as sort of a middle ground between going all in on k8s versus transitioning slowly from a non-clustered deployment.
You can always use something like Docker Swarm or Nomad which achieves the same end result as Kubernetes (clustered container applications) without the complexity of having to manage Kubernetes.
Just spawn another VPS with your application and connect to load balancer. Even better - use Fedora CoreOS with Butane config and make that VPS immutable.
We run large, auto scaling clusters with compose and a small orchestration thing we have been using since the 90s (written in perl) (before compose we had our own compose-like with chroot). No issues. For decades.
Cool - everyone knows it’s possible to hack together something, or anything else. But Is it a good idea to spend time and resources starting to do that now if you haven’t been doing it with a sketchy Perl script since the 90s? Not really.
You do you. I think k8s is overly complicated stuff that almost no companies (outside google/meta/etc) need and some peopple here peddle for some reason. That's fine. We like making profit instead of waste time with the latest nonsense.
I don't think it's a good idea to spend time and resources learning complexity you never need. You are probably going to say that's not true as your cloud hoster has this all set up, but yeah, that's where the making profits comes in. We save millions/year on not cloud hosting. And managing yourown k8s cluster apparently is quite hard (at least that's what even the fanboiz here say).
Starting today, I would use Go probably instead of Perl, but I would do the same. It's much simpler than kubernetes. But sure; for our goals; we like simplicity and don't need resumes to drive.
I migrated several services from Kubernetes to compose and couldn't be happier. K8s was a maintenance nightmare, constantly breaking in ways that were difficult to track down and unclear how to fix once you did. Compose configs are simple and readable with all the flexibility I need and I've literally never had an issue caused by the platform itself.
That's what I thought the first day, the first month and maybe even the first half year.
Now it's 5 years+ running, all fine. No major hassle (crossing fingers!), no configuration hell. Never touched k8s, and not going to in the near future.
Maybe it just works and is sufficient for my case.
Sure. But that’s not the point of the article. The articles point is “Docker Compose is too complicated”, then proposes a half baked implementation of a solution that hand waves away of all that complexity with a scenario that will fulfill their specific use case but will completely fall apart when it diverges from the specific way they’ve decided to build the abstraction layer.
The problem that the author refuses to accept is that deployment is inherently complex. It does require configuration of a lot of things because it supports every use case. So you either have a generic complex tool that everyone can use (like Docker Compose or Kube) or you have some specific tool that only works for a tiny subset of all users that is simpler that satisfies your use case.
Note that I’m not saying Docker Compose is perfect. The syntax is a bit arcane it’s complex to understand etc. But removing the complexity by removing configuration options is not the solution here. Instead the author should focus on different approaches and tools to manage the same existing level of abstraction.
And for what it’s worth, that’s essentially what Helm is for kube - a way to manage and hide away the complexity of kube manifests (but still use those manifests under the hood). But frankly, docker compose doesn’t need a helm. Because docker compose, as you point out, has value not as a deployment tool, but as a single file that developers can manage and spin up on their local machines in a manageable way that doesn’t have the author fighting YAML all day.
I would say if the author was actually interested in solving this problem in a productive way they should first try to see if docker itself is amenable to altering their constructs to provide optional higher abstractions over common concepts via the compose interface natively. If the source tools roll out those abstractions everyone will get them and adopt them.
> I would say if the author was actually interested in solving this problem in a productive way they should first try to see if docker itself is amenable to altering their constructs to provide optional higher abstractions over common concepts via the compose interface natively.
docker-compose is a lot of things to a lot of people. When it was created I doubt anyone realized it would eventually be the de facto standard for deploying to homelabs. It's an amazing tool, but it could be better for that specific use. I don't think that segment is important enough to the team that maintains it to warrant the change you're suggesting.
docker compose doesn't need to be overly complex though; I think if it starts to feel that way you're likely doing it wrong(tm). K8s IS very complex, and likely overkill but exactly what you should do if you need it in production. This was a very long ad with an unconvincing argument for a product that addresses the wrong problem.
So you took the time to write a commentary about how the parent wasn't worthy of your counterpoints, but no actual rebuttal. That sort of low-effort / high opinion / no value / zero impact comment (and this post too) is perhaps what's really wrong with internet discourse. It's not about sharing and learning, but hearing your own voice.
you're too lazy to post counterpoints and then call the place sad. you are the problem, my friend! i don't like posting this kind of comment but felt it needed to be called out. i'd prefer if neither of our comments were posted.
I worked for a medium size company that served, and still is, ~150 clients (some Fortune 500 included) by deploying prod with docker-compose. It can be done.
To those who are pointing out that compose isn't meant for production, keep in mind that this product that they're selling appears to be designed for the small time family self-hoster [0]. They're not targeting production web apps in a corporate setting, they seem to really be targeting people who wish that they could self-host something on their own network but don't have the technical background to use the docker compose files that most self-hostable apps provide as the 'easy' option.
I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
Yeah, we're very early building this, the blog post is just a way for me to organize my thoughts and start fights online. It's, uh, embarrassingly useful to yell semi-coherent thoughts into the void and have experts yell back with a decade or more of experience and information about tools I haven't heard of.
> I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
Yes, this is an excellent point. I did not articulate it well anywhere, but the goal is for users to have something more like Sandstorm, with a UI to install things. The TOML is for application developers, not end users. It'll either go in a separate database or, ideally, in the source code of the applications to be installed similar to a Dockerfile. I haven't started yet, but eventually we need to work with application developers to support things they want and to make it easier to treat Tealok as the "easy option" rather than docker compose.
Oh, that makes way more sense! Yeah, that actually sounds like it could work well if you can get buy in from application devs.
The trickiest thing doing it this late in the game is going to be that docker compose has truly become the standard at this point. I self-host a ton of different apps and I almost never have to write my own docker compose file because there's always one provided for me. At this point even if your file format is objectively better for the purpose, it's going to be hard to overcome the inertia.
Yeah, I agree, we're going to need a really compelling use-case not just for end users that run the application, but for the application developers as well. Nobody wants to maintain 3+ extra deployment files for the various also-rans competing with docker-compose.
What do you use to manage all those compose files? Do you have off-site backups? I'm constantly reading and re-writing docker-compose and bash scripting everything to fit in with the rest of my infrastructure it'd be good to hear about someone with a better way.
I have a single GitHub repo that contains all the compose files for my main server. Each application gets a folder with the compose file and any version-controllable configuration (which gets bound to volumes in the docker containers).
I periodically run Renovate [0], which submits PRs against the infrastructure repo on my local Forgejo to update all my applications. I have a script in the repo which pulls the git changes onto the server and pulls and restarts the updated apps.
Data is all stored in volumes that are mapped to subfolders in a ~/data directory. Each application has a Borgmatic [1] config that tells Borgmatic which folder to back up for that app and tells it to stop the compose file before backup and resume it afterwards. They all go to the same BorgBase repository, but I give each app its own config (with its own retention/consistency prefix) because I don't want to have network-wide downtime during backups.
At the moment the backup command is run by me by hand, with BorgBase configured to send me emails if I forget to do it for a week. Eventually that will be a cron job, but for now it takes less time to just do it myself, and I don't change my data often enough for a week of lost work to hurt much.
All the applications bind to ports which are firewalled, with Caddy and Pihole being the only applications that run on exposed ports (53, 80, 443). Caddy has a wildcard DNS cert from LetsEncrypt for HTTPS and directs traffic from a bunch of local domain names to the correct applications. I just use Pihole to define my local DNS names (custom.list, which is where Pihole keeps the local DNS definitions, is a volume that's committed to the repo).
people dramatically overestimate how difficult it is to write a program that controls docker for you. This is one of those things where you can write like two pages of Python and ignore... all this:
> Tealok is a runtime we’re building for running containers.
If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.
It's more than just a runtime for running containers, from the main landing page it looks like they're really specifically targeting self-hosting for the barely-technical [0]. In that context this article makes some degree of sense: their target audience will probably be legitimately overwhelmed by that pihole example and wouldn't know how to write a Python script to save their life.
If they can pull it off more power to them, but I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports, volumes, and even database implementations behind a magic text file. At some point you just have to bite the bullet and admit that your audience needs a GUI (like Sandstorm [1]).
I'm actually a fan of Sandstorm, and think it got a lot of things right. I'd love to be able to talk to Kenton Varda about why he thinks adoption on it was weak. Personally I think that it put a bit too much burden on application developers since it required them to develop applications specifically for sandstorm.
> I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports
I fully agree, the end user would not be writing TOML DSL files. The end user would get something much closer to an app store, or what Sandstorm did, with one (or a few) click installs. The TOML DSL would be written by developers familiar with the application and stored either in a separate database, or ideally in the applications source control like a Dockerfile.
> I'd love to be able to talk to Kenton Varda about why he thinks adoption on it was weak.
Oh hai.
Honestly I'm not sure I'm a reliable source for why we failed. It's tempting to convince myself of convenient excuses.
But I really don't think the problem was with the idea. We actually had a lot of user excitement around the product. I think we screwed up the business strategy. We were too eager to generate revenue too early on, and that led us to focus efforts in the wrong areas, away from the things that would have been best for long-term growth. And we were totally clueless about enterprise sales, but didn't realize how clueless we were until it was too late (a classic blunder). Investors really don't like it when you say you're going to try for revenue and then you don't, so we were pretty much dead at that point.
Oh, hey, holy shit! You're one of my heroes. I've read through your internal discussions on protobuf within Google, you did amazing work there, and held your own against a difficult political environment.
It sounds like you have no criticism of the technical approach, then, but rather just the business mechanics? That's eye-opening, given how much has changed in self-hosted deployment since Sandstorm started. If you started something similar today, ignoring business needs, would you build something technically similar?
You should probably take me with a grain of salt: of course, I, the technical architect of Sandstorm, still think the technical architecture is great. ;) Whereas I never claimed to be good at business so I'm not adverse to reporting I am bad at business.
But I do think there is a lot of work needed to get to a baseline of functionality that people will use. Bootstrapping a new platform is hard and needs a long runway. Could you find investors willing to back it? I don't know. I hated fundraising, though.
Of course, there are many lower-level details I would change, like:
* Consider using V8 isolates instead of containers. Sandstorm had a big problem with slow cold starts and high resource usage which V8 isolates could do much better with. It would, of course, make porting existing apps much harder, but ports to Sandstorm were always pretty janky... maybe would have been better to focus on building new apps that target Sandstorm from the start.
* Should never have used Mongo to store platform metadata... should have been SQLite!
* The shell UI should have been an app itself.
* We should never have built Blackrock (the "scalable" version of Sandstorm that was the basis for the Oasis hosting service). Or at least, it should have come much later. Should have focused instead on making it really easy to deploy to many different VPSes and federate per-user instances.
I'm not dev-ops-familiar with Docker, so you might be more familiar with the problem space, but it seems like "You can just write a Python script to do what you want" is the sort of thing people say to justify not giving people the config infrastructure they need to solve their problems without having to write executable code. Like, sure, you often can make up for not having a zoom lens by walking towards what you're photographing, but that doesn't mean zoom lenses aren't useful to most, and critical to some.
> you often can make up for not having a zoom lens by walking towards what you're photographing
Nitpicking: You can't really make up for not having a zoom lens as your field of view will stay the same. Hitchcock's famous "dolly zoom" demonstrates that quite nicely.
It brings a new layer of complexity, which means more surface area for bugs and vulnerabilities.
It’s often easier for developers to write a script using standardized tooling than dig into deep configuration docs for a complex application you’re not familiar with, but that’s where the benefits stop. Configuring built-in functionality makes sense for the same reason using an existing framework/os/etc authentication system makes sense. It often seems like way more effort to learn how to use it than rolling your own simple system, but most of that complexity deals with edge cases you haven’t been bitten by yet. Your implementation doesn’t need to get very big or last very long before the tail of your pipeline logic gets unnecessarily long. Those features don’t exist merely because some people are scared of code.
Additionally, if you’re just telling the application to do something it already does, writing code using general purpose tools will almost certainly be more verbose. Even if not, 10 to 1 is fantastically hyperbolic. And unless you write code for a living— and many dev ops people do not— defining a bunch of boolean and string values to control heavily tested, built-in application functionality (that you should understand anyway before making a production deployment) requires way less mental overhead than writing secure, reliable, production-safe pipeline code that will then need to be debugged and maintained as a separate application when anything it touches gets changed. Updating configuration options is much simpler than figuring out how to deal with application updates in code, and the process is probably documented in release notes so you’re much more likely to realize you need to change something before stuff breaks.
This is a hilarious take given the overwhelming number of outages that are caused by "bad config".
If you can't code, then yeah, I bet config is easier. But as a person who codes every day, I much prefer something that I can interact with, test, debug, type check, and lint _before_ I push to prod (or push anywhere, for that matter).
Ha. Just did almost exactly that, but with a Go script--I wanted my Docker Compose to auto-update when I built on my CI server.
I found Watchtower, but polling just struck me as the Wrong Answer. Both too much overhead to keep pinging for the latest builds, and too slow to actually download the latest build. So I took some hints from Watchtower as to what I needed to do (mount the docker sock as a volume) and wrote a tiny Go server that, when pinged with a shared secret, would cause it to run `docker compose up -d --pull always`.
Probably took me an hour.
Then I added the ability to purge images before each update, because my tiny VM kept running out of disk space in the Docker partition. Oops. Scripts FTW.
I was already using the suggestion in the article about having a single reverse proxy server to redirect different paths (and different domains) to different servers hosted in the Compose file. Seemed like the obvious answer.
And I've configured k8s for my day job, so I could be using that. But I'm using Compose because I know how much of a pain k8s can be, especially around upgrades, where it has a habit of deprecating older versions of various interfaces. I'll put in the work for someone who's paying me to do it, but I'd rather work on my side project, not on configuring k8s.
> I found Watchtower, but polling just struck me as the Wrong Answer. Both too much overhead to keep pinging for the latest builds, and too slow to actually download the latest build. So I took some hints from Watchtower as to what I needed to do (mount the docker sock as a volume) and wrote a tiny Go server that, when pinged with a shared secret, would cause it to run `docker compose up -d --pull always`.
Is the code for this public? I had the same desire when setting up the server for my personal project but as you mentioned I eventually decided it was OK to just poll every two minutes and I would rather work on the project than on configuring docker. What I would like though is cleaning up the older images that are no longer needed.
> If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.
… aand there is even Ansible playbook for that! Maybe overkill.
IME people overestimate how hard it should be to write a program, but underestimate how hard it actually is after they're done overengineering the problem to death.
Shameless plug: I’m building https://canine.sh as a way of turning any managed kubernetes cluster into something as easy to use as Heroku.
It’s been super frustrating in the past to be stuck on expensive PaaS vendors, but then rolling a deployment solution from scratch ended up with us trying to stitch together GitHub actions.
Been just using canine for my own projects and I’ve been able to host 4 rails & 1 Rust app pretty seamlessly with it on a $24/month digital ocean server. Would love any feedback!
How would you rate the "overhead" of your tool+k8s (also wht specs does your server have?) because my main use for docker-compose so far has been "host a ton of small apps on 5EUR hetzner instances with just 2GB RAM". If I had a beefier box I'd probably tried k8s again but in my recollection of when I last used it was bad if you had e.g. 5x 2GB RAM/2 CPU vs 1x 8GB RAM on one box.
Yeah Hetzner is amazing and I think if you’re looking at a single app for costs in the $5 range, it’s probably not worth it to use k8s.
The magic with k8s is being able to easily host third party packages via helm. I often find for anything I want to run in production, I usually want something like sentry, grafana, etc, and those items can get expensive if you try to buy hosted versions. Kubernetes makes it trivial to just host it yourself.
In terms of performance, I’ve gotten reasonable performance out of a $24 kubernetes server, which is 2vCPU + 4GB memory. These days, DigitalOcean and linode don’t charge a markup on their managed kubernetes at all, above their regular VPS pricing.
Heztner is much cheaper than both those options though, and they don’t have a managed kubernetes product
Yep! I should mention that on the website. The only thing that was innovative about that is doing docker-in-docker builds which is as simple as mounting /var/run/docker.sock:/var/run/docker.sock in the container. Zero downtime deploys make it sensible for it to deploy itself, and then restart. And it really breaks, I still know how to deploy from my laptop to get myself out of a pickle.
It has a very brief introduction to Docker and Docker Compose, but then it mostly has a lot of FUD that's designed to scare people off of Docker Compose but is not remotely convincing to someone who's actually worked with it. The way they frame that pihole example ("Whew! That’s a lot of stuff!") is just silly.
Looking at their website, I think they started out trying to make self-hosting really easy for a barely-technical user [0], which accounts for their attitude towards docker-compose, but it also means that the post was pretty devoid of useful information for someone who actually wants to self-host with the tech that's already ubiquitous today.
> The way they frame that pihole example ("Whew! That’s a lot of stuff!") is just silly.
Yeah, you're probably right. Originally that line was in there when I had a breakdown of what each line in the docker-compose was doing. My editor thought that was unnecessary - it's unlikely people reading the post would need that kind of breakdown. So I rewrote parts to assume more baseline knowledge. I should have noticed that line and taken it out.
You're right about what we're trying to do, and I agree that the post doesn't really help someone be successful today deploying things. The post is more meant to gauge whether or not I'm alone in having pain deploying a couple dozen services with docker compose on a single box.
I want more people to have the power to host their own services. I think we can do that, but we have to figure out the right thing to build to do it.
Agreed. By default I'm against content marketing. But as a person that has played around a lot in the /r/selfhosted scene, I had to agree with all of the use-case issues mentioned in the article when it comes to managing your self-hosted apps in Docker Compose (and I say this as someone that legitimately enjoys working with Docker).
I'm still not clear from the website what Tealok actually is in terms of a product offering, but I appreciate that the blog post was legit and not just SEO spam.
More interesting is that there are tons of tools that convert docker-compose files to Kubernetes on the fly. Minikube and friends are pretty good and solves all the problems, in a more battle-tested way.
It says docker compose is at the "wrong-level of abstraction", but I kept feeling the author was instead expecting docker compose to solve different problems that it was ever meant to solve.
In fact, they seem to be expecting a highly-opinionated, high-level interface which solves problems that I don't think anyone using docker compose in prod should even be worried about.
A lot of concerns seem to be around avoiding spinning up duplicate instances of reverse proxies, databases, and caches. First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern. This is a nuanced and context-dependent issue, but generally it won't even make the list of top-100 performance bottlenecks for most applications. Yet the article takes for granted that the benefits of solving this problem outweigh the many cons of coupling them together.
Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Beyond the specific issues I see, the fundamental attitude seems to be "force everyone to architect their applications in a very specific way, and don't even try to support any use cases which fall outside of it". You need a damn good reason to be that opinionated about these sorts of things, and it by definition will only work well in specific contexts. I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
These are great points, and probably worth their own blog post to answer.
> First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern
Idle threads have very low impact on CPU utilization, probably, if the application is well-behaved (and I expect most databases and caching layers to be well-behaved in this way). The application itself, however, will need memory and the way containers are built prevents the usual de-deplication of system libraries.
> but generally it won't even make the list of top-100 performance bottlenecks for most applications
True, but it makes the short list of "how much stuff can I run on a $100 computer", and it's one of the relatively few concerns an application operator has when they are not the application developer.
> Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
To make self-hosting much simpler. If the container orchestrator doesn't do it, what do you think should do it?
> Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
The DB backups are meant for disaster recovery rather than supporting multiple nodes. I guess that's multiple nodes through time... But, yeah, I agree, docker-compose is not a good fit.
> Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Yeah, agreed, I'll be writing other things to discuss what I think the correct solution should be. I'm curious to find out if other people have existing solutions to the problems I outlined. If it's a solved problem and I just don't know about it, that'd be better.
> I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
After reading it I don't get what the author is trying to tell me. Docker compose works just fine for what it's made. I can have multiple different docker containers (deployed with compose) all with the same port and just use Traefik to route the requests, without having to change the default ports or even use ipv6.
What am I missing?
Interesting, I wasn't aware that Traefik could do this without significant modification to the docker-compose configuration provided by the application developer. I also thought that Traefik required some sort of higher-level container orchestration like Docker Swarm or Kubernetes.
Not sure why comments aren't jumping on this more. Swarm is for production. compose is for development. That's always been the case.
And Swarm is barely more complicated than compose anyway. Don't know why people who are claiming they use compose in production just don't learn the few extra steps to do what Docker recommends.
It basically looks for a custom external hostname property which is used to generate the haproxy configuration and issue SSL certs.
For ports I personally prefer to just use a consistent container port as much as possible (eg: 80) - I'm sticking a reverse proxy in front of everything anyway so no need for unique ports bound to the host that I won't remember.
I wouldn't call this approach production grade - I'm very much in the use managed Kubernetes/SQL camp there, but it's been working great for my personal needs
Docker Compose/SSH and some custom GitHub Actions to run DB Migrations is simple, straightforward and was more than enough to manage CI deployments all our Apps [1].
Although we're currently migrating to Kamal [2] to take advantage of its nicer remote setup, management and monitoring features.
I'm having this problem now with our home automation stack. Compose is super easy to start with but scales poorly and makes things like "I want to access this service at https://foo.bar instead of http://somehost:61234, no HTTPS because the app doesn't support it natively and I don't have time to roll nginx."
Kubernetes makes this so easy once you get past the YAML and abstractions, so I'll probably FINALLY move this stack over to it. Long overdue.
And this is coming from someone who for the longest time recommended that developers new to containers start with compose before considering Kubernetes!
I also think that both docker-compose & k8s & helm are wrong layer of abstraction. I see a lot of people building a opinionated way to run containers and way for them to communicate with another, to request a DB or a redis.
I like to name some such attempts:
- .NET Aspire
- Avito Plato (home grown PaaS of ¨russian Amazon")
- infrastructure layer of our ZIIoT (MES platform on top of k8s)
I’ve been building a Docker Swarm based server dashboard for a couple years now. It should help with some of problems mentioned in the post: https://lunni.dev/
I like the ideas mentioned though! Docker Compose is pretty good, but something more declarative would definitely be a plus. Would love to see where the project goes (and maybe will borrow some things for the future versions of mine :-)
This is a fun problem to see on here (for me) - because I solved a lot of these problems at my last company when I built them a dev enviroment that ran over 30 services, and growing.
It was written as a makefile that wrapped docker compose commands, but that's just because it was the easiest to mvp.
I considered it a "framework for running a large collection of containers locally".
I solved the ports issue by using env vars like a switchboard, with an iPortService=<inside container port>
and
oPortService=<outside network port unique to network-layer>
Then i used two directories, one for a data layer that ran datastores, and one for app, for apps. You controlled the layers as groups.
You could operate the layers separately, initialize the dbs, and also reset their storage state definitely.
Then I had to solve the "docker ps" output problem, since its unreadable. So, i just split it to 3 commands: state,ports, & running. It outputs what you expect.
I should re-write it into better code and open source it. It was fun to build.
Incredibly easy to setup and manage for usual small scale deployments. I use it as one node swarms, and I have setup
- backups
- automatic https certificates setup and renewal
- automatic upgrades to new images
- easy setup of persistence of data on the server
I'm very surprised it is not more popular, with quite some people trying to replicate swarm's features with docker compose, but often in harder to maintain setups.
I love Swarm and don't see the appeal of K8s when something as simple as Swarm exists. I do however run K8s in prod for work and would never run Swarm in prod due to Docker seeming to have its days numbered. Idk where that leaves us aside from ECS. But I also have no need to run something any more robust than ECS in AWS for my workload.
We are moving our EKS workload over to ECS over the next year. I expect needing to down size my team because of it.
One thing K8s is not is cheap. That shit takes a well oiled team or a couple of hot shots so do right. We probably did a lot of what makes it expensive to ourselves by not switching to managed add-ons sooner and never evolving the apps that run in the cluster. I've only been lead for about 5 months now, but I'm finally able to start making significant progress on the necessary evolution that I've been trying to make happen for 2 years before my promotion. The enterprise is a big ship. Takes time to turn. Thanks for reading what turned into a rambling vent session.
I agree it depends on the situation, but there still are some situations (like small apps) where I see swarm as the way to go. Yours is probably different.
That's indeed the opinion of the author.
Note however that at this time all elements used in the setup described on dockerswarm.rocks are maintained. I started using swarm in 2022 and I documented my decision [1], and my reasoning for my kind of needs as not changed. The investment is very low, as well as the risk. Migrating away from swarm should not be very problematic for me, and in the meantime I enjoy an easy to maintain setup. I still think it's better than tweaking a maybe working solution with compose.
I'm not expecting to convince anyone but wanted to share an alternative approach(only applicable to certain setup)
Maybe I am a boiled frog, but I personally like it. TOML looks fine for simple cases, but yaml is fine there as well. Also, yaml has very clean approaches for dealing with literal text. For instance, if your yaml file needs to include an xml file, a json file and a bash script, it all ends up being very readable.
I may consider using HOCON [0] if I see any traction, bust after writing a lot of YAML and even making my own YAML-driven tools, I feel its shortcomings are overstated. I got bit by corner cases maybe three or four times, and they didn't take long to debug.
For the authors, I suggest a TL;DR with some basic examples. People tune out pretty quickly. You want the 10s take away to be "I like it" and "this person has really thought through it". You have achieved the second objective with what you have.
Strangely enough I'm at the complete opposite end of the author. Docker Compose imo operates exactly at the right abstraction layer and to me the author never really explains what the problems with Docker Compose actually are.
Imo Docker Compose absolutely should not handle databases and web servers. It should orchestrate containers. It's up to the user what kind of containers should be orchestrated.
Also the author somehow never got the idea to run the reverse proxy in a container? You don't need to do any port mapping then.
To save someone the click, this is a content-based marketing article.
TL;DR: Author believes docker compose is too complicated, and has a grudge against YAML for some reason. Author proposes an alternative configuration syntax that hides implementation details behind named templates.
So despite what the author wants us to believe, this isn't ACTUALLY a replacement for docker compose. This is yet another "easy-to-use" container orchestration product where there's extra upstream development between you and the docker image. Docker compose can run anything you can stuff into a container. This cannot, without some additional development. That may be value in that, but I'm not sure blasting docker compose as old and busted right out of the starting gate is a terrific marketing strategy.
Weird editorialization. I included a TL;DR at the top, you could have just copy-pasted it.
"Docker-compose is a tool for working with Docker containers. It solves very real problems with deploying complex applications. By itself it is not enough to make self-hosting applications simple enough for the mass-market."
One thing I really dislike about compose is that its a CLI tool but it doesn't exist in library form with a similar API.
I wrote a backup tool for compose/swarm based setups based on restic but ended up having to reimplementing half of traefik because there was no more straightforward API and compose really is just a thin layer over docker.
i've been at the edges of docker-compose. We were always conceptually recreating kubernetes. I'm so happy we've got k8s now and sparingly use some operators where needed, couldn't be happier.
The obvious example was certificate management. Cert-manager alone is almost worth setting up K8s if you have many ingresses/fqdns.
I'm moving into self-hosting for myself (photos, books, GPS tracks, exercise tracking) and so far Docker Compose has been perfect for me.
I'm using Synology NAS which has a nice GUI manager for Docker-Compose. So I put all the data volumes on btrfs, and I snapshot them every day. Then I sync snapshots to Wasabi via duplicacy.
It works amazingly well. No problems whatsoever. I guess having a nice front-end that shows the overall overview would be nice, but I can live without it.
I don’t get why docker doesn’t include a simple command (docker backup volumexyz mybackup.tar.gz) that creates an archive from a volume. Why do I have to do that myself by mounting the volume in a second container? Ok, there might be volumes that can’t be backed up (mounted sockets for example), but that could just give an error message.
Docker is simply a packaging format with docker compose being a wrapper for docker run. Author seems to confuse general application issues with docker's.
Article wasn't convincing enough about docker compose's shortcomings. The linked github's README even less convincing. But to each his own.
If docker compose is not enough I would suggest to look into some of the lightweight k8s distributions like k3s, with low ressource consumption and able to operate on a single node.
Curious! I think of docker-compose as squeezing a cluster onto a workstation—not exactly a tool you'd look to for anything other than home servers or conveniently-low workloads.
You know you can run multiple processes inside a single container, right?
The solution to the complexity of applications that are distributed as collections of interdependent containers is to put all of the different pieces inside a single container. This is equivalent to what people did before docker existed, and it's still perfectly viable today.
scaling docker compose to multiple machines is traditionally the realm of kubernetes. I’m not seeing any suggestion here that’s better than dropping in k3s
Accidental complexity, accidental complexity, accidental complexity and when we use a tool that is designed for problem A for a problem B it is not sufficiently dealing with complexity. This is why we need a new tool.
While a bit of a hot take, you're not wrong. We need something that's less scalability focused than Kubernetes/Mesos/Docker Swarm but that doesn't put too much burden on application developers. Something that focuses on being secure, reliable, and understandable, in that order. I'm not aware of anything going for that niche. That means a new tool is in order.
I think we need a composable system but I am not sure if the current frame where this problem and these tools exist is good enough. We might need to rethink how we handle access and usage patterns well. I only have wrong answers. Docker compose is amazing for local dev env, k8s is terrible for production. These are my experiences with this domain.
I know this is content marketing fluff for their own product, but complaining that Docker Compose isn't a full-fledged web application platform is idiotic. Use something like Dokku if you need such a solution.
Being able to tell a fellow developer "do docker compose up and you can work" is a lot better than navigating them through the installation of a bunch of tools, each with their own quirks.
I'm not convinced anyone below google needs kubernetes, but containers for quickly setting up and running something on my machine are a blessing.
Agreed. Running a modern "computer program" is more than just executing a single binary file.
When you have a open-source application with five or six moving parts (monitors, backend, a database, etc.) being able to deploy it to a VPS with a single docker compose and have them all containers act on an internal dockerized network without conflicting ports, etc. is a GOD SEND.
Containers are a convenient work-around for the problem where programs have incompatible dependencies, and additionally the problem where security isn't as good as it should be.
For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful. With docker containers, you can just containerize each application with its own Python version and have a consistent environment that you can run on lots of different machines (even on different OSes).
They're also a lot more convenient than having to go through the arcane and non-standard installation procedures that a lot of software applications (esp. proprietary ones) have.
Yeah, honestly it kinda sucks that we're adding this layer of inefficiency and bloat to things, but these tools were invented for a reason.
> For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful.
This is because the Linux model of global system wide shared dependencies is stupid, bad, and wrong. Docker and friends are a roundabout a way of having a program shipping its dependencies.
The Linux model works fine (very well, in fact, because of less HD space and much more importantly, less memory used for shared libraries) for programs that are normally included in the Linux distribution, since the whole thing is built together by the same organization as a cohesive whole. If every random little 20kB utility program were packaged with all its dependencies, the bloat would be massive.
It doesn't work very well for 3rd-party software distributed separately from the OS distro and installed by end-users.
The problem I've seen is that, while pre-Docker there was really nothing preventing ISVs from packaging their own versions of dependencies, they still only targeted specific Linux distros and versions, because they still had dependencies on things included in that distro, instead of just packaging their own. The big thing is probably glibc.
As I recall, Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
> because of less HD space and much more importantly, less memory used for shared libraries
Literally not in the Top 1000 problems for modern software.
> Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
Not really. A 20 year old piece of windows software prettt much “just works”. Meanwhile it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
>A 20 year old piece of windows software prettt much “just works”
No, it only works because Windows basically included something much like WINE (they call it WoW) in Windows, so old pieces of software aren't running on the modern libraries.
>it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
Sure you can, with Docker. It's effectively doing the same thing Windows does with WoW.
So the problem is 1) port mapping and 2) backing up data volumes?
There are simple solutions to each
1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for details see https://docs.docker.com/compose/how-tos/multiple-compose-fil...
2) this is literally as simple as running a small bash script that creates a backup, tars and gzips the file and then uploads it to an S3 repository. Sure, maybe not out of the box, but generally pretty simple. Likewise i'd always have a script that does the inverse (downloads the latest backup, loads it into the db and restores the files).
Not exactly rocket science...
For 1, docker has a handy special file to make this even easier! `compose.override.yaml` loads automatically on top of the main `compose.yaml`. We keep dev configs in there so Devs only need to run `docker compose up` locally without any environment variables, but on prod (which is automated) we set the compose file to be `compose.yaml:compose.prod.yaml`.
I do it the other way round, so in production you just have to use `docker compose up`. I did not understand the `compose.yaml:compose.prod.yaml` syntax or never saw it. What does this do?
Oh if you set the environment variable COMPOSE_FILE, you can specify a colon-separated list of compose files, which get merged together. This lets us have our core services in our main compose.yaml, since they're shared between dev and production, and then have extra services that are prod-only, or toggle certain prod-only environment variables, inside the compose.prod.yaml . And extra services/settings which we only want in the dev environment go in compose.override.yaml .
Eg
My work is OS so you can check it out here if you wish: https://github.com/internetarchive/openlibrary/Yup, I was surprised how weak the arguments were. I mean, surely there are better reasons why docker compose fails to scale? I have been pretty happy with it right now, and haven't felt the need to try something like k8s or docker swarm.
Sounds more like a skill issue. I am quite glad with traeffik and docker-compose. But I don't have that high loads or interactions on my servers
It is a skill issue, but not a problem. The developer who just wants to write unrelated code doesn't need Docker to write the code that goes into Docker.
For a lot of developers who have their primary focus set on writing code and documentation, trying to remember Docker or Kube on top of the rest of your stack is burdensome. Deployment is an end-user task. Obviously the development environment fits the target use-case already.
So please understand that there are lots of people who are building the things that go inside your containers who don't want, need, or see the value of these containers. For you they might offer value, but for someone like me they bring cognitive load and troubleshooting.
Too much abstraction can be just as bad as not enough. Like Akin's Laws of Spacecraft Design, but for software.
How many steps is it for you to add a new service to your system? Say you wanted to try out Paperless NGX, what would you need to do?
(Not parent) The quickest way I could do it is if I just used my existing wireguard setup to directly access the docker container IP. In that case it's just one compose up away even without mapping/exposing ports.
Thanks, interesting.
Sounds like your container has some kind of side-car that makes it directly addressable over Wireguard without needing to address the host IP. Does that mean you'd need to modify the docker-compose in some way before `docker-compose up`?
How do you know which port Paperless is using for HTTP? When you want to load up Paperless in a web browser, are you typing in a service name, or the container IP address? If it's a service name, how are you doing DNS? Do you have TLS?
I can already directly access the docker network because it's in the allowedips setting of wireguard. But for convenience, yes, when I do get a docker-compose.yml file I change 127.0.0.1 IPs to that of the wireguard interface IP. (I can leave 0.0.0.0 IPs as is because this host does not have a public IP) This way I am exposing the ports, but I am exposing them in a way that they are not world-accessible but still accessible to me conveniently.
For services open to the public internet I just add a subdomain + a reverse proxy entry into an existing caddy instance and point to the docker IP.
> Sounds like your container has some kind of side-car that makes it directly addressable over Wireguard
Not necessary. You can access the deployed docker container without exposing any ports or having any reverse proxy (what you've likely thought about with sidecar, which is a k8s concept, not docker) or anything else by using the ipadress of the started container and the ports the started service used. This is usually only possible from localhost, but wireguard can be configured as what's essentially a bastion host and exit node, this would let connecting clients also address containers that were started on that server, without opening any ports.
You can technically also do that without wireguard even, as long as you configure the docker host to route relevant traffic into the docker ethernet and define the docker subnet as a static route that points to the docker host, but that's another story
It took me a moment to put this together too, so to be clearer - the wireguard endpoint is in docker, so you're adding the docker bridge to your vpn. So DNS is handled by docker, just as containers can already address each other by name - you're on the container network with them.
I don't actually do this. I either access the services by IP or add a private IP to dns. (I think this is not widely supported but cloudflare does support it.)
Your explanation is interesting though. Would that actually work?
Add the labels to the docker-compose file and then run it.
I regularly try out new services or tools with docker-compose
[dead]
You can solve anything with enough scripts, but that in no way invalidates his point that the abstraction level seems a bit off.
I'm not sure that his solution is really an improvement, however, because then you have to bake in the semantics for a fixed set of product types, and when that list gets extended, you have to bake-in a new one.
Not only that he wants to re-invent messaging and daemons, essentially.
There are pre-existing paradigms for service brokers, message passing, etc., they all try to solve this problem, and they all share this same fault, they become in-extensible...
Which is the core programming tradeoff, flexibility vs robust constrained behavior.
On the extreme end of flexibility you have AI goop you barely understand, but arguably it is far more flexible than anything you ever wrote, even if you can't make any guarantees about what it does, on the other hand you have these declarative things that work fantastic if that's what you wanted, but probably not worth getting into computers for...
This is a really good conceptual model, the tradeoff between flexibility and constrained declarative frameworks. The goals is to make self-hosting applications extremely easy and extremely reliable. With that as a goal, being highly constrained seems like the way to go.
Borgmatic works great for #2. before_backup runs docker compose stop, after_backup runs docker compose up, done.
> So the problem is…
This feels like quite a shallow reading. Being able to manage system-wide things (DNS, and a centralised Caddy, centralised database) from individual compose units in such a way that you're not going through a ton of setup each service. This much might just need Caddy to work better with other instances of itself but it isn't a solved problem.
I'll grant you that backing up volumes isn't hard.
I think 2) actually has the most merit. If you have a large number of e.g. temp files a container creates, retaining that file history may not be particularly valuable.
Some sort of "hey back this data up, it's important to my function standard would be nice, and not just for container apps.
Trying to figure out where your app hid its configuration and save files is a perennial user frustration.
I use the convention that all my containers are getting a /data/ folder where data can be stored and will get an automatic daily backup.
It is easy to set the right storage behind /data/.
Just a convention, nothing more, smoothly working for the past "many" years.
I assume you're saying this as a container image author, not as someone who is deploying containers. It'd be great if every other container image author standardized on something like this. We just need someone to create the standard and some tools to make the standard a well-paved path.
In this particular case, I am both creating the images and deploying them.
For #2 I use https://kopia.io/ and upload to Backblaze b3 (S3 api)
I guess it’s simple if you already know which S3 service you want to use? For those of us who don’t, it means it’s time to go shopping for one.
As someone who is self-hosting with a docker on my own server, I don't see the negatives mentioned in this article as being a problem at all. Quite the opposite, it gives you the freedom to do your docker setup how you want it.
It took me some time initially to figure things out and I had to learn some things. But now it's a breeze. Once I had reverse proxy and automatic certificate renewal in place it has been working ever since then without me having to do anything. Adding a new service like Immisch or Jellyfin takes me an hour or less. Which can be quicker, but I adjust every docker compose to my setup and make it more secure. E.g. I create a new non-root user for each service. Basically I have the whole setup figured out; I have notes and checklists for the new services I add. I don't need to figure out things anymore and in 95% of the cases things just work.
Updating existing services takes minutes: just increment the version in the compose file and rebuild.
For my setup, I use macvlan as opposed to the default docker bridge network. So, the `ports: - "8000:5000"` docker compose config is being ignored. Instead, each docker container gets its own unique IP and MAC address I assign to it. It can then use any port it wants. Thanks to this, on my home network, I can access any container I want by IP. And any container can access any other container. I then add some restrictions for security purposes on the host machine using nftables.
For reverse proxy, I use Nginx Proxy Manager which is a simplified GUI around Nginx. I'm slowly moving to just using Nginx.
For the database, I run multiple instances of Postgres and MySQL and I don't see any issues. They take an insignificant amount of resources compared to the applications. If the application is not in use, its database instance is not being used as well.
I've always understood docker-compose to be a development or personal tool, not for production. Is that not usually the case?
Also aside, "docker compose" (V2) is different from "docker-compose" (V1) [0], it was rewritten and integrated into docker as a plugin. It should still be able to handle most old compose files, but there were some changes.
[0] https://docs.docker.com/compose/releases/migrate/
For many people self-hosting implies a personal server; it's not really "development" but it's not "production" either. In that context many people find k8s or other PaaS to be too heavyweight so Docker Compose is pretty popular. For more production-oriented self-hosting there are various newer tools like Kamal but it will take a while for them to catch up.
I've managed to keep my personal server to just docker, no compose. None of the services I run (Jellyfin, a Minecraft server, prowlarr, a Samba server [this is so much easier to configure for basic use cases via Docker than the usual way], pihole) need to talk to one another, so I initialize each separately with shell scripts. Run the script for a new service once, and Docker takes care of restarts on system reboot or if the container crashes, don't even have to interact with my base OS's init system or really care about anything on it. When I want to upgrade a service, I destroy the container, edit the script to specify the newer version I want, and run the script again. Easy.
On my personal home server I abuse Docker Compose so I don't have to use a huge command line to spin up/spin down containers but I can just say "docker compose up -d" and be done with it.
> For many people self-hosting implies a personal server; it's not really "development" but it's not "production" either.
There's Docker swarm mode for that. It supports clustering too.
It's nuts how people look at a developer tool designed to quickly launch a preconfigured set of containers and think it's reasonable to use it to launch production services.
It's even more baffling how anyone looks at a container orchestration tool and complains it doesn't backup the database they just rolled out.
> In that context many people find k8s or other PaaS to be too heavyweight so Docker Compose is pretty popular.
...and proceed to put pressure to shitify it by arguing it should do database backups, something that even Kubernetes stays clear from.
The blogger doesn't even seem to have done any research whatsoever on reverse proxies. If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about. He would also have researchdd what it means to support TLS and how this is not a container orchestration responsibility.
Quite bluntly, this blog post reads as if it was written by someone who researched nothing on the topic and decided instead to jump to
Hey, blog post author here.
I'm curious how that last sentence was going to end.
Let's say I agree with you and that TLS termination is not a container orchestration responsibility. Where does the responsibility of container orchestration start and TLS termination end? Many applications need to create URLs that point to themselves so they have to have a notion of the domain they are being served under. There has to be a mapping between whatever load-balancer or reverse proxy you're using and the internal address of the application container. You'll likely need service discovery inside the orchestration system, so you could put TLS termination inside it as well and leverage the same mechanisms for routing traffic. It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
In the end we all build systems to do things to make people's lives better. I happen to think that separating out backups and managing ports as an exercise for the deployment team raises the barrier to people that could be hosting their own services.
I could be totally wrong. This may be a terrible idea. But I think it'll be interesting to try.
> If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about
I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes cluster, and later on a stack of Odroids using k3s. This was years ago, so it may have changed a lot since then, but it seemed at the time that I needed an advanced degree in container orchestration studies to properly configure it. It felt like Kubernetes was designed to solve problems you only get above 100 nodes, then k3s tried to bang that into a shape small enough to fit in a home lab, but couldn't reduce the cognitive load on the operator because it was using the same conceptual primitives and APIs. Traefik, reasonably, can't hide that level of complexity, and so was extremely hard to configure.
I'm impressed at both what Kubernetes and k3s have done. I think no home lab should run it unless you have an express goal to learn how to run Kubernetes. If Traefik is as it was years ago, deeply tied to that level of complexity, then I think small deployments can do better. Maybe Caddy is a superior solution, but I haven't tried to deploy it myself.
If you want an HTTPS ingress controller that's simple, opinionated, but still flexible enough to handle most use cases, I've enjoyed this one: https://github.com/SteveLTN/https-portal
> Let's say I agree with you and that TLS termination is not a container orchestration responsibility.
It isn't. It's not a problem, either. That's my point: your comments were in the "not even wrong" field.
> (...) It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
No. My point is that you should invest some time into learning the basics of deploying a service, review your requirements, and them take a moment to realize that they are all solved problems, specially in containerized applications.
> I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes (...)
I recommend you read up on Traefik. None of your scenarios you mentioned are relevant to the discussion.
The whole point of bringing up Traefik is that it's main selling point is that it provides support fo route configuration through container tags. It's the flagship feature of Traefik. That's the main reason why people use it.
Your non sequitur on Traefik and Kubernetes also suggests you're talking about things that haven't really clicked with you. Traefik can indeed be used as an ingress controller in Kubernetes, but once deployed you do not interact with it. You just define Kubernetes services, and that's it. You do interact directly with Traefik if you use it as an ingress controller in Docker swarm mode or even docker-compose, which makes your remark even more baffling.
> I'm impressed at both what Kubernetes and k3s have done. (...) If Traefik is as it was years ago,(...)
Kubernetes represents the interface, as well as the reference implementation. k3s is just another Kubernetes distribution. Traefik is a reverse proxy/load balancer used as an ingress controller in container orchestration systems such as Kubernetes or Docker swarm. The "level of complexity" is tagging a container.
Frankly, your comment sounds like you tried to play buzzword bingo without having a clue whether the buzzwords would fit together. If anything, you just validated my previous comment.
My advise: invest some time reading on the topic to go through the basics before you feel you need to write a blog post about it.
it's perfectly fine if everything can run on a single server instance. which is probably the majority of things.
i've run production instances with 2PB of data being scraped per month by 100+ CPUs and 256GB of RAM using docker compose. some of the machines (smaller instances) have run flawlessly with zero reboots for years on end. both on cloud and on-prem.
i'd say its 100% production ready.
+1 to this. Smaller datapoint from my side I guess, but anyway, docker is the core of my self hosting setup and it is one of the things that I don't have to fiddle with, it just works.
> I'd say its 100% production ready.
That's fine. Some people also manually launch containers individually with Docker as their production system.
They'd be wasting their time and making their lives needlessly harder, though.
They already have tools that do all the legwork for them. Why not do the research and opt to force a square peg into a round hole?
Research Docker swarm mode, reverse proxies like Traefik, and Let's Encrypt. Kubernetes can be even easier with implementations like microk8s, which are a snap away from any Ubuntu installation. Look at what you're doing and figure out the right tool for the job. Don't just whine about how you're doing something wrong and it's the tool that needs to change to fix your own mistakes.
Not sure what axe you have to grind, but the person you replied to didn't whine about anything as far as I can see. I've also been hosting a lot of stuff on docker-compose and it is perfectly fine for a lot of things. No one said it's perfect or does everything.
>They'd be wasting their time and making their lives needlessly harder, though.
Using Kubernetes where it's not needed is just that - wasting your time and making your life harder.
Before you say that I just need to research more: I know Docker swarm mode, I run my personal server on Kubernetes using Traefik and Let's Encrypt, I professionally work with Kubernetes (both as an admin and working on Kubernetes security, which is tough to get right), most services in my dayjob run on Kubernetes, and I was the person who introduced CI/CD pipelines there some years ago.
I still claim that there are production usecases that are better served by docker-compose.
> Using Kubernetes where it's not needed is just that - wasting your time and making your life harder.
I think this is a very ignorant and misguided take.
Kubernetes is a container orchestration system, just like Docker swarm mode or even Docker compose. If you need to deploy sets of containerized apps into your own VMs, you can pick up any Kubernetes implementation. You don't even need Helm or anything. Plain old kustomize scripts will do. Some aren't even longer than a docker-compose.yml.
More to the point, Kubernetes is an interface. One that you can use in local deployment and in major cloud providers.
You should really check your notes because your comments contrast with the realities of actually running a service.
> Before you say that I just need to research more: (...)
All your appeals to authority are falsified by your claims.
I, on the other hand, actually use Kubernetes both professionally and in personal projects, as well as Docker swarm mode, and can tell you I'm no uncertain terms that none of your points have any traction in reality.
> I still claim that there are production usecases that are better served by docker-compose.
I'm sorry, but your comments simply sound deeply uninformed and misguided.
I mean, it makes absolutely no sense to comment on using docker compose in production when Docker swarm mode is far more capable and Docker swarm mode stacks already share most of the schema with docker compose. You literally have virtually nothing to do to adapt a docker-compose script to launch a stack.
Unbelievable.
Most people dont need rolling release, 24/7 availability, auto scaling, etc.. on their home server, so managing k8s just add way more complexity. My main reason to not use is because I would need to host a artifact service somewhere else which is PITA. Some k8s runtime support local building but is not as easy as compose in my experience.
> Most people dont need rolling release, 24/7 availability, auto scaling, etc..
That's perfectly fine. That's not the reason why you are better off running your apps in Kubernetes though.
You are better off running your apps in Kubernetes because it handles everything you ever need to effortlessly run containerized apps. You don't even need to install tooling or deployment tools or anything at all. You can have a single kustomize script that defines your services and deployments, have in place an ingress and Let's Encrypt, and you're done. You don't need to bother with anything else. Just run kubectl apply and go grab a coffee.
Is this something no one would ever need?
> You don't need to bother with anything else.
Hey that is not true, at least last time I tried I spent considerable time trying to integrate a bare metal load balancer from a 9 stars github repo plugin because apparently exposing the port is not recommended. Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
One last point, the runtime ram and cpu overhead is far from minimal.
> Hey that is not true, at least last time I tried I spent considerable time trying to integrate a bare metal load balancer from a 9 stars github repo plugin because apparently exposing the port is not recommended.
You sound like you tried to put together an ad-hoc ingress controller without knowing what an ingress controller is and why abusing NodePort is a mistake in the context of a cluster.
You can check this out to go through the basics.
https://kubernetes.io/docs/concepts/services-networking/ingr...
> Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
You should look at the problem to see if an approach makes sense. If you're using vanilla Kubernetes on a one-box deployment then you're abusing a large-scale high-scalable cluster management system by shoving it into a single box. It can work, but most of the resources will be wasted on managing your cluster of one node.
There are plenty of Kubernetes distributions that are designed to handle well small clusters, and even one-box systems. Minikube, microk8s, and even k3s come to mind. I'm partial towards microk8s because installing it is just a matter of installing a single package from the distro's official repository.
And I've seen compose services not surviving a banal host restart and require manual massaging.
So it's 100% production ready... for you.
I've used it for both.... but for production only in extremely constrained/limited environments. Its every bit as stable as anything else... you can version the compose you deploy so rollbacks are easy, etc etc.
far more useful for development IMO, but when push comes to shove and you need a few things running together on a machine in the middle of nowhere with limited/unreliable internet access.... you can't do much better... a few volume mounts for durability and you're really in decent shape.
logs, networking, system health etc etc.... just all a few docker and docker-compose commands away
Its not going to be amazingly professional and 100% best practice, but you can set up docker-composes and/or design containers to pull everything they need on first run.
That plus a decent backup system would work for a small array of servers with fail-safes. Though I would die inside if a production user-focused app under any level of proper load was set up like that.
You'll need more than a backup system. At least some sort of a load balancer to switch between different groups of running docker containers (so that upgrades, backups, etc... can happen without service being interrupted).
not every single system requires 100% uptime... some folks still use release windows for specific applications/hardware. I'd argue that most systems out there can support a release window.... and the overlap between "docker-compose works for us" and "we have release windows" is probably quite high.
We needed to do inference on remote machines stationed at random points across North America.... the machines had work to do depending on external factors that were rigidly scheduled. really easy to make upgrades, and the machines were _very_ beefy so docker-compose gave us everything we needed.... and since we shipped the data off to a central point regularly (and the data had a short shelf life of usability) we could wipe the machines and do it all again with almost no consequences.
I needed to coordinate a few small services and it did the trick beautifully
My experience has been that the vast majority of systems could tolerate a few minutes offline per month for upgrades. Many could tolerate a couple hours per month. No or negligible actual business harm done, enormous cost savings and higher development velocity from not adding the complexity needed for ultra-high uptime and zero-downtime upgrades.
What's vital is being able to roll back a recent update, recover from backups, and deploy from scratch, all quickly. Those are usually (not always) far easier to achieve than ultra-high-uptime architecture (which also needs those things, but makes them all more complicated) and can be simple enough that they can be operated purely over ssh with a handful of ordinary shell commands documented in runbooks, but skipping them is how you cheap out in a bad way and end up with a system down for multiple days, or one that you're afraid to modify or update.
Early this year I consulted for a company that used docker-compose as their software deployment mechanism. They created new compute, but compose gave them the sidecars you’d want from k8s without all the other overhead that came with.
To go even further, podman supports actual sidecars (one or many forming a pod) and even supports actual Kubernetes deployment manifests without needing a kube-apiserver.
How so? I run docker compose in production on 30 bare metals without any issue.
According to docker: "Compose has traditionally been focused on development and testing workflows, but with each release we're making progress on more production-oriented features."
https://docs.docker.com/compose/intro/features-uses/
Sounds like dev, and test tool with production aspirations that isn't actually there yet to me.
Same here, we never use compose in production.
There is already a third version of compose.
> I've always understood docker-compose to be a development or personal tool, not for production. Is that not usually the case?
Tons of sites are launched manually and have been for years in production.
Docker compose is a huge step up from that, so why wouldn't it be?
Just use k8s for prod. Docker compose is great for local dev, i would never dream of deploying to prod with compose.
Just use Podman with plain simple .container file which is controlled entirenly by SystemD. Rock solid and easy to fix.
That's great for system services, not for widely deployed applications.
Why not? I can't imagine you'd deploy an application without some form of service management (even if it's just throwing it in tmux) and unless you've gone out of your way to use a non systemd distro systemd is builtin and works for both user and root containers.
Most places (though not all) that I've seen using docker or docker-compose are throwing them in systemd units anyway.
What happens when your machine dies? You get paged, your service is down, you have to migrate the services to a new machine.
With Kubernetes, it starts new container on new machine, and you don't get paged.
You could write scripts to do failover, but that takes works and will be buggy.
I was more questioning why a .container file would work for system services versus application services, since basically all those same problems occur for system services too.
Either way this type of argument just comes down to "should I cluster or not" but to think out loud for a bit that's just basic HA planning: Simplest solution is keepalived/etc for stateful services, standard load balancing for stateless. Don't want load balanced services running all the time? Socket activation. Don't have a separate machine? Auto-restart the service and you can't cluster anyway. The only thing you'd really have to script is migrating application data over if you're not already using a shared storage solution, but I'm not sure there's any easier solutions in Kubernetes
What’s the advantage of configuring, testing and maintaining all of that instead of a 10 line Kubernetes manifest that does more?
Not having to install and manage Kubernetes? Unless you're paying someone else to run it for you (in which case this entire conversation is sort of moot as that's way out of scope for comparison) that stuff is all still running somewhere and you have to configure it. e.g. even in small-scale setups like k3s you have to set up shared storage or kube-vip yourself for true high availability. It's not some magic bullet for getting out of all operational planning.
Also even in separate components it's not really "all of that". Assume an example setup where the application is a container with a volume on an NFS share for state: on a given node we'd need to install podman and keepalived, a .container and .volume file, and the keepalived conf. An average keepalived conf file is probably 20ish lines long (including notification settings for failover, so drop like 8 lines if you don't care about that or monitor externally) and looking at an application I have deployed a similar .container file is 24 lines (including whitespace and normal systemd unit file boilerplate) and the NFS .volume file is 5 lines. So ballpark 50 lines of config, if you or others wanted to compare configuration complexity.
Also, fun fact, you could still even use that Kubernetes manifest. Podman accepts .kube files to manage resources against itself or a Kubernetes cluster; I've been recommending it as sort of a middle ground between going all in on k8s versus transitioning slowly from a non-clustered deployment.
You can always use something like Docker Swarm or Nomad which achieves the same end result as Kubernetes (clustered container applications) without the complexity of having to manage Kubernetes.
Just spawn another VPS with your application and connect to load balancer. Even better - use Fedora CoreOS with Butane config and make that VPS immutable.
We run large, auto scaling clusters with compose and a small orchestration thing we have been using since the 90s (written in perl) (before compose we had our own compose-like with chroot). No issues. For decades.
Cool - everyone knows it’s possible to hack together something, or anything else. But Is it a good idea to spend time and resources starting to do that now if you haven’t been doing it with a sketchy Perl script since the 90s? Not really.
You do you. I think k8s is overly complicated stuff that almost no companies (outside google/meta/etc) need and some peopple here peddle for some reason. That's fine. We like making profit instead of waste time with the latest nonsense.
I don't think it's a good idea to spend time and resources learning complexity you never need. You are probably going to say that's not true as your cloud hoster has this all set up, but yeah, that's where the making profits comes in. We save millions/year on not cloud hosting. And managing your own k8s cluster apparently is quite hard (at least that's what even the fanboiz here say).
Starting today, I would use Go probably instead of Perl, but I would do the same. It's much simpler than kubernetes. But sure; for our goals; we like simplicity and don't need resumes to drive.
I migrated several services from Kubernetes to compose and couldn't be happier. K8s was a maintenance nightmare, constantly breaking in ways that were difficult to track down and unclear how to fix once you did. Compose configs are simple and readable with all the flexibility I need and I've literally never had an issue caused by the platform itself.
This depends on your use case of course, compose autoscaling is maybe harder but I don't personally need that.
That's what I thought the first day, the first month and maybe even the first half year.
Now it's 5 years+ running, all fine. No major hassle (crossing fingers!), no configuration hell. Never touched k8s, and not going to in the near future.
Maybe it just works and is sufficient for my case.
Sure. But that’s not the point of the article. The articles point is “Docker Compose is too complicated”, then proposes a half baked implementation of a solution that hand waves away of all that complexity with a scenario that will fulfill their specific use case but will completely fall apart when it diverges from the specific way they’ve decided to build the abstraction layer.
The problem that the author refuses to accept is that deployment is inherently complex. It does require configuration of a lot of things because it supports every use case. So you either have a generic complex tool that everyone can use (like Docker Compose or Kube) or you have some specific tool that only works for a tiny subset of all users that is simpler that satisfies your use case.
Note that I’m not saying Docker Compose is perfect. The syntax is a bit arcane it’s complex to understand etc. But removing the complexity by removing configuration options is not the solution here. Instead the author should focus on different approaches and tools to manage the same existing level of abstraction.
And for what it’s worth, that’s essentially what Helm is for kube - a way to manage and hide away the complexity of kube manifests (but still use those manifests under the hood). But frankly, docker compose doesn’t need a helm. Because docker compose, as you point out, has value not as a deployment tool, but as a single file that developers can manage and spin up on their local machines in a manageable way that doesn’t have the author fighting YAML all day.
I would say if the author was actually interested in solving this problem in a productive way they should first try to see if docker itself is amenable to altering their constructs to provide optional higher abstractions over common concepts via the compose interface natively. If the source tools roll out those abstractions everyone will get them and adopt them.
> I would say if the author was actually interested in solving this problem in a productive way they should first try to see if docker itself is amenable to altering their constructs to provide optional higher abstractions over common concepts via the compose interface natively.
docker-compose is a lot of things to a lot of people. When it was created I doubt anyone realized it would eventually be the de facto standard for deploying to homelabs. It's an amazing tool, but it could be better for that specific use. I don't think that segment is important enough to the team that maintains it to warrant the change you're suggesting.
docker compose doesn't need to be overly complex though; I think if it starts to feel that way you're likely doing it wrong(tm). K8s IS very complex, and likely overkill but exactly what you should do if you need it in production. This was a very long ad with an unconvincing argument for a product that addresses the wrong problem.
I could come out with some counterpoints, but you’re too abrasive to even engage with. It’s sad what this place has become.
So you took the time to write a commentary about how the parent wasn't worthy of your counterpoints, but no actual rebuttal. That sort of low-effort / high opinion / no value / zero impact comment (and this post too) is perhaps what's really wrong with internet discourse. It's not about sharing and learning, but hearing your own voice.
you're too lazy to post counterpoints and then call the place sad. you are the problem, my friend! i don't like posting this kind of comment but felt it needed to be called out. i'd prefer if neither of our comments were posted.
Huh, why not? If you aren't building huge scaling apps, then docker (compose) does the job good enough.
I worked for a medium size company that served, and still is, ~150 clients (some Fortune 500 included) by deploying prod with docker-compose. It can be done.
To those who are pointing out that compose isn't meant for production, keep in mind that this product that they're selling appears to be designed for the small time family self-hoster [0]. They're not targeting production web apps in a corporate setting, they seem to really be targeting people who wish that they could self-host something on their own network but don't have the technical background to use the docker compose files that most self-hostable apps provide as the 'easy' option.
I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
[0] https://tealok.tech/
Yeah, we're very early building this, the blog post is just a way for me to organize my thoughts and start fights online. It's, uh, embarrassingly useful to yell semi-coherent thoughts into the void and have experts yell back with a decade or more of experience and information about tools I haven't heard of.
> I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
Yes, this is an excellent point. I did not articulate it well anywhere, but the goal is for users to have something more like Sandstorm, with a UI to install things. The TOML is for application developers, not end users. It'll either go in a separate database or, ideally, in the source code of the applications to be installed similar to a Dockerfile. I haven't started yet, but eventually we need to work with application developers to support things they want and to make it easier to treat Tealok as the "easy option" rather than docker compose.
Oh, that makes way more sense! Yeah, that actually sounds like it could work well if you can get buy in from application devs.
The trickiest thing doing it this late in the game is going to be that docker compose has truly become the standard at this point. I self-host a ton of different apps and I almost never have to write my own docker compose file because there's always one provided for me. At this point even if your file format is objectively better for the purpose, it's going to be hard to overcome the inertia.
Yeah, I agree, we're going to need a really compelling use-case not just for end users that run the application, but for the application developers as well. Nobody wants to maintain 3+ extra deployment files for the various also-rans competing with docker-compose.
What do you use to manage all those compose files? Do you have off-site backups? I'm constantly reading and re-writing docker-compose and bash scripting everything to fit in with the rest of my infrastructure it'd be good to hear about someone with a better way.
I have a single GitHub repo that contains all the compose files for my main server. Each application gets a folder with the compose file and any version-controllable configuration (which gets bound to volumes in the docker containers).
I periodically run Renovate [0], which submits PRs against the infrastructure repo on my local Forgejo to update all my applications. I have a script in the repo which pulls the git changes onto the server and pulls and restarts the updated apps.
Data is all stored in volumes that are mapped to subfolders in a ~/data directory. Each application has a Borgmatic [1] config that tells Borgmatic which folder to back up for that app and tells it to stop the compose file before backup and resume it afterwards. They all go to the same BorgBase repository, but I give each app its own config (with its own retention/consistency prefix) because I don't want to have network-wide downtime during backups.
At the moment the backup command is run by me by hand, with BorgBase configured to send me emails if I forget to do it for a week. Eventually that will be a cron job, but for now it takes less time to just do it myself, and I don't change my data often enough for a week of lost work to hurt much.
All the applications bind to ports which are firewalled, with Caddy and Pihole being the only applications that run on exposed ports (53, 80, 443). Caddy has a wildcard DNS cert from LetsEncrypt for HTTPS and directs traffic from a bunch of local domain names to the correct applications. I just use Pihole to define my local DNS names (custom.list, which is where Pihole keeps the local DNS definitions, is a volume that's committed to the repo).
[0] https://github.com/renovatebot/renovate
[1] https://torsion.org/borgmatic/
people dramatically overestimate how difficult it is to write a program that controls docker for you. This is one of those things where you can write like two pages of Python and ignore... all this:
> Tealok is a runtime we’re building for running containers.
If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.
It's more than just a runtime for running containers, from the main landing page it looks like they're really specifically targeting self-hosting for the barely-technical [0]. In that context this article makes some degree of sense: their target audience will probably be legitimately overwhelmed by that pihole example and wouldn't know how to write a Python script to save their life.
If they can pull it off more power to them, but I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports, volumes, and even database implementations behind a magic text file. At some point you just have to bite the bullet and admit that your audience needs a GUI (like Sandstorm [1]).
[0] https://tealok.tech/
[1] https://sandstorm.io/
I'm actually a fan of Sandstorm, and think it got a lot of things right. I'd love to be able to talk to Kenton Varda about why he thinks adoption on it was weak. Personally I think that it put a bit too much burden on application developers since it required them to develop applications specifically for sandstorm.
> I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports
I fully agree, the end user would not be writing TOML DSL files. The end user would get something much closer to an app store, or what Sandstorm did, with one (or a few) click installs. The TOML DSL would be written by developers familiar with the application and stored either in a separate database, or ideally in the applications source control like a Dockerfile.
> I'd love to be able to talk to Kenton Varda about why he thinks adoption on it was weak.
Oh hai.
Honestly I'm not sure I'm a reliable source for why we failed. It's tempting to convince myself of convenient excuses.
But I really don't think the problem was with the idea. We actually had a lot of user excitement around the product. I think we screwed up the business strategy. We were too eager to generate revenue too early on, and that led us to focus efforts in the wrong areas, away from the things that would have been best for long-term growth. And we were totally clueless about enterprise sales, but didn't realize how clueless we were until it was too late (a classic blunder). Investors really don't like it when you say you're going to try for revenue and then you don't, so we were pretty much dead at that point.
Oh, hey, holy shit! You're one of my heroes. I've read through your internal discussions on protobuf within Google, you did amazing work there, and held your own against a difficult political environment.
It sounds like you have no criticism of the technical approach, then, but rather just the business mechanics? That's eye-opening, given how much has changed in self-hosted deployment since Sandstorm started. If you started something similar today, ignoring business needs, would you build something technically similar?
Oh, well, thank you!
You should probably take me with a grain of salt: of course, I, the technical architect of Sandstorm, still think the technical architecture is great. ;) Whereas I never claimed to be good at business so I'm not adverse to reporting I am bad at business.
But yeah I still think it's the right architecture. Still believe in what's written here: https://sandstorm.io/how-it-works
But I do think there is a lot of work needed to get to a baseline of functionality that people will use. Bootstrapping a new platform is hard and needs a long runway. Could you find investors willing to back it? I don't know. I hated fundraising, though.
Of course, there are many lower-level details I would change, like:
* Consider using V8 isolates instead of containers. Sandstorm had a big problem with slow cold starts and high resource usage which V8 isolates could do much better with. It would, of course, make porting existing apps much harder, but ports to Sandstorm were always pretty janky... maybe would have been better to focus on building new apps that target Sandstorm from the start.
* Should never have used Mongo to store platform metadata... should have been SQLite!
* The shell UI should have been an app itself.
* We should never have built Blackrock (the "scalable" version of Sandstorm that was the basis for the Oasis hosting service). Or at least, it should have come much later. Should have focused instead on making it really easy to deploy to many different VPSes and federate per-user instances.
I'm not dev-ops-familiar with Docker, so you might be more familiar with the problem space, but it seems like "You can just write a Python script to do what you want" is the sort of thing people say to justify not giving people the config infrastructure they need to solve their problems without having to write executable code. Like, sure, you often can make up for not having a zoom lens by walking towards what you're photographing, but that doesn't mean zoom lenses aren't useful to most, and critical to some.
> you often can make up for not having a zoom lens by walking towards what you're photographing
Nitpicking: You can't really make up for not having a zoom lens as your field of view will stay the same. Hitchcock's famous "dolly zoom" demonstrates that quite nicely.
> to solve their problems without having to write executable code.
Some people would rather write 1000 lines of yaml than 100 lines of python, and I really don't understand why.
It brings a new layer of complexity, which means more surface area for bugs and vulnerabilities.
It’s often easier for developers to write a script using standardized tooling than dig into deep configuration docs for a complex application you’re not familiar with, but that’s where the benefits stop. Configuring built-in functionality makes sense for the same reason using an existing framework/os/etc authentication system makes sense. It often seems like way more effort to learn how to use it than rolling your own simple system, but most of that complexity deals with edge cases you haven’t been bitten by yet. Your implementation doesn’t need to get very big or last very long before the tail of your pipeline logic gets unnecessarily long. Those features don’t exist merely because some people are scared of code.
Additionally, if you’re just telling the application to do something it already does, writing code using general purpose tools will almost certainly be more verbose. Even if not, 10 to 1 is fantastically hyperbolic. And unless you write code for a living— and many dev ops people do not— defining a bunch of boolean and string values to control heavily tested, built-in application functionality (that you should understand anyway before making a production deployment) requires way less mental overhead than writing secure, reliable, production-safe pipeline code that will then need to be debugged and maintained as a separate application when anything it touches gets changed. Updating configuration options is much simpler than figuring out how to deal with application updates in code, and the process is probably documented in release notes so you’re much more likely to realize you need to change something before stuff breaks.
This is a hilarious take given the overwhelming number of outages that are caused by "bad config".
If you can't code, then yeah, I bet config is easier. But as a person who codes every day, I much prefer something that I can interact with, test, debug, type check, and lint _before_ I push to prod (or push anywhere, for that matter).
Ha. Just did almost exactly that, but with a Go script--I wanted my Docker Compose to auto-update when I built on my CI server.
I found Watchtower, but polling just struck me as the Wrong Answer. Both too much overhead to keep pinging for the latest builds, and too slow to actually download the latest build. So I took some hints from Watchtower as to what I needed to do (mount the docker sock as a volume) and wrote a tiny Go server that, when pinged with a shared secret, would cause it to run `docker compose up -d --pull always`.
Probably took me an hour.
Then I added the ability to purge images before each update, because my tiny VM kept running out of disk space in the Docker partition. Oops. Scripts FTW.
I was already using the suggestion in the article about having a single reverse proxy server to redirect different paths (and different domains) to different servers hosted in the Compose file. Seemed like the obvious answer.
And I've configured k8s for my day job, so I could be using that. But I'm using Compose because I know how much of a pain k8s can be, especially around upgrades, where it has a habit of deprecating older versions of various interfaces. I'll put in the work for someone who's paying me to do it, but I'd rather work on my side project, not on configuring k8s.
> I found Watchtower, but polling just struck me as the Wrong Answer. Both too much overhead to keep pinging for the latest builds, and too slow to actually download the latest build. So I took some hints from Watchtower as to what I needed to do (mount the docker sock as a volume) and wrote a tiny Go server that, when pinged with a shared secret, would cause it to run `docker compose up -d --pull always`.
Is the code for this public? I had the same desire when setting up the server for my personal project but as you mentioned I eventually decided it was OK to just poll every two minutes and I would rather work on the project than on configuring docker. What I would like though is cleaning up the older images that are no longer needed.
> If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.
… aand there is even Ansible playbook for that! Maybe overkill.
Do you have some reference that you can link to? Would be curious to see something about this
Here’s the python module you would use: https://docker-py.readthedocs.io/en/stable/
IME, people dramatically overestimate how hard it is to write any program.
IME people overestimate how hard it should be to write a program, but underestimate how hard it actually is after they're done overengineering the problem to death.
Shameless plug: I’m building https://canine.sh as a way of turning any managed kubernetes cluster into something as easy to use as Heroku.
It’s been super frustrating in the past to be stuck on expensive PaaS vendors, but then rolling a deployment solution from scratch ended up with us trying to stitch together GitHub actions.
Been just using canine for my own projects and I’ve been able to host 4 rails & 1 Rust app pretty seamlessly with it on a $24/month digital ocean server. Would love any feedback!
How would you rate the "overhead" of your tool+k8s (also wht specs does your server have?) because my main use for docker-compose so far has been "host a ton of small apps on 5EUR hetzner instances with just 2GB RAM". If I had a beefier box I'd probably tried k8s again but in my recollection of when I last used it was bad if you had e.g. 5x 2GB RAM/2 CPU vs 1x 8GB RAM on one box.
I wrote my own tool to solve this:
https://harbormaster.readthedocs.io
It's a very thin layer over Compose, basically just auto-pulls git repos and restarts containers if they have any new changes.
Yeah Hetzner is amazing and I think if you’re looking at a single app for costs in the $5 range, it’s probably not worth it to use k8s.
The magic with k8s is being able to easily host third party packages via helm. I often find for anything I want to run in production, I usually want something like sentry, grafana, etc, and those items can get expensive if you try to buy hosted versions. Kubernetes makes it trivial to just host it yourself.
In terms of performance, I’ve gotten reasonable performance out of a $24 kubernetes server, which is 2vCPU + 4GB memory. These days, DigitalOcean and linode don’t charge a markup on their managed kubernetes at all, above their regular VPS pricing.
Heztner is much cheaper than both those options though, and they don’t have a managed kubernetes product
Is canine.sh hosted on Canine?
Yep! I should mention that on the website. The only thing that was innovative about that is doing docker-in-docker builds which is as simple as mounting /var/run/docker.sock:/var/run/docker.sock in the container. Zero downtime deploys make it sensible for it to deploy itself, and then restart. And it really breaks, I still know how to deploy from my laptop to get myself out of a pickle.
Definitely burying the lede that Tealok is building an alternative to docker compose. Interesting
It's called content marketing. IMO this article is content marketing done right since it has useful information even if you don't read to the end.
Does it, though?
It has a very brief introduction to Docker and Docker Compose, but then it mostly has a lot of FUD that's designed to scare people off of Docker Compose but is not remotely convincing to someone who's actually worked with it. The way they frame that pihole example ("Whew! That’s a lot of stuff!") is just silly.
Looking at their website, I think they started out trying to make self-hosting really easy for a barely-technical user [0], which accounts for their attitude towards docker-compose, but it also means that the post was pretty devoid of useful information for someone who actually wants to self-host with the tech that's already ubiquitous today.
[0] https://tealok.tech/
> The way they frame that pihole example ("Whew! That’s a lot of stuff!") is just silly.
Yeah, you're probably right. Originally that line was in there when I had a breakdown of what each line in the docker-compose was doing. My editor thought that was unnecessary - it's unlikely people reading the post would need that kind of breakdown. So I rewrote parts to assume more baseline knowledge. I should have noticed that line and taken it out.
You're right about what we're trying to do, and I agree that the post doesn't really help someone be successful today deploying things. The post is more meant to gauge whether or not I'm alone in having pain deploying a couple dozen services with docker compose on a single box.
I want more people to have the power to host their own services. I think we can do that, but we have to figure out the right thing to build to do it.
Anecdotal, but I learned stuff from this article
Agreed. By default I'm against content marketing. But as a person that has played around a lot in the /r/selfhosted scene, I had to agree with all of the use-case issues mentioned in the article when it comes to managing your self-hosted apps in Docker Compose (and I say this as someone that legitimately enjoys working with Docker).
I'm still not clear from the website what Tealok actually is in terms of a product offering, but I appreciate that the blog post was legit and not just SEO spam.
Looks like a grift. Note the copyright on the Tealok page, then have a gander; https://gleipnirinc.com/investors
Blog post author here - that's the right company name, but not the right company website. Our company doesn't have a website yet.
Looks like we may share a company name with some grifters. That's what we get for making our name an obscure mythological reference.
More interesting is that there are tons of tools that convert docker-compose files to Kubernetes on the fly. Minikube and friends are pretty good and solves all the problems, in a more battle-tested way.
So what is the real business opportunity here?
I'm pretty confused by this article.
It says docker compose is at the "wrong-level of abstraction", but I kept feeling the author was instead expecting docker compose to solve different problems that it was ever meant to solve.
In fact, they seem to be expecting a highly-opinionated, high-level interface which solves problems that I don't think anyone using docker compose in prod should even be worried about.
A lot of concerns seem to be around avoiding spinning up duplicate instances of reverse proxies, databases, and caches. First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern. This is a nuanced and context-dependent issue, but generally it won't even make the list of top-100 performance bottlenecks for most applications. Yet the article takes for granted that the benefits of solving this problem outweigh the many cons of coupling them together.
Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Beyond the specific issues I see, the fundamental attitude seems to be "force everyone to architect their applications in a very specific way, and don't even try to support any use cases which fall outside of it". You need a damn good reason to be that opinionated about these sorts of things, and it by definition will only work well in specific contexts. I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
These are great points, and probably worth their own blog post to answer.
> First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern
Idle threads have very low impact on CPU utilization, probably, if the application is well-behaved (and I expect most databases and caching layers to be well-behaved in this way). The application itself, however, will need memory and the way containers are built prevents the usual de-deplication of system libraries.
> but generally it won't even make the list of top-100 performance bottlenecks for most applications
True, but it makes the short list of "how much stuff can I run on a $100 computer", and it's one of the relatively few concerns an application operator has when they are not the application developer.
> Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
To make self-hosting much simpler. If the container orchestrator doesn't do it, what do you think should do it?
> Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
The DB backups are meant for disaster recovery rather than supporting multiple nodes. I guess that's multiple nodes through time... But, yeah, I agree, docker-compose is not a good fit.
> Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Yeah, agreed, I'll be writing other things to discuss what I think the correct solution should be. I'm curious to find out if other people have existing solutions to the problems I outlined. If it's a solved problem and I just don't know about it, that'd be better.
> I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
It is not, and you're right, it needs discussion.
Ty, all good points
After reading it I don't get what the author is trying to tell me. Docker compose works just fine for what it's made. I can have multiple different docker containers (deployed with compose) all with the same port and just use Traefik to route the requests, without having to change the default ports or even use ipv6. What am I missing?
Interesting, I wasn't aware that Traefik could do this without significant modification to the docker-compose configuration provided by the application developer. I also thought that Traefik required some sort of higher-level container orchestration like Docker Swarm or Kubernetes.
I'll have to look in to that.
Comparing oneself to docker compose is a straw man, when docker’s production option is docker swarm.
Not sure why comments aren't jumping on this more. Swarm is for production. compose is for development. That's always been the case.
And Swarm is barely more complicated than compose anyway. Don't know why people who are claiming they use compose in production just don't learn the few extra steps to do what Docker recommends.
I thought Swarm had been deprecated?
I just took a quick look into it and (classic) Swarm was deprecated and ... replaced by Swarm (mode).
Wait until you find out about 'docker-compose' vs 'docker compose'!
Probably before llms figure it out
I've cobbled together some scripts around docker compose that largely solves these problems for me (https://github.com/mnahkies/shoe-string-server)
It basically looks for a custom external hostname property which is used to generate the haproxy configuration and issue SSL certs.
For ports I personally prefer to just use a consistent container port as much as possible (eg: 80) - I'm sticking a reverse proxy in front of everything anyway so no need for unique ports bound to the host that I won't remember.
Data I basically just bind mount into a structure that I can easily backup from the host. I've also managed to do some major postgres upgrades without any issue (https://github.com/mnahkies/shoe-string-server/blob/master/d...)
I wouldn't call this approach production grade - I'm very much in the use managed Kubernetes/SQL camp there, but it's been working great for my personal needs
Docker Compose/SSH and some custom GitHub Actions to run DB Migrations is simple, straightforward and was more than enough to manage CI deployments all our Apps [1].
Although we're currently migrating to Kamal [2] to take advantage of its nicer remote setup, management and monitoring features.
[1] https://docs.servicestack.net/ssh-docker-compose-deploment
[2] https://kamal-deploy.org
I'm having this problem now with our home automation stack. Compose is super easy to start with but scales poorly and makes things like "I want to access this service at https://foo.bar instead of http://somehost:61234, no HTTPS because the app doesn't support it natively and I don't have time to roll nginx."
Kubernetes makes this so easy once you get past the YAML and abstractions, so I'll probably FINALLY move this stack over to it. Long overdue.
And this is coming from someone who for the longest time recommended that developers new to containers start with compose before considering Kubernetes!
I also think that both docker-compose & k8s & helm are wrong layer of abstraction. I see a lot of people building a opinionated way to run containers and way for them to communicate with another, to request a DB or a redis.
I like to name some such attempts: - .NET Aspire - Avito Plato (home grown PaaS of ¨russian Amazon") - infrastructure layer of our ZIIoT (MES platform on top of k8s)
I’ve been building a Docker Swarm based server dashboard for a couple years now. It should help with some of problems mentioned in the post: https://lunni.dev/
I like the ideas mentioned though! Docker Compose is pretty good, but something more declarative would definitely be a plus. Would love to see where the project goes (and maybe will borrow some things for the future versions of mine :-)
This is a fun problem to see on here (for me) - because I solved a lot of these problems at my last company when I built them a dev enviroment that ran over 30 services, and growing.
It was written as a makefile that wrapped docker compose commands, but that's just because it was the easiest to mvp.
I considered it a "framework for running a large collection of containers locally".
I solved the ports issue by using env vars like a switchboard, with an iPortService=<inside container port> and oPortService=<outside network port unique to network-layer>
Then i used two directories, one for a data layer that ran datastores, and one for app, for apps. You controlled the layers as groups.
You could operate the layers separately, initialize the dbs, and also reset their storage state definitely.
Then I had to solve the "docker ps" output problem, since its unreadable. So, i just split it to 3 commands: state,ports, & running. It outputs what you expect.
I should re-write it into better code and open source it. It was fun to build.
I am particularly happy with docker swarm with traefik as described here: https://dockerswarm.rocks/traefik/
Incredibly easy to setup and manage for usual small scale deployments. I use it as one node swarms, and I have setup - backups - automatic https certificates setup and renewal - automatic upgrades to new images - easy setup of persistence of data on the server
I'm very surprised it is not more popular, with quite some people trying to replicate swarm's features with docker compose, but often in harder to maintain setups.
I love Swarm and don't see the appeal of K8s when something as simple as Swarm exists. I do however run K8s in prod for work and would never run Swarm in prod due to Docker seeming to have its days numbered. Idk where that leaves us aside from ECS. But I also have no need to run something any more robust than ECS in AWS for my workload.
We are moving our EKS workload over to ECS over the next year. I expect needing to down size my team because of it.
One thing K8s is not is cheap. That shit takes a well oiled team or a couple of hot shots so do right. We probably did a lot of what makes it expensive to ourselves by not switching to managed add-ons sooner and never evolving the apps that run in the cluster. I've only been lead for about 5 months now, but I'm finally able to start making significant progress on the necessary evolution that I've been trying to make happen for 2 years before my promotion. The enterprise is a big ship. Takes time to turn. Thanks for reading what turned into a rambling vent session.
I agree it depends on the situation, but there still are some situations (like small apps) where I see swarm as the way to go. Yours is probably different.
Looks like https://dockerswarm.rocks says that the site is deprecated.
https://dockerswarm.rocks/swarm-or-kubernetes/ says "it's not sensible to build a new product using Docker Swarm Mode"
That's indeed the opinion of the author. Note however that at this time all elements used in the setup described on dockerswarm.rocks are maintained. I started using swarm in 2022 and I documented my decision [1], and my reasoning for my kind of needs as not changed. The investment is very low, as well as the risk. Migrating away from swarm should not be very problematic for me, and in the meantime I enjoy an easy to maintain setup. I still think it's better than tweaking a maybe working solution with compose.
I'm not expecting to convince anyone but wanted to share an alternative approach(only applicable to certain setup)
1 https://www.yvesdennels.com/posts/docker-swarm-in-2022/
>> I’m not using YAML, it’s garbage
I suspect I probably felt like that at one time but it seems fine now after repeated use.
I'm using YAML despite it being garbage. (feels more common)
Maybe I am a boiled frog, but I personally like it. TOML looks fine for simple cases, but yaml is fine there as well. Also, yaml has very clean approaches for dealing with literal text. For instance, if your yaml file needs to include an xml file, a json file and a bash script, it all ends up being very readable.
I may consider using HOCON [0] if I see any traction, bust after writing a lot of YAML and even making my own YAML-driven tools, I feel its shortcomings are overstated. I got bit by corner cases maybe three or four times, and they didn't take long to debug.
[0] https://github.com/lightbend/config/blob/main/HOCON.md
For the authors, I suggest a TL;DR with some basic examples. People tune out pretty quickly. You want the 10s take away to be "I like it" and "this person has really thought through it". You have achieved the second objective with what you have.
Strangely enough I'm at the complete opposite end of the author. Docker Compose imo operates exactly at the right abstraction layer and to me the author never really explains what the problems with Docker Compose actually are.
Imo Docker Compose absolutely should not handle databases and web servers. It should orchestrate containers. It's up to the user what kind of containers should be orchestrated.
Also the author somehow never got the idea to run the reverse proxy in a container? You don't need to do any port mapping then.
No mention of Kubernetes (and k3s, etc.)
My first thought as well.
To save someone the click, this is a content-based marketing article.
TL;DR: Author believes docker compose is too complicated, and has a grudge against YAML for some reason. Author proposes an alternative configuration syntax that hides implementation details behind named templates.
So despite what the author wants us to believe, this isn't ACTUALLY a replacement for docker compose. This is yet another "easy-to-use" container orchestration product where there's extra upstream development between you and the docker image. Docker compose can run anything you can stuff into a container. This cannot, without some additional development. That may be value in that, but I'm not sure blasting docker compose as old and busted right out of the starting gate is a terrific marketing strategy.
Weird editorialization. I included a TL;DR at the top, you could have just copy-pasted it.
"Docker-compose is a tool for working with Docker containers. It solves very real problems with deploying complex applications. By itself it is not enough to make self-hosting applications simple enough for the mass-market."
I manage production with docker compose just fine. This article is not great.
One thing I really dislike about compose is that its a CLI tool but it doesn't exist in library form with a similar API.
I wrote a backup tool for compose/swarm based setups based on restic but ended up having to reimplementing half of traefik because there was no more straightforward API and compose really is just a thin layer over docker.
i've been at the edges of docker-compose. We were always conceptually recreating kubernetes. I'm so happy we've got k8s now and sparingly use some operators where needed, couldn't be happier.
The obvious example was certificate management. Cert-manager alone is almost worth setting up K8s if you have many ingresses/fqdns.
I'm moving into self-hosting for myself (photos, books, GPS tracks, exercise tracking) and so far Docker Compose has been perfect for me.
I'm using Synology NAS which has a nice GUI manager for Docker-Compose. So I put all the data volumes on btrfs, and I snapshot them every day. Then I sync snapshots to Wasabi via duplicacy.
It works amazingly well. No problems whatsoever. I guess having a nice front-end that shows the overall overview would be nice, but I can live without it.
I don’t get why docker doesn’t include a simple command (docker backup volumexyz mybackup.tar.gz) that creates an archive from a volume. Why do I have to do that myself by mounting the volume in a second container? Ok, there might be volumes that can’t be backed up (mounted sockets for example), but that could just give an error message.
Docker is simply a packaging format with docker compose being a wrapper for docker run. Author seems to confuse general application issues with docker's.
Article wasn't convincing enough about docker compose's shortcomings. The linked github's README even less convincing. But to each his own.
They don't mention it in the article, but by default ports 80 and 443 require elevated privileges.
There's some (namespaces) knob to avoid that, but the lack of nod to it makes me worried.
OTOH containers as security boundary is iffy. But I still like them to not be root in case of compromise
If docker compose is not enough I would suggest to look into some of the lightweight k8s distributions like k3s, with low ressource consumption and able to operate on a single node.
Curious! I think of docker-compose as squeezing a cluster onto a workstation—not exactly a tool you'd look to for anything other than home servers or conveniently-low workloads.
Speaking as someone who has deployed docker-compose apps to hundreds of servers at one company... It's fine. Depends what you're doing.
You know you can run multiple processes inside a single container, right?
The solution to the complexity of applications that are distributed as collections of interdependent containers is to put all of the different pieces inside a single container. This is equivalent to what people did before docker existed, and it's still perfectly viable today.
There's no need to over-complicate things.
Is there a popular project which puts systemd and multiple processes in a container? I've seen only single process containers so far.
LXD/Incus for hypervisor-for-containers use-case.
Podman for "like Docker, but made by Linux people". It supports both application containers and system containers (which have systemd as PID 1).
systemd-nspawn might be what you are looking for: https://wiki.debian.org/nspawn
scaling docker compose to multiple machines is traditionally the realm of kubernetes. I’m not seeing any suggestion here that’s better than dropping in k3s
In summary:
Accidental complexity, accidental complexity, accidental complexity and when we use a tool that is designed for problem A for a problem B it is not sufficiently dealing with complexity. This is why we need a new tool.
¯\_(ツ)_/¯
While a bit of a hot take, you're not wrong. We need something that's less scalability focused than Kubernetes/Mesos/Docker Swarm but that doesn't put too much burden on application developers. Something that focuses on being secure, reliable, and understandable, in that order. I'm not aware of anything going for that niche. That means a new tool is in order.
I think we need a composable system but I am not sure if the current frame where this problem and these tools exist is good enough. We might need to rethink how we handle access and usage patterns well. I only have wrong answers. Docker compose is amazing for local dev env, k8s is terrible for production. These are my experiences with this domain.
It is enough for me, and the title doesn't mention that this post promotes the alternative.
I know this is content marketing fluff for their own product, but complaining that Docker Compose isn't a full-fledged web application platform is idiotic. Use something like Dokku if you need such a solution.
or, just use your hardware without unnecessary isolation.
[flagged]
Docker Compose is … not for that use case. Docker Swarm is for hosting, but really only for trivial cases.
Anything more complex deserves a full container orchestration platform.
Containers were a mistake. This is all radically more complicated than it needs to be. Running a computer program is not that complicated.
Being able to tell a fellow developer "do docker compose up and you can work" is a lot better than navigating them through the installation of a bunch of tools, each with their own quirks.
I'm not convinced anyone below google needs kubernetes, but containers for quickly setting up and running something on my machine are a blessing.
Agreed. Running a modern "computer program" is more than just executing a single binary file.
When you have a open-source application with five or six moving parts (monitors, backend, a database, etc.) being able to deploy it to a VPS with a single docker compose and have them all containers act on an internal dockerized network without conflicting ports, etc. is a GOD SEND.
Containers are a convenient work-around for the problem where programs have incompatible dependencies, and additionally the problem where security isn't as good as it should be.
For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful. With docker containers, you can just containerize each application with its own Python version and have a consistent environment that you can run on lots of different machines (even on different OSes).
They're also a lot more convenient than having to go through the arcane and non-standard installation procedures that a lot of software applications (esp. proprietary ones) have.
Yeah, honestly it kinda sucks that we're adding this layer of inefficiency and bloat to things, but these tools were invented for a reason.
> For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful.
This is because the Linux model of global system wide shared dependencies is stupid, bad, and wrong. Docker and friends are a roundabout a way of having a program shipping its dependencies.
The Linux model works fine (very well, in fact, because of less HD space and much more importantly, less memory used for shared libraries) for programs that are normally included in the Linux distribution, since the whole thing is built together by the same organization as a cohesive whole. If every random little 20kB utility program were packaged with all its dependencies, the bloat would be massive.
It doesn't work very well for 3rd-party software distributed separately from the OS distro and installed by end-users.
The problem I've seen is that, while pre-Docker there was really nothing preventing ISVs from packaging their own versions of dependencies, they still only targeted specific Linux distros and versions, because they still had dependencies on things included in that distro, instead of just packaging their own. The big thing is probably glibc.
As I recall, Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
> because of less HD space and much more importantly, less memory used for shared libraries
Literally not in the Top 1000 problems for modern software.
> Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
Not really. A 20 year old piece of windows software prettt much “just works”. Meanwhile it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
>A 20 year old piece of windows software prettt much “just works”
No, it only works because Windows basically included something much like WINE (they call it WoW) in Windows, so old pieces of software aren't running on the modern libraries.
>it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
Sure you can, with Docker. It's effectively doing the same thing Windows does with WoW.
It is an abstraction.
Many abstractions are very bad. Stacking bad abstractions on bad abstractions is why modern software is so slow, laggy, and bloated.
What would be your solution to running, say, 15 different Python applications on the same machine, each requiring a unique set of library versions?
https://github.com/claceio/clace is a project I am building for that use case. Run multiple apps on the same machine, Clace acts as an application server which manages containers https://clace.io/blog/appserver/
Create 15 pex files. Or possibly 15 PyInstaller executables. Then simply run them like normal programs.