![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Containers and the future of Unix
Mar. 3rd, 2018 12:09 amA lot of my speculations concern the future of new, alternative operating systems which could escape from old-fashioned, sometimes ill-conceived models and languages.
But I do spend some time thinking about what is happening with Linux, with FOSS Unix in general, and especially with container technologies, something I deal with in my current and recent day-jobs more and more.
One answer to legacy nastiness for years now has been to virtualise it. Today, that's changing to "containerise it".
There is a ton of cruft in Linux and in the BSDs and so on which nobody is ever going to fix. It's too hard, it would break too much stuff... but most of all, there is no commercial pressure to do it, so it's not going to happen.
I can certainly see potentialities. There are parallels that run quite deep.
For instance, consider a few unrelated technologies:
- FreeBSD jails and Solaris Zones. Start here.
They indirectly evolved into LXC, the container mechanism in the Linux kernel which gets relatively little attention. (Docker has critical mass, systemd namespaces are trendier in some niches, CRIO is gaining a little bit of traction.)
Docker now means Linux containers are a known thing, already widely-used with money being poured into their R&D.
Joyent, a company with some vision, saw a chance here. It took Illumos, the FOSS fork of Solaris, and revived and modernised some long-dead Sun code: lxrun, the Linux runtime for Solaris. Joyent SmartOS is therefore a tiny Solaris derivative -- it runs entirely from RAM, booted off a USB stick, but can efficiently scale to hundreds of CPU cores and many terabytes of RAM -- which can natively run Docker Linux containers.
You don't need to run a hypervisor. (It is a hypervisor, if you want that.) You don't need to partition the machine. You don't even need a single copy of Linux on it. You have a rack of x86-64 boxes running SmartOS, and you can throw tens of thousands of Docker containers at them.
It gives capacities and scalability that only IBM mainframes can approach.
Now, if one small company can do this with some long-unmaintained code, then consider what else could be done with it.
- Want more resilient hosts for long-lived containers? Put some work into Minix 3 until it can efficiently run Linux containers. A proper fully-modular-all-the-way-down microkernel which can detect when its constituent in-memory services fail and restart them. It can in principle even undergo binary version upgrades, piecemeal, on a running system. This is stuff Linux vendors can't even dream of. It would, for a start, make quite a lot of the Spectre and Meltdown vulnerabilities moot, because there's no shared kernel memory space.
Unlike Darwin and xnu, it's a proper microkernel -- no huge in-kernel servers for anything here. (Don't eve try to claim WinNT is a microkernel or I will slap you.) Unlike the GNU HURD, it's here, it works, it's being very widely used for real workloads. And it's 100% FOSS.
- Want a flexible cluster host which can migrate containers around a globe-spanning virtual datacenter?
Put some work into Plan 9's APE, its Linux runtime. Again, make it capable of running Linux containers. To Plan 9 they'd just be processes and it was built to efficiently fling them around a network.
I have looked into container-hosting Linux distros for several different dayjobs. I can't give details, but they scare me. One I've tried has a min spec of 8GB of RAM and 40GB of disk per cluster node, and a minimum of 3-4 nodes.
This is not small efficient tech. But it could be; SmartOS shows that.
- Hell, more down to earth -- many old Linux hands are deserting to FreeBSD in disgust over systemd. FreeBSD already has containers and a quite current Linux runtime, the Linuxulator. It would be relatively easy to put them together and have FreeBSD host Linux containers, but the sort of people who dislike systemd also dislike containers.
Not everything would run under containers, sure, no. But they're suitable for far bigger workloads than is generally expected. You can migrate a whole complex Linux server into a container -- P2V migration as was once common when moving to hypervisors. I've talked to people doing it.
Ubuntu LXD is specifically intended for this, because Ubuntu isn't certified for SAP, only SUSE is, so Ubuntu wants to be able to run SLE userlands. Ditto some RHEL-only stuff.
But what if it doesn't work with containers at all?
Well, as parallels...
[1] A lot of Win32 stuff got abandoned with the move to WinXP. People liked the new OS enough that stuff that didn't work got left behind.
[2] Apple formalised this with Carbon after the NeXT acquisition. The MacOS APIs were not clean and suitable for a pre-emptive multitasking OS. So Apple stripped them out and said "if you use this subset, your app can be ported. If you don't, it can't."
Over the next few years, the old OS was forcibly phased out -- there is a generation of late-era gigahertz-class G4 and G5 PowerMac that refuses to boot classic MacOS. Apple tweaked the firmware to prevent it. You _had_ to run OS X on them, and although versions >= 10.4 could run a Classic MacOS VM, not everything worked in a VM.
So the developers had to migrate. And they did, because although it was a lot of work, they wanted to keep selling software.
It worked so well that in the end the migration from PowerPC to Intel was less painful than the one from classic MacOS to OS X.
So maybe Linux workloads that won't work in containers will just go away, replaced by ones that will -- and apps that play nice in a container don't care what distro they're on, and that means that they will run on top of SmartOS and FreeBSD and maybe in time Minix 3 or Plan 9.
And so we'll get that newer, cleaner, reworked Unix after all, but not by any incremental process, by a quite dramatic big-bang approach.
And if there comes a point when it's desirable to run these alternative OSes for some users, because they provide useful features in nice handy easy ways, well, maybe they'll gain traction.
And if that happened, then maybe some people will investigate native ports instead of containerised Linux versions, and gain some edge, and suddenly the Unix world will be blown wide open again.
Might happen. Might not. It's not what I am really interested in, TBH. But it's possible -- existing products, shipping for a few years, show that.
But I do spend some time thinking about what is happening with Linux, with FOSS Unix in general, and especially with container technologies, something I deal with in my current and recent day-jobs more and more.
One answer to legacy nastiness for years now has been to virtualise it. Today, that's changing to "containerise it".
There is a ton of cruft in Linux and in the BSDs and so on which nobody is ever going to fix. It's too hard, it would break too much stuff... but most of all, there is no commercial pressure to do it, so it's not going to happen.
I can certainly see potentialities. There are parallels that run quite deep.
For instance, consider a few unrelated technologies:
- FreeBSD jails and Solaris Zones. Start here.
They indirectly evolved into LXC, the container mechanism in the Linux kernel which gets relatively little attention. (Docker has critical mass, systemd namespaces are trendier in some niches, CRIO is gaining a little bit of traction.)
Docker now means Linux containers are a known thing, already widely-used with money being poured into their R&D.
Joyent, a company with some vision, saw a chance here. It took Illumos, the FOSS fork of Solaris, and revived and modernised some long-dead Sun code: lxrun, the Linux runtime for Solaris. Joyent SmartOS is therefore a tiny Solaris derivative -- it runs entirely from RAM, booted off a USB stick, but can efficiently scale to hundreds of CPU cores and many terabytes of RAM -- which can natively run Docker Linux containers.
You don't need to run a hypervisor. (It is a hypervisor, if you want that.) You don't need to partition the machine. You don't even need a single copy of Linux on it. You have a rack of x86-64 boxes running SmartOS, and you can throw tens of thousands of Docker containers at them.
It gives capacities and scalability that only IBM mainframes can approach.
Now, if one small company can do this with some long-unmaintained code, then consider what else could be done with it.
- Want more resilient hosts for long-lived containers? Put some work into Minix 3 until it can efficiently run Linux containers. A proper fully-modular-all-the-way-down microkernel which can detect when its constituent in-memory services fail and restart them. It can in principle even undergo binary version upgrades, piecemeal, on a running system. This is stuff Linux vendors can't even dream of. It would, for a start, make quite a lot of the Spectre and Meltdown vulnerabilities moot, because there's no shared kernel memory space.
Unlike Darwin and xnu, it's a proper microkernel -- no huge in-kernel servers for anything here. (Don't eve try to claim WinNT is a microkernel or I will slap you.) Unlike the GNU HURD, it's here, it works, it's being very widely used for real workloads. And it's 100% FOSS.
- Want a flexible cluster host which can migrate containers around a globe-spanning virtual datacenter?
Put some work into Plan 9's APE, its Linux runtime. Again, make it capable of running Linux containers. To Plan 9 they'd just be processes and it was built to efficiently fling them around a network.
I have looked into container-hosting Linux distros for several different dayjobs. I can't give details, but they scare me. One I've tried has a min spec of 8GB of RAM and 40GB of disk per cluster node, and a minimum of 3-4 nodes.
This is not small efficient tech. But it could be; SmartOS shows that.
- Hell, more down to earth -- many old Linux hands are deserting to FreeBSD in disgust over systemd. FreeBSD already has containers and a quite current Linux runtime, the Linuxulator. It would be relatively easy to put them together and have FreeBSD host Linux containers, but the sort of people who dislike systemd also dislike containers.
Not everything would run under containers, sure, no. But they're suitable for far bigger workloads than is generally expected. You can migrate a whole complex Linux server into a container -- P2V migration as was once common when moving to hypervisors. I've talked to people doing it.
Ubuntu LXD is specifically intended for this, because Ubuntu isn't certified for SAP, only SUSE is, so Ubuntu wants to be able to run SLE userlands. Ditto some RHEL-only stuff.
But what if it doesn't work with containers at all?
Well, as parallels...
[1] A lot of Win32 stuff got abandoned with the move to WinXP. People liked the new OS enough that stuff that didn't work got left behind.
[2] Apple formalised this with Carbon after the NeXT acquisition. The MacOS APIs were not clean and suitable for a pre-emptive multitasking OS. So Apple stripped them out and said "if you use this subset, your app can be ported. If you don't, it can't."
Over the next few years, the old OS was forcibly phased out -- there is a generation of late-era gigahertz-class G4 and G5 PowerMac that refuses to boot classic MacOS. Apple tweaked the firmware to prevent it. You _had_ to run OS X on them, and although versions >= 10.4 could run a Classic MacOS VM, not everything worked in a VM.
So the developers had to migrate. And they did, because although it was a lot of work, they wanted to keep selling software.
It worked so well that in the end the migration from PowerPC to Intel was less painful than the one from classic MacOS to OS X.
So maybe Linux workloads that won't work in containers will just go away, replaced by ones that will -- and apps that play nice in a container don't care what distro they're on, and that means that they will run on top of SmartOS and FreeBSD and maybe in time Minix 3 or Plan 9.
And so we'll get that newer, cleaner, reworked Unix after all, but not by any incremental process, by a quite dramatic big-bang approach.
And if there comes a point when it's desirable to run these alternative OSes for some users, because they provide useful features in nice handy easy ways, well, maybe they'll gain traction.
And if that happened, then maybe some people will investigate native ports instead of containerised Linux versions, and gain some edge, and suddenly the Unix world will be blown wide open again.
Might happen. Might not. It's not what I am really interested in, TBH. But it's possible -- existing products, shipping for a few years, show that.
