![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Apparently I'm renowned for my enthusiam for ancient software. I find it fun to play with kit that I couldn't dream of affording when it was current, in my youth. And it's very instructive to compare it with new stuff. But I also use some of it for actual paying work.
I run MS Word 97 under WINE to do my own writing. Not because I need any features in that version -- I don't. I'd be perfectly happy with Word 6, which is functionally near-identical, smaller and faster. The only reason I don't run Word 6 is that it doesn't support the scroll wheel or proportional thumbs -- the widget in the scrollbar is always a square so you get no visual feedback on how long your document is and what relative size chunk you're looking at. That's very useful info.
But I hunted, for years, for Word 6 for NT, the 32-bit version. Word 6 being a Windows 3 app. Why? Because the NT version supports long filenames and they're jolly handy. But not proportional thumbs or the scroll wheel. There's also the small convenience that Word 97 uses the same native file format that was used until Word 2003, and everything can import it: LibreOffice, Pages, AbiWord, you name it. The previous versions' file format -- also called .DOC -- is far less widely-supported, but then, it was phased out 22 years ago.
I've tried Word 6 for DOS, WinWord 6 (for Win3), Word 6 NT, Word 95 (v7), Word 97 (v8), Word 2000, Word XP, and Word 2003. Hunted 'em all down, tried on 32-bit and 64-bit Windows and WINE.
I have really looked deeply into this, mainly out of curiosity but also because I made my living using this tool.
One WINE compatibility test for me is: can I install the service releases under WINE? 97 yes, 2000 no, XP no, 2003 yes.
That falls down on Word 95. Why? Because there were no service releases.
This is an app I know inside-out and use hard and heavily, mainly because of the outlining feature that no other modern editor still offers. MS Word has gained no new features I need or use since 1993.
That was the first version that ran on both 16-bit and 32-bit Windows. Then came Office 95, 32-bit only, with zero functional changes but a new look and feel: square toolbar buttons, font formatting in Windows title bars, stuff like that. Trivial.
But Word 97 had no new functionality either, just a new .DOC file format, and yet Office 97 needed 3 service releases. The first release didn't even write a Word 6 .DOC file when you asked it to. It wrote an RTF file but gave it a .DOC extension. The older versions were smart enough to detect the format automatically and import it, so nobody noticed before release. How come?
The point of this tedious little aside ;-) being that the app has grown massively since then and it's now about 10× the size, probably more -- I haven't checked recent versions (since 2007) because I detest the ribbon interface.
I think a full install of Word 97 is 14 MB. I thought Office 97 was bloated when it came out. 14 MB may have been bloated in 1997 but today it seems svelte, almost minimalist.
I pick this one app because the additional functionality is amazingly trivial -- Word 97 has a virtual yellow highlighter pen and AFAIK that is the only functional addition. Word 2000 allows tables to be nested inside table cells, and AFAIK that is it.
My point being, it's a very mature codebase, and yet it's still bloated hugely in the ~20y since it last gained a major new feature, and it's also picked up major bugs in that time, so you really do want the service releases.
The greater point being that tech support is now even worse than when I left it, because the products are massively bigger, have a lot more functionality but not as much as you might expect in many places, and things originally introduced to reduce size and complexity, such as shared libraries, have in fact done the reverse and hugely increased size and complexity so that now we need major new OS features such as containers simply so that multiple apps with multiple specific versions of their shared libraries can co-exist on the same OS without causing horrible clashes.
Plan 9, I recently learned, doesn't have shared libraries. So when it was new, the Unix people of the time decried it, as it was missing something they thought important.
Plan 9's designers were right, but nobody now recognises that, because it's so long ago that the people inventing containers to fix shared-library versioning clashes don't know that there was a major successor product to Unix that identified shared libraries as a problem and eliminated it.
It was a generation ago. My colleagues sitting behind me, diligently testing an enterprise Linux distro, probably have never even heard of Plan 9.
A quondam GNU spokesman on Twitter spent a while last month lambasting me as a know-nothing idiot, and got a gang of his mates in to harangue and pillory me, because I told him that Linux is a Unix.
He thinks that's obviously wrong. He has probably never seen another Unix in his life, so obviously the old fart disagreeing with him is an idiot.
But to get to my larger point...
Tradeoffs were made in the design of Unix in the 1960s, to keep it very simple, because it was designed for a very low-end system: a PDP-7 with 4 kilowords (9 kB) of RAM.
Then in the 1970s it was rewritten in C, and further different tradeoffs were made in C to keep it very simple, to make it cross-platform and yet small and easy to compile.
Then in the 1980s, yet more tradeoffs were made in Minix to make it small and simple enough for students to understand, and yet still cross-platform on very low-end machines with no MMU etc. Shades of the original 1960s design, but nobody much noticed that.
Then in the 1990s, tradeoffs in the design of Linux were made, to make it simple to implement and to get it working in a reasonable timeframe, because commercial Unix was too expensive and Minix was too limited and Dr Andy Tanenbaum ("AST") wouldn't accept complex patches to make it 386-aware, give it proper virtual memory, etc.
And Torvalds and his followers vigorously defended those compromises, because they were pragmatic and meant it was doable and understandable and could be made to work, whereas the HURD couldn't.
Well now, we're further from the introduction of Linux (1991, 28y ago) than Linux was from Thompson & Ritchie's original PDP-7 Unix (1969, 22y before Linux). 50 years. Two human generations. [EDIT: corrected arithmetic error -- thank you Stefan!]
The sensible pragmatic compromises to get a simple monolithic kernel actually working and reasonably performant on a 25 MHz 80386 now mean a vast and vastly complex monolithic OS for gigahertz-class hardware with dozens of cores, with hundreds of thousands of full-time programmers around the world working flat out just to keep the thing working, patched and broadly generally mostly kinda safe-ish.
IMHO, AST was right.
Linux was Unix reimplemented as FOSS, from scratch, exactly as it had been done 2 decades earlier, partly because politics stopped the GNU project adopting the BSD kernel in the late 1980s. They looked at it and discarded it. If they'd adopted it, the GNU Project would have had a complete, working, FOSS Unix by about 1989 or 1990 and Linux would never have happened:
They learned nothing from proprietary Unix. They improved nothing. They just redid it, the same, but as FOSS.
It was the last good chance to make some structural improvements, and we blew it.
So now, we have a maintenance nightmare. A global catastrophe of software project management, but it's enabled an industry worth hundreds of billions of dollars, which pays to keep the entire cancerous mess working.
And not a single person in the entire industry has the guts to point out that it's a catastrophic mess, because by wasting all the finest programming minds of an entire human generation (bar the small fraction employed by Microsoft to work on Windows, which is just as bad, because it didn't even learn from Unix), it still works.
As the human race busily causes catastrophic global ecosystem collapse, which has happened in the same 25y period, we have failed to materially improve on the computing state of the art from the beginning of the 1970s.
Two generations of progress lost, because if you throw enough money at the problem, even a mistake can be made to work. As TWA's Paul E. Richter Jr. said, "Give me enough power and I can fly a barn door." A monolithic 1970s OS can power the entire world, if the programming effort of the entire world is devoted to keeping it working.
But it's not efficient. C enthusiasts think it's efficient and low-level. It isn't. There are tens of millions of lines of error-checking, test frameworks and all sorts, and much of that is because of limitations in C, because of the monolithic kernel, because of those shared libraries and the very complex recursive package managers needed to suck in the millions of dependencies and sensibly provide what's needed.
This sounds absurd, but here are some citations.
A Google project containing some 2000 files and 4.2MB of source sucked over 8 gigabytes of code through the C compiler due to nested #includes.
Firefox depends on 122 packages including libTIFF, but it can't render TIFFs.
Maybe you will protest: "But that's just how software is! It needs to be that way! Modern software has to cope with so many things!"
"Hello world" in C++ involves 18,000+ lines of code. Getting a modern C++ compiler to build itself on modern hardware can take an hour or so.
Compare with Oberon, where a modern, TCP/IP-networked, just about web-capable OS written in Oberon and also called Oberon was implemented in about 12,500 lines of code.
The Oberon compiler can build itself in 3 seconds on a 25 MHz CPU.
Pick a different but still low-level language and an entire OS, far more functional than Linux was back when it became self-hosting, takes less code than "hello world".
This is not just a minor difference of degree. This is very badly wrong. This is a trillion-dollar issue. It is not an exaggeration to say this kind of issue threatens the continuing existence of our civilisation.
"But microkernels are so inefficient! All those context switches!"
I am absolutely sure that microkernels would have been less efficient, a human generation ago when the 80386SX was the most powerful computer that most people could afford.
But now, if we'd done Free UNIX properly, with modular codebases, isolated services instead of monolithic kernels, integrated networking and clustering -- all known, established, standardised stuff that was working and in production before Linux 0.01 -- I am very confident that now, we'd have something which required a very great deal less maintenance.
It's an alternate history. I can't put numbers on it because it didn't happen.
But let's look at the tech changes in that time span.
Between the invention of Unix and the invention of Linux, there were multiple generation shifts:
Then came Linux.
Since Linux came along, we've gone through about half as many significant technological shifts, I'd say.
Basically, the PC industry moved to 32-bit Windows 25 years ago and hasn't substantially changed since. NT took over in the form of Windows XP in 2002-2003, around the same time as Mac OS X 10.2, the version that made Mac owners start to want to switch.
This is also around the time that Web commerce starts to drive adoption of Linux server farms, aided by VMware.
By which point, the entire computer industry was running 32-bit C-based multiuser OSes: Unix (including Mac OS X), Linux and Win NT. But their application ecosystem was shared with the previous generation of OSes from a decade before, and the underlying designs of those OSes are from the 1970s: Unix and DEC's VMS.
The industry attempted to move to new, home-grown OSes, such as OS/2 and Apple's Copland. It failed. So instead, OS design actually took a step backwards when 32-bit micros became usefully able to run 1970s minicomputer OSes, which emulated the previous generation of 16/32-bit microcomputer OSes well enough that users could keep their apps.
Since then, I would argue there has been no big shift. Just incremental performance and capacity improvements.
We had total computer generation shifts every decade or so from the invention of the reprogrammable digital computer just after WW2, for 2 human generations -- about 50 years. Everyone in the industry was used to it. It was normal.
It happened to me personally about 2-3 times between the time when I got interested as a schoolchild in the early 1980s: from a 48K Spectrum and cassettes, then microdrive, to a Spectrum 128 and MGT DISCiPLE and floppy drives, to an Amstrad PCW, to an Acorn Archimedes, to a 386 laptop running OS/2 2 (because I couldn't get Slackware to install), and then to a 486 running Windows 95.
Every time, I had to totally relearn the entire OS, learn new languages, and replace (or rewrite) every single program I used.
But since the mid-1990s, we've just been iterating the same design.
Now, we have a generation of programmers who have never in their lives, since childhood, ever seen a shift such as the ones I endured multiple times. The thought of throwing out all their software and starting again is unimaginable, inconceivable to them.
But before the mid-1990s, the entire computer industry did just that, at least every other hardware generation from the 1950s to the 1990s.
Then we stopped. Not because we'd achieved perfection -- we very definitely hadn't -- but we had stuff that was good enough, which with effort worked and kept working.
So we stopped developing new stuff, and just kept polishing the same things, more and more. Now they're very shiny indeed. As Douglas Adams said:
"It is very easy to be blinded to the essential uselessness of [their products] by the sense of achievement you get from getting them to work at all. In other words—and this is the rock solid principle on which the whole of the Corporation's Galaxy-wide success is founded—their fundamental design flaws are completely hidden by their superficial design flaws."
The result is that we've accumulated an almost inconceivably vast amount of cruft, but everyone thinks that that is perfectly normal. Even old hands like me. We use tools that are at least a quarter of a century old on OSes whose design is twice as old as that and we like it!
That is not normal, even for this industry.
There's been no substantial, revolutionary innovation since the GUI was invented in the late 1970s, and all we did with that was bolt that onto 1970s minicomputer OSes.
This has to end.
About 2012, pursing an retrocomputing story that hadn't been done to death already, I came across Lisp Machines and a war that has been successfully entirely covered up. That in the 1970s and 1980s, on the new class of personal workstations, there was a battle between two entirely different ways to write software and operating systems.
One side used radical high-level languages and designed special, elaborate processors to run them natively. The entire OSes were single, dynamic entities, whose code could be inspected and modified as it ran. No compilers, no linking, no static binaries. A whole new world, a wonderful programmer's playground.
The other side adapted a minimalistic decade-old minicomputer OS, written in the most minimalistic language around, bolted a clumsy GUI on top, and ran it on stripped-down, minimalist processors. It wasn't fancy or clever, but it was cheap and fast.
Cheap and fast won, of course.
Now, you never even hear that there was another way.
The rich dynamic languages hang on as programming tools on the not-even-slightly minimalistic descendants of those fast-and-cheap boxes, after they merged with IBM-compatible PCs. Even the sleek fast chips went away.
Only a few old folks are nostalgic for the good old times. 30, 40 and 50somethings, including my cohort, rhapsodize instead about 1980s home computer OSes: Amiga OS, ST GEM, RISC OS. They were small, fast and simple, and stomped all over PC OSes of the time. But the PC OSes won, and whenever those '80s OSes survive -- such as AROS or modern RISC OS -- they're hopelessly crippled and compromised compared to their modern rivals.
I wondered if there was anything that had the strengths but not the weaknesses. That was small, simple and fast, but also clean and elegant, which ran on modern hardware, on different architectures, and could exploit multiple CPUs, 64-bit machines with many gigabytes of RAM and so on.
So I found it very interesting that, after years of reading about this field, that I basically stumbled across Oberon and found that it fits what I was looking for remarkably well.
It's very small and simple (cf. Sinclair QDOS and kin).
It's FOSS (cf. QDOS descendants Minerva and SMSQ/E, or AROS, or AFROS, or RISC OS).
It has a current version and it does actually run properly on modern hardware, not just in an emulator. (Unlike all those, except RISC OS.)
It's a clean, simple design, done in 1 language. (Unlike all of them, really.)
It was actually used, by real people in the real world, for years. It has 3rd party apps, for instance. (Like all the above, albeit less so.)
It's very obscure but among those that know about it, it's widely admired.
It's renowned for its elegance.
It was very influential in its day. (Mostly, unlike all of them.)
It has survived in some places by becoming just another language-stroke-development environment for more mainstream OSes (like Smalltalk and Lisp before it, but unlike the others).
Unlike, say, Lisp, it's not arcane: its relatives Turbo Pascal & Delphi were once some of the world's favourite development tools, used worldwide by tens to hundreds of thousands, and so admired within MS that they went to great lengths to poach their programming lead.
Its most modern offshoot, A2/Bluebottle, is SMP-aware and can be used on the Web, a personal acid test of mine. It's only about as usable in this role as RISC OS, i.e. not very, but that it does it at all is impressive.
Haiku (the FOSS BeOS) ticks a lot of these boxes too, but it's x86 only, is only partially compatible with its predecessor, while being bigger and slower, and as a big, rich, modern 1990s OS that aspired to go mainstream, it's not some nice simple student-friendly thing.
For the avoidance of doubt, I am not saying that we should all just abandon Unix and switch to A2, or Lisp Machines, or Smalltalk boxes, or anything like them.
My thinking is more like this:
64-bit didn't do it. Barely a blip in the end, remarkably.
Java didn't do it.
SSDs didn't do it. Smartphones and tablets didn't do it. They run modified existing OSes, concealed behind simplified UIs and a lot of sandboxing.
What's left? Persistent memory, AKA nonvolatile RAM.
On servers, its impact could be quite minor. But setting servers aside, on client devices, it could be very radical indeed. If you have a terabyte or so of cheap fast main memory that's nonvolatile, why would you need a disk drive? Turn the computer off and when you turn it back on, it picks up where it left off. No suspending, no resuming. No shuffling data from RAM onto flash disks and back. No flash. No disks.
But if you have no disks, whereas you certainly can emulate a hard disk by partitioning your NVRAM and formatting a bit of it, why would you? There's no need. There's no need for a resident filesystem at all. The computer never normally boots, unless it's updated; it just stops and starts.
Can you imagine Unix, the OS where "everything is a file", if there is no filesystem and no files?
Can you imagine Windows without a C: drive? With no drives at all?
It doesn't really work. Their central metaphor is the notion of files on disk, which either contain binaries or data. Binaries are "loaded" into RAM. They read data from other files -- config, working matter -- then they put it back when they're done.
Eliminate that and there isn't much left.
So, if that happens, what will the resulting systems look like?
I went looking for previous systems that didn't have the filesystem-centric metaphor. I found just four good solid examples.
This leaves just two:
The central working abstractions for both types of machine are in-memory structures, specific to their programming languages, rather than the filesystems which are the bedrock concept in Windows and Unix.
This seems to me to be a fairly good fit for a machine with a single level store of nonvolatile memory.
Yes, of course they will still need to support filesystems. We're going to want to back these things up sometimes, install new versions and additional functionality, and we'll want to support removable media, and connections to existing remote servers, etc.
The key point here is that although filesystems are present, that the filesystem won't be the central, defining abstraction.
Before we got caught in the software tar pits of the mid-1990s, it was normal for old technology to gradually atrophy away. Those mid-1990s PCs evolved from early-'80s home computers with a ROM BASIC which loaded and saved to an audio cassette drive via a dedicated port, but that functionality's long gone from absolutely all PCs today -- and nobody misses it.
Another comparison:
Early mainframes had vast, complex front panels for their operators, festooned with the famous "blinkenlights". In later ones, they just gradually went away, almost unnoticed.
Minicomputers and the first mid-1970s microcomputers went through phases of toggle switches for entering code bit by bit, and a row of lights for output. This was the lowest common denominator of programming. Then the switches went away, replaced by teletypes and paper tape. Then the teletypes went away, replaced by glass terminals, and paper tape was replaced by floppy drives.
Although VMS, Windows and Unix all have roots in the era of front panel switches, paper tape and teletypes, they either support them very poorly or not at all today. It's just not relevant. But they didn't just disappear overnight. They gradually went from central and inseparable to less important to barely-maintained code nobody uses any more to a novelty feature that gets made into amusing Youtube videos.
Now, if you're rich, you have a terabyte of nonvolatile storage in your pocket, driven by multiple 64-bit RISC cores, with which you interact via a huge 24-bit graphical framebuffer and speech. This would be unimaginable to someone in the late 1960s or early 1970s. That's not a computer, that's SF. Clarke's 3rd law, and so on.
I think they might be even more incredulous if you told them that, deep inside, the OS would still be usable via a TTY and paper tape, if you were extraordinarily patient.
It should be incredible. It's wrong. We had richer interaction models 40 or 50 years ago, from Sketchpad to the Canon Cat to Oberon's TUI.
And it all went nowhere. A mere sideline. It seems to me that it went something like:
Well, I think we're coming to a tipping point where a cheap technological development will become so commonplace that keeping the old disk-centric model will become a millstone around our collective neck.
The problem is that everyone is so used to the status quo it that it's going to be very hard to persuade them to let it go.
So it will have to be something that offers radical improvements to tempt people to have a go*.
Like, for instance, banishing entire categories of bugs and errors, while giving improvements in compilation time of 3-4 orders of magnitude, delivered by an environment that is much richer than a command line but simpler than a WIMP desktop.
Something as shocking as a Mac or Amiga was in 1984.
Inventing that from scratch* would be almost unimaginably hard.
So I'm saying, let's try to re-create some of those 1970s systems on modern kit, in FOSS, ready for the NVRAM computers when they arrive.
Re-implementing stuff that has been done before, and improving on it, is something the FOSS world excels at.
Let's get on with so we're ready.
* Puns intended
I run MS Word 97 under WINE to do my own writing. Not because I need any features in that version -- I don't. I'd be perfectly happy with Word 6, which is functionally near-identical, smaller and faster. The only reason I don't run Word 6 is that it doesn't support the scroll wheel or proportional thumbs -- the widget in the scrollbar is always a square so you get no visual feedback on how long your document is and what relative size chunk you're looking at. That's very useful info.
But I hunted, for years, for Word 6 for NT, the 32-bit version. Word 6 being a Windows 3 app. Why? Because the NT version supports long filenames and they're jolly handy. But not proportional thumbs or the scroll wheel. There's also the small convenience that Word 97 uses the same native file format that was used until Word 2003, and everything can import it: LibreOffice, Pages, AbiWord, you name it. The previous versions' file format -- also called .DOC -- is far less widely-supported, but then, it was phased out 22 years ago.
I've tried Word 6 for DOS, WinWord 6 (for Win3), Word 6 NT, Word 95 (v7), Word 97 (v8), Word 2000, Word XP, and Word 2003. Hunted 'em all down, tried on 32-bit and 64-bit Windows and WINE.
I have really looked deeply into this, mainly out of curiosity but also because I made my living using this tool.
One WINE compatibility test for me is: can I install the service releases under WINE? 97 yes, 2000 no, XP no, 2003 yes.
That falls down on Word 95. Why? Because there were no service releases.
This is an app I know inside-out and use hard and heavily, mainly because of the outlining feature that no other modern editor still offers. MS Word has gained no new features I need or use since 1993.
That was the first version that ran on both 16-bit and 32-bit Windows. Then came Office 95, 32-bit only, with zero functional changes but a new look and feel: square toolbar buttons, font formatting in Windows title bars, stuff like that. Trivial.
But Word 97 had no new functionality either, just a new .DOC file format, and yet Office 97 needed 3 service releases. The first release didn't even write a Word 6 .DOC file when you asked it to. It wrote an RTF file but gave it a .DOC extension. The older versions were smart enough to detect the format automatically and import it, so nobody noticed before release. How come?
The point of this tedious little aside ;-) being that the app has grown massively since then and it's now about 10× the size, probably more -- I haven't checked recent versions (since 2007) because I detest the ribbon interface.
I think a full install of Word 97 is 14 MB. I thought Office 97 was bloated when it came out. 14 MB may have been bloated in 1997 but today it seems svelte, almost minimalist.
I pick this one app because the additional functionality is amazingly trivial -- Word 97 has a virtual yellow highlighter pen and AFAIK that is the only functional addition. Word 2000 allows tables to be nested inside table cells, and AFAIK that is it.
My point being, it's a very mature codebase, and yet it's still bloated hugely in the ~20y since it last gained a major new feature, and it's also picked up major bugs in that time, so you really do want the service releases.
The greater point being that tech support is now even worse than when I left it, because the products are massively bigger, have a lot more functionality but not as much as you might expect in many places, and things originally introduced to reduce size and complexity, such as shared libraries, have in fact done the reverse and hugely increased size and complexity so that now we need major new OS features such as containers simply so that multiple apps with multiple specific versions of their shared libraries can co-exist on the same OS without causing horrible clashes.
Plan 9, I recently learned, doesn't have shared libraries. So when it was new, the Unix people of the time decried it, as it was missing something they thought important.
Plan 9's designers were right, but nobody now recognises that, because it's so long ago that the people inventing containers to fix shared-library versioning clashes don't know that there was a major successor product to Unix that identified shared libraries as a problem and eliminated it.
It was a generation ago. My colleagues sitting behind me, diligently testing an enterprise Linux distro, probably have never even heard of Plan 9.
A quondam GNU spokesman on Twitter spent a while last month lambasting me as a know-nothing idiot, and got a gang of his mates in to harangue and pillory me, because I told him that Linux is a Unix.
He thinks that's obviously wrong. He has probably never seen another Unix in his life, so obviously the old fart disagreeing with him is an idiot.
But to get to my larger point...
Tradeoffs were made in the design of Unix in the 1960s, to keep it very simple, because it was designed for a very low-end system: a PDP-7 with 4 kilowords (9 kB) of RAM.
Then in the 1970s it was rewritten in C, and further different tradeoffs were made in C to keep it very simple, to make it cross-platform and yet small and easy to compile.
Then in the 1980s, yet more tradeoffs were made in Minix to make it small and simple enough for students to understand, and yet still cross-platform on very low-end machines with no MMU etc. Shades of the original 1960s design, but nobody much noticed that.
Then in the 1990s, tradeoffs in the design of Linux were made, to make it simple to implement and to get it working in a reasonable timeframe, because commercial Unix was too expensive and Minix was too limited and Dr Andy Tanenbaum ("AST") wouldn't accept complex patches to make it 386-aware, give it proper virtual memory, etc.
And Torvalds and his followers vigorously defended those compromises, because they were pragmatic and meant it was doable and understandable and could be made to work, whereas the HURD couldn't.
Well now, we're further from the introduction of Linux (1991, 28y ago) than Linux was from Thompson & Ritchie's original PDP-7 Unix (1969, 22y before Linux). 50 years. Two human generations. [EDIT: corrected arithmetic error -- thank you Stefan!]
The sensible pragmatic compromises to get a simple monolithic kernel actually working and reasonably performant on a 25 MHz 80386 now mean a vast and vastly complex monolithic OS for gigahertz-class hardware with dozens of cores, with hundreds of thousands of full-time programmers around the world working flat out just to keep the thing working, patched and broadly generally mostly kinda safe-ish.
IMHO, AST was right.
Linux was Unix reimplemented as FOSS, from scratch, exactly as it had been done 2 decades earlier, partly because politics stopped the GNU project adopting the BSD kernel in the late 1980s. They looked at it and discarded it. If they'd adopted it, the GNU Project would have had a complete, working, FOSS Unix by about 1989 or 1990 and Linux would never have happened:
RMS was a very strong believer -- wrongly, I think -- in a very greedy-algorithm approach to code reuse issues. My first choice was to take the BSD 4.4-Lite release and make a kernel. I knew the code, I knew how to do it. It is now perfectly obvious to me that this would have succeeded splendidly and the world would be a very different place today.-- Friar Thomas Bushnell; also see The H.RMS wanted to work together with people from Berkeley on such an effort. Some of them were interested, but some seem to have been deliberately dragging their feet: and the reason now seems to be that they had the goal of spinning off BSDI. A GNU based on 4.4-Lite would undercut BSDI.
So RMS said to himself, "Mach is a working kernel, 4.4-Lite is only partial, we will go with Mach." It was a decision which I strongly opposed. But ultimately it was not my decision to make, and I made the best go I could at working with Mach and doing something new from that standpoint.
This was all way before Linux; we're talking 1991 or so.
They learned nothing from proprietary Unix. They improved nothing. They just redid it, the same, but as FOSS.
It was the last good chance to make some structural improvements, and we blew it.
So now, we have a maintenance nightmare. A global catastrophe of software project management, but it's enabled an industry worth hundreds of billions of dollars, which pays to keep the entire cancerous mess working.
And not a single person in the entire industry has the guts to point out that it's a catastrophic mess, because by wasting all the finest programming minds of an entire human generation (bar the small fraction employed by Microsoft to work on Windows, which is just as bad, because it didn't even learn from Unix), it still works.
As the human race busily causes catastrophic global ecosystem collapse, which has happened in the same 25y period, we have failed to materially improve on the computing state of the art from the beginning of the 1970s.
Two generations of progress lost, because if you throw enough money at the problem, even a mistake can be made to work. As TWA's Paul E. Richter Jr. said, "Give me enough power and I can fly a barn door." A monolithic 1970s OS can power the entire world, if the programming effort of the entire world is devoted to keeping it working.
But it's not efficient. C enthusiasts think it's efficient and low-level. It isn't. There are tens of millions of lines of error-checking, test frameworks and all sorts, and much of that is because of limitations in C, because of the monolithic kernel, because of those shared libraries and the very complex recursive package managers needed to suck in the millions of dependencies and sensibly provide what's needed.
This sounds absurd, but here are some citations.
A Google project containing some 2000 files and 4.2MB of source sucked over 8 gigabytes of code through the C compiler due to nested #includes.
Firefox depends on 122 packages including libTIFF, but it can't render TIFFs.
Maybe you will protest: "But that's just how software is! It needs to be that way! Modern software has to cope with so many things!"
"Hello world" in C++ involves 18,000+ lines of code. Getting a modern C++ compiler to build itself on modern hardware can take an hour or so.
Compare with Oberon, where a modern, TCP/IP-networked, just about web-capable OS written in Oberon and also called Oberon was implemented in about 12,500 lines of code.
The Oberon compiler can build itself in 3 seconds on a 25 MHz CPU.
Pick a different but still low-level language and an entire OS, far more functional than Linux was back when it became self-hosting, takes less code than "hello world".
This is not just a minor difference of degree. This is very badly wrong. This is a trillion-dollar issue. It is not an exaggeration to say this kind of issue threatens the continuing existence of our civilisation.
"But microkernels are so inefficient! All those context switches!"
I am absolutely sure that microkernels would have been less efficient, a human generation ago when the 80386SX was the most powerful computer that most people could afford.
But now, if we'd done Free UNIX properly, with modular codebases, isolated services instead of monolithic kernels, integrated networking and clustering -- all known, established, standardised stuff that was working and in production before Linux 0.01 -- I am very confident that now, we'd have something which required a very great deal less maintenance.
It's an alternate history. I can't put numbers on it because it didn't happen.
But let's look at the tech changes in that time span.
Between the invention of Unix and the invention of Linux, there were multiple generation shifts:
- the invention of the microprocessor (mid 1970s)
- the spread of simple, standalone, 8-bit business micro computers with mass storage on floppies -- the CP/M and S100 bus generation (late 1970s)
- their replacement with simpler, standalone, 8-bit home computers, mostly without mass storage, and simultaneously, 8/16-bit IBM PC-compatible business computers (early 1980s)
- 8-bit home computers' replacement with GUI-based 16-bit home (and wealthier business) computers (mid 1980s)
- Their replacement with GUI-based 32-bit-capable PCs (running a 16-bit OS, Windows 3)
Then came Linux.
Since Linux came along, we've gone through about half as many significant technological shifts, I'd say.
- 32-bit GUI OSes become universal on the PC (mid-1990s)
- the switch to NT/OS X, along with ubiquitous high-speed Internet connections (turn of the century)
- Appearance of 64-bit X86 (mid noughties - introduced 2003)
- Multi-core PCs became common (late noughties)
Basically, the PC industry moved to 32-bit Windows 25 years ago and hasn't substantially changed since. NT took over in the form of Windows XP in 2002-2003, around the same time as Mac OS X 10.2, the version that made Mac owners start to want to switch.
This is also around the time that Web commerce starts to drive adoption of Linux server farms, aided by VMware.
By which point, the entire computer industry was running 32-bit C-based multiuser OSes: Unix (including Mac OS X), Linux and Win NT. But their application ecosystem was shared with the previous generation of OSes from a decade before, and the underlying designs of those OSes are from the 1970s: Unix and DEC's VMS.
The industry attempted to move to new, home-grown OSes, such as OS/2 and Apple's Copland. It failed. So instead, OS design actually took a step backwards when 32-bit micros became usefully able to run 1970s minicomputer OSes, which emulated the previous generation of 16/32-bit microcomputer OSes well enough that users could keep their apps.
Since then, I would argue there has been no big shift. Just incremental performance and capacity improvements.
We had total computer generation shifts every decade or so from the invention of the reprogrammable digital computer just after WW2, for 2 human generations -- about 50 years. Everyone in the industry was used to it. It was normal.
It happened to me personally about 2-3 times between the time when I got interested as a schoolchild in the early 1980s: from a 48K Spectrum and cassettes, then microdrive, to a Spectrum 128 and MGT DISCiPLE and floppy drives, to an Amstrad PCW, to an Acorn Archimedes, to a 386 laptop running OS/2 2 (because I couldn't get Slackware to install), and then to a 486 running Windows 95.
Every time, I had to totally relearn the entire OS, learn new languages, and replace (or rewrite) every single program I used.
But since the mid-1990s, we've just been iterating the same design.
Now, we have a generation of programmers who have never in their lives, since childhood, ever seen a shift such as the ones I endured multiple times. The thought of throwing out all their software and starting again is unimaginable, inconceivable to them.
But before the mid-1990s, the entire computer industry did just that, at least every other hardware generation from the 1950s to the 1990s.
Then we stopped. Not because we'd achieved perfection -- we very definitely hadn't -- but we had stuff that was good enough, which with effort worked and kept working.
So we stopped developing new stuff, and just kept polishing the same things, more and more. Now they're very shiny indeed. As Douglas Adams said:
"It is very easy to be blinded to the essential uselessness of [their products] by the sense of achievement you get from getting them to work at all. In other words—and this is the rock solid principle on which the whole of the Corporation's Galaxy-wide success is founded—their fundamental design flaws are completely hidden by their superficial design flaws."
The result is that we've accumulated an almost inconceivably vast amount of cruft, but everyone thinks that that is perfectly normal. Even old hands like me. We use tools that are at least a quarter of a century old on OSes whose design is twice as old as that and we like it!
That is not normal, even for this industry.
There's been no substantial, revolutionary innovation since the GUI was invented in the late 1970s, and all we did with that was bolt that onto 1970s minicomputer OSes.
This has to end.
About 2012, pursing an retrocomputing story that hadn't been done to death already, I came across Lisp Machines and a war that has been successfully entirely covered up. That in the 1970s and 1980s, on the new class of personal workstations, there was a battle between two entirely different ways to write software and operating systems.
One side used radical high-level languages and designed special, elaborate processors to run them natively. The entire OSes were single, dynamic entities, whose code could be inspected and modified as it ran. No compilers, no linking, no static binaries. A whole new world, a wonderful programmer's playground.
The other side adapted a minimalistic decade-old minicomputer OS, written in the most minimalistic language around, bolted a clumsy GUI on top, and ran it on stripped-down, minimalist processors. It wasn't fancy or clever, but it was cheap and fast.
Cheap and fast won, of course.
Now, you never even hear that there was another way.
The rich dynamic languages hang on as programming tools on the not-even-slightly minimalistic descendants of those fast-and-cheap boxes, after they merged with IBM-compatible PCs. Even the sleek fast chips went away.
Only a few old folks are nostalgic for the good old times. 30, 40 and 50somethings, including my cohort, rhapsodize instead about 1980s home computer OSes: Amiga OS, ST GEM, RISC OS. They were small, fast and simple, and stomped all over PC OSes of the time. But the PC OSes won, and whenever those '80s OSes survive -- such as AROS or modern RISC OS -- they're hopelessly crippled and compromised compared to their modern rivals.
I wondered if there was anything that had the strengths but not the weaknesses. That was small, simple and fast, but also clean and elegant, which ran on modern hardware, on different architectures, and could exploit multiple CPUs, 64-bit machines with many gigabytes of RAM and so on.
So I found it very interesting that, after years of reading about this field, that I basically stumbled across Oberon and found that it fits what I was looking for remarkably well.
It's very small and simple (cf. Sinclair QDOS and kin).
It's FOSS (cf. QDOS descendants Minerva and SMSQ/E, or AROS, or AFROS, or RISC OS).
It has a current version and it does actually run properly on modern hardware, not just in an emulator. (Unlike all those, except RISC OS.)
It's a clean, simple design, done in 1 language. (Unlike all of them, really.)
It was actually used, by real people in the real world, for years. It has 3rd party apps, for instance. (Like all the above, albeit less so.)
It's very obscure but among those that know about it, it's widely admired.
It's renowned for its elegance.
It was very influential in its day. (Mostly, unlike all of them.)
It has survived in some places by becoming just another language-stroke-development environment for more mainstream OSes (like Smalltalk and Lisp before it, but unlike the others).
Unlike, say, Lisp, it's not arcane: its relatives Turbo Pascal & Delphi were once some of the world's favourite development tools, used worldwide by tens to hundreds of thousands, and so admired within MS that they went to great lengths to poach their programming lead.
Its most modern offshoot, A2/Bluebottle, is SMP-aware and can be used on the Web, a personal acid test of mine. It's only about as usable in this role as RISC OS, i.e. not very, but that it does it at all is impressive.
Haiku (the FOSS BeOS) ticks a lot of these boxes too, but it's x86 only, is only partially compatible with its predecessor, while being bigger and slower, and as a big, rich, modern 1990s OS that aspired to go mainstream, it's not some nice simple student-friendly thing.
For the avoidance of doubt, I am not saying that we should all just abandon Unix and switch to A2, or Lisp Machines, or Smalltalk boxes, or anything like them.
My thinking is more like this:
- Like Yellowstone blowing its top or the San Andreas Fault slipping, we're overdue for a big shift.
- This is really going to hurt when it comes.
- Probably much of the current stuff will be wiped away. Eventually that will be seen as a good thing. (Unless it doesn't happen, in which case humanity's last biggest opportunity since the moon landings will be wasted.)
64-bit didn't do it. Barely a blip in the end, remarkably.
Java didn't do it.
SSDs didn't do it. Smartphones and tablets didn't do it. They run modified existing OSes, concealed behind simplified UIs and a lot of sandboxing.
What's left? Persistent memory, AKA nonvolatile RAM.
On servers, its impact could be quite minor. But setting servers aside, on client devices, it could be very radical indeed. If you have a terabyte or so of cheap fast main memory that's nonvolatile, why would you need a disk drive? Turn the computer off and when you turn it back on, it picks up where it left off. No suspending, no resuming. No shuffling data from RAM onto flash disks and back. No flash. No disks.
But if you have no disks, whereas you certainly can emulate a hard disk by partitioning your NVRAM and formatting a bit of it, why would you? There's no need. There's no need for a resident filesystem at all. The computer never normally boots, unless it's updated; it just stops and starts.
Can you imagine Unix, the OS where "everything is a file", if there is no filesystem and no files?
Can you imagine Windows without a C: drive? With no drives at all?
It doesn't really work. Their central metaphor is the notion of files on disk, which either contain binaries or data. Binaries are "loaded" into RAM. They read data from other files -- config, working matter -- then they put it back when they're done.
Eliminate that and there isn't much left.
So, if that happens, what will the resulting systems look like?
I went looking for previous systems that didn't have the filesystem-centric metaphor. I found just four good solid examples.
- Multics, but I've barely been able to find anything much about it -- what there is is vast and impenetrable.
- IBM OS/400 -- ditto.
This leaves just two:
- Lisp Machines: everything is lists, inside one giant interpreter with special CPUs to make it quick enough to be just about usable. One language all the way down, with no clear divide between "OS" and "apps" and "data".
- Smalltalk boxes: everything is objects, inside one giant interpreter. This runs on top of a fairly conventional OS on a fairly conventional CPU, not that you see it, and the first examples were written in something vaguely Algol-like. One language for all the stuff you can see and work with.
The central working abstractions for both types of machine are in-memory structures, specific to their programming languages, rather than the filesystems which are the bedrock concept in Windows and Unix.
This seems to me to be a fairly good fit for a machine with a single level store of nonvolatile memory.
Yes, of course they will still need to support filesystems. We're going to want to back these things up sometimes, install new versions and additional functionality, and we'll want to support removable media, and connections to existing remote servers, etc.
The key point here is that although filesystems are present, that the filesystem won't be the central, defining abstraction.
Before we got caught in the software tar pits of the mid-1990s, it was normal for old technology to gradually atrophy away. Those mid-1990s PCs evolved from early-'80s home computers with a ROM BASIC which loaded and saved to an audio cassette drive via a dedicated port, but that functionality's long gone from absolutely all PCs today -- and nobody misses it.
Another comparison:
Early mainframes had vast, complex front panels for their operators, festooned with the famous "blinkenlights". In later ones, they just gradually went away, almost unnoticed.
Minicomputers and the first mid-1970s microcomputers went through phases of toggle switches for entering code bit by bit, and a row of lights for output. This was the lowest common denominator of programming. Then the switches went away, replaced by teletypes and paper tape. Then the teletypes went away, replaced by glass terminals, and paper tape was replaced by floppy drives.
Although VMS, Windows and Unix all have roots in the era of front panel switches, paper tape and teletypes, they either support them very poorly or not at all today. It's just not relevant. But they didn't just disappear overnight. They gradually went from central and inseparable to less important to barely-maintained code nobody uses any more to a novelty feature that gets made into amusing Youtube videos.
Now, if you're rich, you have a terabyte of nonvolatile storage in your pocket, driven by multiple 64-bit RISC cores, with which you interact via a huge 24-bit graphical framebuffer and speech. This would be unimaginable to someone in the late 1960s or early 1970s. That's not a computer, that's SF. Clarke's 3rd law, and so on.
I think they might be even more incredulous if you told them that, deep inside, the OS would still be usable via a TTY and paper tape, if you were extraordinarily patient.
It should be incredible. It's wrong. We had richer interaction models 40 or 50 years ago, from Sketchpad to the Canon Cat to Oberon's TUI.
And it all went nowhere. A mere sideline. It seems to me that it went something like:
- Xerox PARC invented something amazing.
- Apple loved it, although it didn't understand it, and delivered something far simpler but still radical.
- This was far too expensive so they made a cheaper, much dumber version.
- Microsoft found a way to graft that onto its existing million-selling product.
- After a decade of iterations, MS made version 3 of this fairly usable and it became a hit.
- After nearly another decade, in 1995 MS made it fairly good, and followed it with a better-architected successor.
- Apple is forced into copying MS's plan -- graft the UI onto more conventional underpinnings -- but does so onto a better base, minus a lot of MS's toxic obsession with backwards-compatibility and marketing.
- The FOSS Unix world copies them, but nobody much notices until it makes it onto phones.
Well, I think we're coming to a tipping point where a cheap technological development will become so commonplace that keeping the old disk-centric model will become a millstone around our collective neck.
The problem is that everyone is so used to the status quo it that it's going to be very hard to persuade them to let it go.
So it will have to be something that offers radical improvements to tempt people to have a go*.
Like, for instance, banishing entire categories of bugs and errors, while giving improvements in compilation time of 3-4 orders of magnitude, delivered by an environment that is much richer than a command line but simpler than a WIMP desktop.
Something as shocking as a Mac or Amiga was in 1984.
Inventing that from scratch* would be almost unimaginably hard.
So I'm saying, let's try to re-create some of those 1970s systems on modern kit, in FOSS, ready for the NVRAM computers when they arrive.
Re-implementing stuff that has been done before, and improving on it, is something the FOSS world excels at.
Let's get on with so we're ready.
* Puns intended
