Some thoughts on programming languages
Mar. 15th, 2013 05:23 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
liam_on_linux(I've bolted this together from a few comments on a previous post, as it was buried some 125 comments down, and promoted it to a post in its own right. Commentary much appreciated, as ever!)
I used to program. I enjoyed it. I wasn't very good at it, but at a recreational level, I found it fun. When I first tried doing it for work purposes, it put me right off. I mentioned this on Twitter recently. (Not sure if that link will open the conversation in its entirety.)
My vague and not-at-all earth-shattering insight is something like this.
For a start, there are at least 4 or 5 levels of programmer. This is counting only those who know that they are programmers, not people, for instance, constructing huge elaborate models in Excel and not realising that what they are doing is in fact programming.
1. There are the very smart people, who have in many cases studied the discipline, read Knuth etc., know multiple languages including very arcane ones, and see and discuss the abstract patterns and behaviours. These people may well have some familiarity with Lisp, Scheme, etc. and concepts like metaprogramming.
2. There are the (often) autodidact engineers, proficient in a number of languages, but probably more expedient, pragmatic tools - C (++/#/etc), Java, Python, Perl, Lua, Tcl/Tk. They see things like Lisp as academic curiosities; languages are a tool for a job and are chosen for their utility and practicality.
3. There are the journeymen, possibly with some more-or-less vocational training, possibly even at Uni as that's what they teach these days AIUI. *Much* more likely to be working on Windows than xNix. Possibly also know some C but may, sometimes secretly, favour things like Delphi or VB. They stand apart from the Unix community and indeed often see Unix as a curiosity, a legacy platform or an irrelevance. They seldom have much familiarity with scripting languages or the like because these things are rarer and harder on Windows; however, Java is penetrating their world, being close to a first-class citizen on Windows now, and so the JVM-hosted languages are gaining traction - as are the family of relatives on the .NET CLR.
4. There are the (sometimes) reluctant amateurs, who can knock together useful tools in VB or possibly Delphi. Generally entirely self-taught with little to no theoretical knowledge.
5. The beginners, who can bash together a shell script (yes, I meant to do that) or batch file/NT CMD file/PowerShell script. If they were around, back in the 1980s could do simple stuff in BASIC, but never graduated any further, either from lack of interest, lack of drive or distractions. This is the group that I am in.
The thing is that - although this is changing these days, as scripting - I would very very roughly estimate that the number of instances goes up by a power each step. There are few in group 1 and tens to hundreds of millions in group 5.
Now the thing is this. From group 2 on down, these people mostly, I suspect - and of course this is all generalisations, mostly quite sweeping here - have little clear conception of the higher levels and the stuff that the higher levels discuss and deal in. The ideas that group 1 toss about casually, if group 2 hear of them, seem like discussing the comparative virtues of Classical Greek and Vulgar Latin.
And from groups 3 and 4, all this stuff is so irrelevant and theoretical that they pay no attention at all, if they even notice it's there.
Meanwhile, the concerns that are over-riding to groups 3 and 4 (is it free? is it readily human-readable? does it come with handy libraries? does it work on my OS? is it easy to get stuff done?) seem trivial to the point of irrelevance to group 1.
They seem unimportant to group 2 - if another language offers advantages, just learn it! However, group 2, possibly fluent in half a dozen languages, miss the deeper significance - closures, say - that group 1 discuss, seeing those as trivialities as well.
Looking down the pyramid *and looking up it*, the big important overriding concerns of the other groups *both above and below* seem to be trivialities.
There is little cross-fertilisation of ideas, is what I am getting at here, and it's a big problem and one that people are largely unaware of.
What I am wondering is something quite simple, mostly just a thought experiment, and it's this. Yes, I know that it is currently a woolly mess.
#1 At present, we have a growing profusion of languages, some quite domain-specific, some less so.
#2 Modern tools like shared VM runtimes (JVM, CLR) are bringing disparate languages together in places.
#3 With the Interwebs, ideas and learning and stuff get shared far more these days. There is actual room & the possibility that some stuff might escape the scented groves of academe and get out into the world and get used rather more than before.
#4 (possibly 3a) between growing class libraries and whatnot and resources like CPAN, there is more and more code-sharing occurring, too. Although as Poul-Henning Kamp points out [q.v. my comment to Geoff earlier], this also means vast amounts of duplication, inclusion of random crap because of poor modularity, etc.
I read lots of smart people talking about ideas and concepts in programming. Many of them seem to concur that, in the hierarchy of programming languages, Lisp is the top of the pyramid: the most powerful, expressive tool. The longest lever for the programmer's mind, if you will. Some lament that it has not caught on more.
I think that one partial answer to that - but a big one - is that if you are a group 1 or even to a degree a group 2 programmer, you realise that syntax is not really very important, and that once you wrap your head around a new one, however weird - be it some very high-level abstraction such as APL, or Lisp S-expressions and lambda calculus, or a very low-level one such as assembly code - you realise that any concept, any algorithm, can be expressed in any language; it's just what tools it lends you that distinguishes one from another.
That kind of stuff doesn't impinge much on group 2. They can master any syntax, even weird stuff, but they seem to tend to miss the bigger important ideas, the real power that group 1 gets hot & bothered about.
Meanwhile, group 3 aren't interested in these weird language-wars that groups 1 & 2 keep getting into; they have a job to do.
Group 4 are barely aware that there are language wars; they don't pick tools, they just get on with it with the tools that are to hand, whatever they may be.
We australopithecines in group 5 look at our less-hairy cousins debating the finer points of banging rocks together to make blades, and unfathomable arcana such as tying rocks onto sticks, and we shrug and go back to cracking open shells with a handy large pebble.
Now this being so - which is guesswork on my part, but based on observation - then it raises a question in my mind.
The question is something like this.
I am taking it as a given that there are terribly clever and sophisticated things about some languages, such as closures and whatnot, that the Giant Branes of group 1 say are terribly important but can't explain to an ape like me in ape-comprehensible terms.
Also, it seems that some languages for slightly-less-giant brains embody some of these ideas, but encapsulate them in more domain-specific languages which limit their deployment potential... e.g. I have seen carefully-worked examples showing how Ruby and Python both can do some of the clever Lisp stuff, but other bigger brains than mine assure me that tools like Python and Ruby are not suitable for writing OSs in, or even for writing large lumps of performance-critical code.
And yet, in the past, Giant Brains /did/ build OSs in weirdo comp-sci wonk languages like Lisp and Smalltalk...
And in the past, other quite-big brains thought that languages like Smalltalk and Dylan were just the trick for user application development. Dylan was intended for the Newton, for instance.
Other big brains tell me that, for instance, C is dangerous because of its lax typing and pointer manipulation and whatnot, and that C code tends to become unreadable, partly /because/ of the clever stuff its syntax permits.
They also say that Pascal makes it hard to do some stuff that's quite easy in C, for instance. And yet the mind behind Pascal went on to create Modula-2, which Acorn /nearly/ built the Archimedes OS in, and then Oberon, which same giant brain & his acolytes /did/ build OSs in.
And my own studies suggest to me that actually, to the still-quite-big (or at least bigger-than-mine) brains of groups 2 and 3 and even 4, syntax and legibility IS really important and that's where Lisp and even Smalltalk fall down.
So what I am wondering is this.
Firstly, is it possible to strike a happy medium? Is there some point along the scale where in a generally-applicable language, you can combine both the really powerful concepts that group 1 say are important - closures, lambda calculus, macros, homoiconicity, other weird stuff that my hairy-ape brain cannot fathom - /and/ yet do it in a legible language that won't scare off the smarter-ape-but-not-giant-brains of group 3 and 4, which are the ones that are around in really large numbers and who actually matter.
Because actually, the tools that most of we uplifted apes use every day, the browsers and word processors and stuff, are actually written by groups 3 and 4 in languages with well-documented problems and limitations and drawbacks. We don't use the giant-brain tools like Lisp Machines, Xerox Smalltalk boxes, or even Plan 9, because when they were in the ascendant, they were vastly expensive and their strengths were not apparent to grunts using C on PDPs and CP/M boxes.
And now that what we all have are much much faster PDPs, Vaxen and CP/M boxes, and the giant-brain tools of yore are extinct and nearly mythical, few remember them and what they were good for.
So what I am wondering is that if using modern tools - cheap fast x86 and ARM machines, FOSS OSs and languages, freeware runtime VMs - it would be possible to resurrect the giant-brain tools of yore in forms accessible to the group 3 and 4 programmers. To bribe them to give up their precious curly braces, pointers and compact i+= syntax with something really powerful and expressive, a Free, C21 Delphi, something so handy and easy that it's worth learning a new syntax and new toolset.
But not an (impenetrable (brain(giant)) (`syntax (`weird)) arcanum) such as Lisp, but something simple and readable so as not to scare off those who know Delphi and VB. Something that you can use as a rock to hammer on other rocks until you master the sharp edges of the weird stuff.
And in the meantime, because it might be necessary to demonstrate that these weird tools did have real power and are worth learning from, to resurrect things like Smalltalk-as-a-whole-OS or a LispM-OS-running-on-COTS-hardware.
I used to program. I enjoyed it. I wasn't very good at it, but at a recreational level, I found it fun. When I first tried doing it for work purposes, it put me right off. I mentioned this on Twitter recently. (Not sure if that link will open the conversation in its entirety.)
My vague and not-at-all earth-shattering insight is something like this.
For a start, there are at least 4 or 5 levels of programmer. This is counting only those who know that they are programmers, not people, for instance, constructing huge elaborate models in Excel and not realising that what they are doing is in fact programming.
1. There are the very smart people, who have in many cases studied the discipline, read Knuth etc., know multiple languages including very arcane ones, and see and discuss the abstract patterns and behaviours. These people may well have some familiarity with Lisp, Scheme, etc. and concepts like metaprogramming.
2. There are the (often) autodidact engineers, proficient in a number of languages, but probably more expedient, pragmatic tools - C (++/#/etc), Java, Python, Perl, Lua, Tcl/Tk. They see things like Lisp as academic curiosities; languages are a tool for a job and are chosen for their utility and practicality.
3. There are the journeymen, possibly with some more-or-less vocational training, possibly even at Uni as that's what they teach these days AIUI. *Much* more likely to be working on Windows than xNix. Possibly also know some C but may, sometimes secretly, favour things like Delphi or VB. They stand apart from the Unix community and indeed often see Unix as a curiosity, a legacy platform or an irrelevance. They seldom have much familiarity with scripting languages or the like because these things are rarer and harder on Windows; however, Java is penetrating their world, being close to a first-class citizen on Windows now, and so the JVM-hosted languages are gaining traction - as are the family of relatives on the .NET CLR.
4. There are the (sometimes) reluctant amateurs, who can knock together useful tools in VB or possibly Delphi. Generally entirely self-taught with little to no theoretical knowledge.
5. The beginners, who can bash together a shell script (yes, I meant to do that) or batch file/NT CMD file/PowerShell script. If they were around, back in the 1980s could do simple stuff in BASIC, but never graduated any further, either from lack of interest, lack of drive or distractions. This is the group that I am in.
The thing is that - although this is changing these days, as scripting - I would very very roughly estimate that the number of instances goes up by a power each step. There are few in group 1 and tens to hundreds of millions in group 5.
Now the thing is this. From group 2 on down, these people mostly, I suspect - and of course this is all generalisations, mostly quite sweeping here - have little clear conception of the higher levels and the stuff that the higher levels discuss and deal in. The ideas that group 1 toss about casually, if group 2 hear of them, seem like discussing the comparative virtues of Classical Greek and Vulgar Latin.
And from groups 3 and 4, all this stuff is so irrelevant and theoretical that they pay no attention at all, if they even notice it's there.
Meanwhile, the concerns that are over-riding to groups 3 and 4 (is it free? is it readily human-readable? does it come with handy libraries? does it work on my OS? is it easy to get stuff done?) seem trivial to the point of irrelevance to group 1.
They seem unimportant to group 2 - if another language offers advantages, just learn it! However, group 2, possibly fluent in half a dozen languages, miss the deeper significance - closures, say - that group 1 discuss, seeing those as trivialities as well.
Looking down the pyramid *and looking up it*, the big important overriding concerns of the other groups *both above and below* seem to be trivialities.
There is little cross-fertilisation of ideas, is what I am getting at here, and it's a big problem and one that people are largely unaware of.
What I am wondering is something quite simple, mostly just a thought experiment, and it's this. Yes, I know that it is currently a woolly mess.
#1 At present, we have a growing profusion of languages, some quite domain-specific, some less so.
#2 Modern tools like shared VM runtimes (JVM, CLR) are bringing disparate languages together in places.
#3 With the Interwebs, ideas and learning and stuff get shared far more these days. There is actual room & the possibility that some stuff might escape the scented groves of academe and get out into the world and get used rather more than before.
#4 (possibly 3a) between growing class libraries and whatnot and resources like CPAN, there is more and more code-sharing occurring, too. Although as Poul-Henning Kamp points out [q.v. my comment to Geoff earlier], this also means vast amounts of duplication, inclusion of random crap because of poor modularity, etc.
I read lots of smart people talking about ideas and concepts in programming. Many of them seem to concur that, in the hierarchy of programming languages, Lisp is the top of the pyramid: the most powerful, expressive tool. The longest lever for the programmer's mind, if you will. Some lament that it has not caught on more.
I think that one partial answer to that - but a big one - is that if you are a group 1 or even to a degree a group 2 programmer, you realise that syntax is not really very important, and that once you wrap your head around a new one, however weird - be it some very high-level abstraction such as APL, or Lisp S-expressions and lambda calculus, or a very low-level one such as assembly code - you realise that any concept, any algorithm, can be expressed in any language; it's just what tools it lends you that distinguishes one from another.
That kind of stuff doesn't impinge much on group 2. They can master any syntax, even weird stuff, but they seem to tend to miss the bigger important ideas, the real power that group 1 gets hot & bothered about.
Meanwhile, group 3 aren't interested in these weird language-wars that groups 1 & 2 keep getting into; they have a job to do.
Group 4 are barely aware that there are language wars; they don't pick tools, they just get on with it with the tools that are to hand, whatever they may be.
We australopithecines in group 5 look at our less-hairy cousins debating the finer points of banging rocks together to make blades, and unfathomable arcana such as tying rocks onto sticks, and we shrug and go back to cracking open shells with a handy large pebble.
Now this being so - which is guesswork on my part, but based on observation - then it raises a question in my mind.
The question is something like this.
I am taking it as a given that there are terribly clever and sophisticated things about some languages, such as closures and whatnot, that the Giant Branes of group 1 say are terribly important but can't explain to an ape like me in ape-comprehensible terms.
Also, it seems that some languages for slightly-less-giant brains embody some of these ideas, but encapsulate them in more domain-specific languages which limit their deployment potential... e.g. I have seen carefully-worked examples showing how Ruby and Python both can do some of the clever Lisp stuff, but other bigger brains than mine assure me that tools like Python and Ruby are not suitable for writing OSs in, or even for writing large lumps of performance-critical code.
And yet, in the past, Giant Brains /did/ build OSs in weirdo comp-sci wonk languages like Lisp and Smalltalk...
And in the past, other quite-big brains thought that languages like Smalltalk and Dylan were just the trick for user application development. Dylan was intended for the Newton, for instance.
Other big brains tell me that, for instance, C is dangerous because of its lax typing and pointer manipulation and whatnot, and that C code tends to become unreadable, partly /because/ of the clever stuff its syntax permits.
They also say that Pascal makes it hard to do some stuff that's quite easy in C, for instance. And yet the mind behind Pascal went on to create Modula-2, which Acorn /nearly/ built the Archimedes OS in, and then Oberon, which same giant brain & his acolytes /did/ build OSs in.
And my own studies suggest to me that actually, to the still-quite-big (or at least bigger-than-mine) brains of groups 2 and 3 and even 4, syntax and legibility IS really important and that's where Lisp and even Smalltalk fall down.
So what I am wondering is this.
Firstly, is it possible to strike a happy medium? Is there some point along the scale where in a generally-applicable language, you can combine both the really powerful concepts that group 1 say are important - closures, lambda calculus, macros, homoiconicity, other weird stuff that my hairy-ape brain cannot fathom - /and/ yet do it in a legible language that won't scare off the smarter-ape-but-not-giant-brains of group 3 and 4, which are the ones that are around in really large numbers and who actually matter.
Because actually, the tools that most of we uplifted apes use every day, the browsers and word processors and stuff, are actually written by groups 3 and 4 in languages with well-documented problems and limitations and drawbacks. We don't use the giant-brain tools like Lisp Machines, Xerox Smalltalk boxes, or even Plan 9, because when they were in the ascendant, they were vastly expensive and their strengths were not apparent to grunts using C on PDPs and CP/M boxes.
And now that what we all have are much much faster PDPs, Vaxen and CP/M boxes, and the giant-brain tools of yore are extinct and nearly mythical, few remember them and what they were good for.
So what I am wondering is that if using modern tools - cheap fast x86 and ARM machines, FOSS OSs and languages, freeware runtime VMs - it would be possible to resurrect the giant-brain tools of yore in forms accessible to the group 3 and 4 programmers. To bribe them to give up their precious curly braces, pointers and compact i+= syntax with something really powerful and expressive, a Free, C21 Delphi, something so handy and easy that it's worth learning a new syntax and new toolset.
But not an (impenetrable (brain(giant)) (`syntax (`weird)) arcanum) such as Lisp, but something simple and readable so as not to scare off those who know Delphi and VB. Something that you can use as a rock to hammer on other rocks until you master the sharp edges of the weird stuff.
And in the meantime, because it might be necessary to demonstrate that these weird tools did have real power and are worth learning from, to resurrect things like Smalltalk-as-a-whole-OS or a LispM-OS-running-on-COTS-hardware.
