23.3.05
Warren Harris on Links
Hi Philip,
I've been thinking more about your Links project, and what might be the most important areas to focus on. Previously we had discussed web applications and using a functional language to span the client and server programming as well as bringing good functional language technology to bear on the problem, e.g. types, pi-calculus, Erlang-style RPC, db comprehensions, etc. I think all this would be great, and a tremendous help to the web application developer. But the more I think about it the more I think there's a more pressing need in the computer world, and that is to have a well-designed (functional) shell language.
A functional shell language (fsh?) would be a program I could use instead of bash when I log into a linux system. It would give me command-line, interactive access to all the things I can do with bash -- list files, pipe output between programs, run programs in the background, etc. The difference would be that fsh could bring all that good functional technology to bear on shell programming: higher-order functional programming with lexical closures, strong typing, type inference, efficient native compilation, structural pattern matching, and most of all, sanity to the basic language design.
The ideal fsh would allow seamless interaction between locally-defined functions and linux programs residing in the file system. Languages like perl still depend on a "real" shell to launch linux programs (using 'system' or 'exec') or when creating input or output streams to them ('open'). This both forces the programmer to shift gears slightly to bridge the shell and perl world, and also runs the risk of introducing security holes. Ideally, fsh would provide an automatic way to call linux programs directly and pipe their output back into the program. Additionally, wrappers could be created around common programs (e.g. 'ls') to format their arguments and parse their output to return structural equivalents for easier down-stream processing.
Another area where significant advancements could be made is in pattern-matching. ML does an excellent job with structural matching (CDUCE goes even farther), but string pattern matching is relegated to library routines. A lot of perl's power comes from having string pattern matching embedded directly in the language, although a lot could be done to improve their usability. For instance, rather than matching a pattern and allowing the matching process to implicitly bind variables (e.g. $1), it might be better to express patterns as expressions that mention variables within them, similar to ML's structural matching. For example one might match a host and port with: /host:string ':' port:int/ rather than /(\w+):(\d+)/. Another idea might be to introduce pattern abstraction so that one might define 'addr' as the above pattern, and then reference it later, e.g.: /(host1, port1):addr ' -> ' (host2, port2):addr/ would match two addr patterns delimited by a literal ' -> ' string, and bind host1, port1, host2, port2 in the process.
I think having a solid functional shell could form the basis of a later all-singing-all-dancing-xml-processing web application language, but first-things-first. The key idea here is that this has to be an interactive language that's capable enough for routine control of the system. It's got to look and smell like a shell to start winning the minds of real-world programmers.
Warren
----
And my reply to Warren, with Warren's reply to my reply interleaved.
Philip Wadler wrote:
I like this style of lateral thinking. But if you do a web search you will find that the functional shell idea has been tried a few times in the past, and never conquered the world. So I'm not convinced that its worth putting the other Links ideas to one side in order to work on a functional shell.
I did a quick search and turned up a few things:
but I think it's safe to say that none of these are on the path to mainstream adoption. There certainly is a lot more to winning the minds of programmers than having solid underpinnings. I think it's mostly driven by practical matters, and building a community as you attempt to solve real problems. I hope that you don't underestimate this part of the effort.
Just to put one more plug in for making Links a shell language... Real web site administrators are driven to strip production machines down to the bare minimum amount of software running on them -- primarily as a security precaution. I've been questioned in my current job about the need for installing perl or cpan modules on machines that don't absolutely need them. Making Links a replacement for bash could facilitate its adoption by acknowledging this need for a minimal environment. If I could propose a mission statement for Links, I'd say, "#!/bin/links".
However, I was already convinced that a new language should build-in string pattern matching, as in Perl and Python. XML regular expression types give you almost for free regular expressions as (static!) types for strings.
Regular expressions with static types would be helpful. I'm reminded of ocaml's static typing for format statements -- also very useful.
I hope that you consider more the idea of pattern abstraction, i.e. being able to macro-ize or functionalize common patterns. This would go a long way in facilitating understanding and maintainability of code. In perl, it's a real chore to study patterns and try to understand what they might match. I'm sure you've heard perl described as a "write-only language". I'd personally like to see something more like a yacc grammar, and less like a packed string with meta-characters. In fact, I believe the packed string approach is appealing primariliy due to the lack of abstraction in patterns.
I think it could also be very useful to use common patterns as printers as well as parsers. This is particularly useful for complex yacc-grammar style patterns, but could be equally as useful for regular expressions. (I recall once writing a language grammar in yacc, and then spending a significant amount of time writing a printer to get all the precedence relationships and parenthesization right.) There is a lot of power in having a "print-read equivalence" as shown by the lisp world, and having more powerful tools to facilitate this would go a long way.
The one new thing I take away from your note is that a proper version of exec should work directly rather than via some shell. That's pretty easy to do, and would make Links attractive as a shell language with almost no extra work. Thanks for the good idea!
You're welcome. In addition to exec, the forking and piping has to be there too. Perhaps pipes could be generalized into a polymorphic synchronization/streaming/functional-composition operator that would work for passing datastructures between threads as well as strings between shell programs.
May I publish your comment on my blog? Cheers, -- P
Absolutely.
Warren
I've been thinking more about your Links project, and what might be the most important areas to focus on. Previously we had discussed web applications and using a functional language to span the client and server programming as well as bringing good functional language technology to bear on the problem, e.g. types, pi-calculus, Erlang-style RPC, db comprehensions, etc. I think all this would be great, and a tremendous help to the web application developer. But the more I think about it the more I think there's a more pressing need in the computer world, and that is to have a well-designed (functional) shell language.
A functional shell language (fsh?) would be a program I could use instead of bash when I log into a linux system. It would give me command-line, interactive access to all the things I can do with bash -- list files, pipe output between programs, run programs in the background, etc. The difference would be that fsh could bring all that good functional technology to bear on shell programming: higher-order functional programming with lexical closures, strong typing, type inference, efficient native compilation, structural pattern matching, and most of all, sanity to the basic language design.
The ideal fsh would allow seamless interaction between locally-defined functions and linux programs residing in the file system. Languages like perl still depend on a "real" shell to launch linux programs (using 'system' or 'exec') or when creating input or output streams to them ('open'). This both forces the programmer to shift gears slightly to bridge the shell and perl world, and also runs the risk of introducing security holes. Ideally, fsh would provide an automatic way to call linux programs directly and pipe their output back into the program. Additionally, wrappers could be created around common programs (e.g. 'ls') to format their arguments and parse their output to return structural equivalents for easier down-stream processing.
Another area where significant advancements could be made is in pattern-matching. ML does an excellent job with structural matching (CDUCE goes even farther), but string pattern matching is relegated to library routines. A lot of perl's power comes from having string pattern matching embedded directly in the language, although a lot could be done to improve their usability. For instance, rather than matching a pattern and allowing the matching process to implicitly bind variables (e.g. $1), it might be better to express patterns as expressions that mention variables within them, similar to ML's structural matching. For example one might match a host and port with: /host:string ':' port:int/ rather than /(\w+):(\d+)/. Another idea might be to introduce pattern abstraction so that one might define 'addr' as the above pattern, and then reference it later, e.g.: /(host1, port1):addr ' -> ' (host2, port2):addr/ would match two addr patterns delimited by a literal ' -> ' string, and bind host1, port1, host2, port2 in the process.
I think having a solid functional shell could form the basis of a later all-singing-all-dancing-xml-processing web application language, but first-things-first. The key idea here is that this has to be an interactive language that's capable enough for routine control of the system. It's got to look and smell like a shell to start winning the minds of real-world programmers.
Warren
----
And my reply to Warren, with Warren's reply to my reply interleaved.
Philip Wadler wrote:
I like this style of lateral thinking. But if you do a web search you will find that the functional shell idea has been tried a few times in the past, and never conquered the world. So I'm not convinced that its worth putting the other Links ideas to one side in order to work on a functional shell.
I did a quick search and turned up a few things:
but I think it's safe to say that none of these are on the path to mainstream adoption. There certainly is a lot more to winning the minds of programmers than having solid underpinnings. I think it's mostly driven by practical matters, and building a community as you attempt to solve real problems. I hope that you don't underestimate this part of the effort.
Just to put one more plug in for making Links a shell language... Real web site administrators are driven to strip production machines down to the bare minimum amount of software running on them -- primarily as a security precaution. I've been questioned in my current job about the need for installing perl or cpan modules on machines that don't absolutely need them. Making Links a replacement for bash could facilitate its adoption by acknowledging this need for a minimal environment. If I could propose a mission statement for Links, I'd say, "#!/bin/links".
However, I was already convinced that a new language should build-in string pattern matching, as in Perl and Python. XML regular expression types give you almost for free regular expressions as (static!) types for strings.
Regular expressions with static types would be helpful. I'm reminded of ocaml's static typing for format statements -- also very useful.
I hope that you consider more the idea of pattern abstraction, i.e. being able to macro-ize or functionalize common patterns. This would go a long way in facilitating understanding and maintainability of code. In perl, it's a real chore to study patterns and try to understand what they might match. I'm sure you've heard perl described as a "write-only language". I'd personally like to see something more like a yacc grammar, and less like a packed string with meta-characters. In fact, I believe the packed string approach is appealing primariliy due to the lack of abstraction in patterns.
I think it could also be very useful to use common patterns as printers as well as parsers. This is particularly useful for complex yacc-grammar style patterns, but could be equally as useful for regular expressions. (I recall once writing a language grammar in yacc, and then spending a significant amount of time writing a printer to get all the precedence relationships and parenthesization right.) There is a lot of power in having a "print-read equivalence" as shown by the lisp world, and having more powerful tools to facilitate this would go a long way.
The one new thing I take away from your note is that a proper version of exec should work directly rather than via some shell. That's pretty easy to do, and would make Links attractive as a shell language with almost no extra work. Thanks for the good idea!
You're welcome. In addition to exec, the forking and piping has to be there too. Perhaps pipes could be generalized into a polymorphic synchronization/streaming/functional-composition operator that would work for passing datastructures between threads as well as strings between shell programs.
May I publish your comment on my blog? Cheers, -- P
Absolutely.
Warren
