> I didn't realise alternative notations were planned and never took off.
See for example this program from McCarthy, 1960. On top is my Common Lisp translation and below you find the original code. The original had to be hand-translated into s-expression syntax to run it in Lisp - in 1960.
> But macros exist in non-homoiconic languages, like Nim and Elixir.
Lots of languages have some kind of macros: C has textual transformations and Elixir has AST transformations.
Lisp macros are different, since they are not AST transformations but token tree transformations. That means Lisp accepts arbitrary token trees (give that the first item is a macro identifier) and one can write programs to transform them.
Languages which support macros for AST transformations, need to have the code parsed into an AST first. That means the expressions it can transform need to be already parsed into an AST - and the language accepted by the parser limits the expressions - unless one can add syntax changes to the parser, too.
ASTs are special purpose data structures, while s-expressions are general purpose data structures and thus are not limited to expressing programs in a particular syntax and are not limited to express something which looks like a program.
Lisp offers a bunch of macros: macros for transforming Lisp code, macros for transforming symbols, compiler macros for optimizing code, macros for transforming s-expressions, ... Additionally the macro has access to a full Lisp - which means it can do arbitrary computations: primitive transformations, complex language transpiling, full parsing&transformation, doing side-effects in the development environment, running at runtime in interpreters, ...
That means for me as a programmer I can automate all kinds of code manipulation. There are many simple and many complex examples.
Another advantage is that the language core can be reduced to a minimum set of syntactic forms and every other syntax can be built on top of it - and the developer uses the same mechanism as the language core: macros. This makes macros very pervasive - a programmer will use a lot of macros and may also program macros.
One of the disadvantages is that macros also cost us: the code we see can be very different from the code which executes and there is an additional programming paradigm to understand: using syntactic abstractions with macros. One can learn some Lisp without touching macros (see for example SICP, which does not use macros), but in real world programs one might see a lot of macros.
An example for a typical use of macros is CLOS, the Common Lisp Object System. It has a three or four - layer architecture:
layer 0 : meta object layer for CLOS - here CLOS is implemented in itself
layer 1 : object layer for CLOS - here CLOS deals with simple representations of itself
layer 2 : functional layer for CLOS - here have functions operating on CLOS objects
layer 3 : macro layer for the CLOS programmer - here we have macros providing ways to configure CLOS programs
Take for example defining a CLOS class:
0) on layer 0 CLOS has CLOS functionality to represent and create instances of classes and metaclasses, he were can also write extensions to new kinds of metaclasses
1) on layer 1 CLOS classes are objects: which have slots which are objects and which have superclasses which are objects
2) on layer 2 CLOS provides functions to create all that stuff: classes, slot objects, ...
3) on layer 3 CLOS provides the DEFCLASS macro which at macro expansion time - which typically will be triggered by compilation - can do all the transformations and can interface to the development environment at compilation time.
The programmer will typically use layer 3 to define a class using the macro DEFCLASS. For more advanced use one could use the function ENSURE-CLASS
Common Lisp originally in 1984 did not have any syntax for defining classes. The first CLOS implementation was a Lisp program which implemented all the layers above and provided the necessary syntax to define the new language objects: generic functions, methods, classes and some others. So it was mostly a user-level program which added a full complex object system - with only little code for implementation specific stuff.
Thus the DEFCLASS macro could introduce any syntax it wanted as a user level program - since it is not constrained by a fronted parser. So the CLOS developers came up with something that was convenient for them to define classes. This could mean that it does something trivial by providing a different order to define things, make things shorter to write or that it does something more complex by actually checking the provided class configuration and doing some code transformations.
Thank you so much for this thoughtful reply — I enjoyed the code samples and the details on Lisp's macro system, and I can see how its approach gives you more control to invent on top of the language. It's given me more to read and think about.
See for example this program from McCarthy, 1960. On top is my Common Lisp translation and below you find the original code. The original had to be hand-translated into s-expression syntax to run it in Lisp - in 1960.
https://gist.github.com/lispm/e44d81c3bb9b86d4313763647e058a...
For a Lisp 2 (Lisp 2 as the successor to the original Lisp language - with the long promised notation) example see: http://www.softwarepreservation.org/projects/LISP/lisp2/SP-2...
> But macros exist in non-homoiconic languages, like Nim and Elixir.
Lots of languages have some kind of macros: C has textual transformations and Elixir has AST transformations.
Lisp macros are different, since they are not AST transformations but token tree transformations. That means Lisp accepts arbitrary token trees (give that the first item is a macro identifier) and one can write programs to transform them.
Languages which support macros for AST transformations, need to have the code parsed into an AST first. That means the expressions it can transform need to be already parsed into an AST - and the language accepted by the parser limits the expressions - unless one can add syntax changes to the parser, too.
ASTs are special purpose data structures, while s-expressions are general purpose data structures and thus are not limited to expressing programs in a particular syntax and are not limited to express something which looks like a program.
Lisp offers a bunch of macros: macros for transforming Lisp code, macros for transforming symbols, compiler macros for optimizing code, macros for transforming s-expressions, ... Additionally the macro has access to a full Lisp - which means it can do arbitrary computations: primitive transformations, complex language transpiling, full parsing&transformation, doing side-effects in the development environment, running at runtime in interpreters, ...
That means for me as a programmer I can automate all kinds of code manipulation. There are many simple and many complex examples.
Another advantage is that the language core can be reduced to a minimum set of syntactic forms and every other syntax can be built on top of it - and the developer uses the same mechanism as the language core: macros. This makes macros very pervasive - a programmer will use a lot of macros and may also program macros.
One of the disadvantages is that macros also cost us: the code we see can be very different from the code which executes and there is an additional programming paradigm to understand: using syntactic abstractions with macros. One can learn some Lisp without touching macros (see for example SICP, which does not use macros), but in real world programs one might see a lot of macros.
Paul Graham wrote a book, which explained use of macros in Lisp: On Lisp. Free download at: http://www.paulgraham.com/onlisp.html
There are more disadvantages and also attempts to improve macros. See for example: http://library.readscheme.org/page3.html
An example for a typical use of macros is CLOS, the Common Lisp Object System. It has a three or four - layer architecture:
Take for example defining a CLOS class:0) on layer 0 CLOS has CLOS functionality to represent and create instances of classes and metaclasses, he were can also write extensions to new kinds of metaclasses
1) on layer 1 CLOS classes are objects: which have slots which are objects and which have superclasses which are objects
2) on layer 2 CLOS provides functions to create all that stuff: classes, slot objects, ...
3) on layer 3 CLOS provides the DEFCLASS macro which at macro expansion time - which typically will be triggered by compilation - can do all the transformations and can interface to the development environment at compilation time.
The programmer will typically use layer 3 to define a class using the macro DEFCLASS. For more advanced use one could use the function ENSURE-CLASS
Common Lisp originally in 1984 did not have any syntax for defining classes. The first CLOS implementation was a Lisp program which implemented all the layers above and provided the necessary syntax to define the new language objects: generic functions, methods, classes and some others. So it was mostly a user-level program which added a full complex object system - with only little code for implementation specific stuff.
Thus the DEFCLASS macro could introduce any syntax it wanted as a user level program - since it is not constrained by a fronted parser. So the CLOS developers came up with something that was convenient for them to define classes. This could mean that it does something trivial by providing a different order to define things, make things shorter to write or that it does something more complex by actually checking the provided class configuration and doing some code transformations.
The whole story of CLOS in CLOS itself is written in AMOP: https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...
See also the pointer there to the first and portable implementation of CLOS: PCL.