Roger Corman, author of Corman Common Lisp (long-form talk).txt

When I was a kid, I believed in Santa Claus, I believed in magic, I believed magicians
were really doing magic when they were on stage, and I believed in the Easter Bunny
and all those magical things.
And when I learned Little Older, like, probably I'll have a similar story, there was no magic,
and it was kind of sad, and some years later, I feel like the magic came back.
And to me, tech, you know, the internet, I mean, it's so far beyond anything I could
have imagined and considered magical, you know, and I always talk about magic when I
talk about tech stuff, because to me I'm still in awe, and it's just amazing to the
more you learn, the more you're in awe of all of these things.
And I've been fascinated with Lisp machines, you know, ever since I got into programming.
It's just really great to be here today, I'm so glad to have been asked, and that you've
got such a great group and all these Lisp people here.
I certainly have always felt that the Lisp programmers that I know, the people that are
in the Lisp are the most brilliant software engineers.
I think you know what I'm talking about, it's like people that really are creative, understand
a lot of things, and it's so just awesome to be here with all of you, to meet a lot
of you tonight.
So I spent a lot of years, thirty-five years now, programming Lisp, programming Lisp systems,
and in talking, working out what I was going to talk about, sounded like there was some
interest in, you know, how Cormin Lisp got built and why, and so I'm going to get into
that some, but I also built two other Lisp's, and I think that it's kind of interesting
to talk about that.
And I also want to talk a little bit about some projects that I've done and am working
on now, and I didn't really know how to put all that together other than in a sort of
a chronological thing, so I don't want it to be like a scrapbook or something, I mean
I want, hopefully there's some kind of message you get out of here, and I'm not just preaching
to the choir because I do start to evangelize about Lisp, and I know there's not a need
for that here, but anyway, I will start with where I started.
A few things we'll touch on, why did I develop three Lisp's?
I learned Lisp from a musician through essentially correspondence, course through the mail, kind
of weird, Franz kind of convinced me to create my own Lisp, and then I think they regretted
it later, so this is kind of interesting.
I like Cormin Lisp, what I would say is great about it is I think it's great for hacking,
hacking in the good sense, the white hat sense, you know.
I'm interested in other Lisp's, lots of Lisp's to me, the fact that there's like so many Lisp's
closure and scheme, and you know, whatever Lisp's that are nonconformant, and ones that
people are inventing, to me that's a feature, not a bug, I mean what other language can
you say this about?
You know where you could just say a name, a simple name that describes a whole bunch
of languages, and everybody pretty much agrees that they all are part of that family.
And why is the current project I'm working on using a trampoline, I might get to that
if I make it through the slides.
So before I got into computing, I was a musician, a jazz pianist, and I still play, I still
do gigs, managed to keep that up over the years, mostly that was while I was in college
that I was doing it professionally, but I was studying music and computer science at
Sonoma State, and computer science classes, I started in with that and learning Pascal,
and it was fun, it was easy, but nothing exciting about it.
And my music professor, Arthur Hills, was a, took a year off, he's an organist and
into medieval music, he took a year off to go study Lisp, and for a sabbatical and write
a Gregorian chant generator, that's what he did for a year, right?
And so he kind of got back in touch, during that year, he got in touch with the people
in the music department and said, Sonoma State, and all the other professors and said, oh
this is great, you've got to learn this new language, you've got to learn computer programming,
this is in 1981, 1982, and none of them, including him, had done anything with computers, and
he asked me, I wasn't a teacher, but he asked me, because I was a student, a computer student
too.
I mean, yeah, okay.
And you know, we started programming on the cyber, it's like a super computer that Sonoma
State had, I mean, it's designed by Cray, but at very weird architecture with six-bit
bytes and 60-bit word lengths, and it took two bytes to store a lowercase character and
one byte to store an uppercase character.
Since he was living in San Francisco, he would, basically it came down to mostly me and him,
he would start to send me letters, and he would write out longhand these letters about
Lisp functions, you know, some code he was working on with his Gregorian chant generator
or some new feature that he learned that he wanted to share with me, and sometimes he
would take a tractor feed print out, you know, they're really wide and fold it all up and
stuff it in the envelope along with a handwritten or typed on his typewriter with lots of corrections.
And so this is an example, this was a postcard, he'd send, hi Roger, I've already conceived
improvements of my own, I haven't tried these yet, but surely they'll work for A-mean and
G-mean, and here's all this code, and after the last parenthesis he even signs his name,
Arthur, down in the corner, in case I don't know who's sending me this Lisp code, right?
And if you look at the post market, it's hard to read, but it's July 17, 1982, which just
about 35 years ago today, so I think that proves I've been using Lisp for that long.
So this was basically Lisp 1.5 back in those days, and I just couldn't not believe how
magical it was, how amazing it was to do symbolic computation, and no numbers.
You know, I've been worked with Pascal and with C, you know, everything's a number really,
and your C program, your characters are numbers, and functions are really, you know, addresses
are numbers, and you know, just across the board, and you know that, and you have to
know that, and you have to understand to work with it back then, but in Lisp, I wrote lots
of programs with no numbers, just lots of lists and random selections and application,
and I made my own rules engine, and I worked on a rules engine to do something more original,
it generated jazz compositions, okay, Arthur was doing Gregorian chants, I was doing jazz
compositions, but this machine that was like a super computer was so fast, it would compile,
you know, 5,000 line Pascal program in a second, you know, and run everything so quickly.
My jazz composer generator would just bring the thing to its knees, you know, it would
sit there for 30 seconds, and the other people in the computer lab are complaining that the
computer's frozen, and finally it would time out, and it was working, it's just that I was doing,
you know, multiple levels of recursion and all kinds of stuff that were very inefficient,
it was my own algorithms, my own fault, wasn't the computer's fault, but that got me like,
well, how does this work? You know, disassembling the code, and looking at how, you know, and I
found out that like, Carr and Cutter were like two instructions or something on that machine,
I mean, it was amazing, it was amazingly efficient when you started looking at it.
After college, I went to work on microcomputers, 8-bit, 16-bit, very little memory, I just assumed
there was no way you could use Lisbon, after my experience with the performance thing,
I said, okay, you know, I mean, we can use basic or fourth or C, but there was no way a Lisp could
possibly run on 64K, but it wasn't entirely true, and after, you know, a couple years,
I started writing my own little Lisp interpreters, and using them as scripting languages inside my
C programs, just as AutoCAD or Emacs uses Lisp for scripting, and today we would say, you know,
you could use it as for automated testing, and macros and things like that, Lisp was good for
that, and so I started putting them into lots of my, you know, I was doing computer graphics
applications, and so that was useful, and they didn't have to be big, they just had to be big
enough to sort of run the different functions of the application that they were embedded in,
but that's how I learned about how you implement a simple Lisp, I mean, Lisp is simple, right,
it is simple to make a Lisp interpreter. I didn't really understand how garbage collection worked,
that was kind of the hardest part of it, if you've ever written a garbage collection collector,
you know, it's like, that's not so easy, you got to deal with registers and stacks and all kinds
of low-level things, but anyway, that was kind of my first exposure, first use of Lisp on PCs,
and when I moved to the Mac, fortunately, because the Mac had more memory and was more
powerful around that time, around 1990, I actually saw Mac common Lisp in action,
and some of you, I'm sure, know Mac common Lisp, Coral common Lisp, now it's called Closure with a Z,
and it was an amazing kind of experience, and it kind of made me realize, wow,
Lisp could be really, really fast, I mean, honestly, it seemed like it was as fast as C,
and how does that work? It starts, you know, read all of their documentation, and like,
trying to figure out, how could, is it really doing what it seems like it's doing?
And it cost $500, and I didn't have $500 to spend on, you know, on a license, since it was more of
less a hobby, at that point, I didn't have a job doing Lisp programming,
so I ported over my PC code and started running that on a Mac, and then started adding all the
common Lisp features, because I liked the common Lisp spec better than the Lisp 1.5 spec that I'd
been working with, and Mac was big enough to support it, and I ended up developing a pretty full
common Lisp in adding Clos, and adding, you know, Loop, and an editor that was built from scratch,
because we didn't have, like, you couldn't just, like, take modules and programs that other people
had written the way you can today, you know, it's like, if you wanted an editor, basically,
you're going to write it yourself. MarketSweep Collector, it had a full 68k,
68,000 compiler, code generator, assembler, disassembler, spent years on this, and it was a lot of fun,
release it as shareware, and a lot of people use it, AOL in those days, we distributed it on AOL,
because we didn't have the web yet, and Apple picked it up and put it on their developer disk,
which was nice, so every month it got sent out to all the Apple developers.
A little notoriety from that. It still didn't begin to compete with Mac common Lisp in speed,
it was cheaper, but it was not anywhere near as fast.
So here it is, running on a PowerBook 170, this is what I developed it, I called it PowerLisp,
because it was developed on a PowerBook, and lest you think this is like an old picture,
I took this two days ago in my office, because that machine still runs, it's a
20-26 year old machine, and I have to sort of plop it on the table top to get the hard drive
to spin up, I have to sort of knock it around a little bit, but then it gets going and it's pretty
good, surprising what it does, and you can see I got the little disassembly there,
and PowerPC came out and I redid it for PowerPC, and got into risk architecture,
and learned how all that instruction set and how arguments are passed, and that was very different,
but you sort of had to do that if you're on Apple in those days, you had to rewrite your stuff for
PowerPC, and did that, released fat versions of it, fat version meant it would run on PowerPC
or 68000 max. Questions are fine. How much of the code was actually a assembler versus everything
that you could write on top of it? So there was a basic, there was very little assembler in this,
mostly a C++ kernel, and then like the compiler and the assembler and all these other parts were
all written in Lisp, so mostly Lisp after the kernel, but it had quite a lot in C++,
and I didn't like that, and when I came along later and did another one, I got rid of most of that
and recoded stuff in Lisp. So kind of a diversion from that path. Now I've never really been able
to develop Lisp systems for my main job, right? It's always been kind of a sideline, kind of a hobby.
I always had to make money, and there's not that much money in that, so I just kept doing it in
my evenings, you know, long nights, and at one point I got an opportunity to build a
decision support system for hospital software, which was pretty cool. It was kind of an AI thing,
kind of an expert system to alert physicians to medical problems and to run rules that were
written in this language called Arden syntax, which is a standard for medical rules, medical logic,
and so I, because I wrote compilers, I got the job, and immediately, you know, had to
build something quickly, and the spec for this language, Arden, was basically, you know,
weakly typed, you know, dynamic typing, it needed garbage collection, it needed a lot of features,
and there's no way I was just going to write this in C, you know, and generate code in C. It just
would have been very inefficient, you know, and you wanted to be able to load rules on the fly,
update rules on the fly, you know, all this stuff, but I thought it would be simple to do this in
Lisp, you know, I could easily generate a Lisp function from these rules, and then those Lisp
functions would all run, and I'd have all the features I needed, and that turned out to have
been a very good decision, so then the problem was, this was back in the mid 90s, companies didn't
want you to use any other language in C++ in those days. It was like, the company I was working
with at the time just was like, no, everything's C++, that's just the way it's going to be, because,
you know, there was this idea that someday everybody was going to be using the same language,
that all programming languages were going to converge, which seems ludicrous today, but
that was a thought for a long time, and so I kind of snuck it in. I just sort of like
brought over my code from the Mac, my Lisp system from the Mac, translated it back to Windows,
stripped out a bunch of stuff I didn't need, and embedded it into the code and said,
it's C++, it's all C++, right, and truly that system, that Arden compiler and everything was
mostly written in C++, but the rules all get generated, all get translated into Lisp,
and all get run as Lisp functions, and this system's been very successful, I can't tell you how
successful it's been, and it's definitely the most successful thing I've worked on. It's still
in the top two or three US hospital systems and use today. I know what it's doing and how it's
being used, and we kept expanding it and expanding it to be used basically as a scripting language
for the whole system, so rules don't just alert physicians, but rules actually fill out forms,
you know, look up data in the database and fill out default values or, you know, prompt the user,
you know, prompt the user, there's all kinds of things they do, and I've had the luck to be sort
of work on it over the years and maintaining it, and I've also built a system that monitors the
use of this system in hospitals and logs it, and in one hospital alone, millions of rules
a day are triggered, you know, millions of these Lisp functions a day are triggered,
every single thing that goes into every single patient record goes through all these rules,
and that's the reason that that system is still around 20 years later, because it's getting kind
of, you know, archaic in terms of its age, but we did port it to .NET at one point, so it's no
longer, so it's no longer native code. This is a screenshot of the editor for that system that's
still in use today. It looks old, you know, the user interface looks old because it is,
but it's, if you can see off on the bottom part on the left, there's part of a rule here,
there's not much showing, but the function definition actually shows on that panel on the right,
and you can see that there's a lambda definition, so that's the Lisp code that got generated from
that rule, and this isn't documented and nobody really knows it's there, so, but thousands of
users are writing these rules, and the fact that Lisp is in there is completely unknown, most of them.
In fact, if you went to the next tab over the execute tab, there's a repel, there's a hidden
repel, and so I could go in there and like just start executing all kinds of lists, nobody even
knows it's there, right, so it's just kind of interesting. This is just some of the hospitals
that it's running in, New York Presbyterian, Columbia University Cornell, Johns Hopkins,
Cleveland Clinic, Memorial Sloan Kettering, Cancer, National Institute of Health, I mean,
that's just the beginning, but it's, you could tell it's, these are big places, so it's,
you know, we've got Lisp running in those things, and I think that's cool.
There, all scripts has been high on the list of hospitals, software that actually get used by
physicians, and as what this is, what CPOE is, physician order entry, or care provider order
entry, so it's been very successful. Any, any questions, any questions before I move on?
I'm doing it on time. How long am I supposed to talk?
Okay, I'll just keep going. I just want to, I probably won't get through all my slides, but
around 1996, I'd say Fritz Kunze, who was the CEO of Franz, makers of Allegro Common Lisp,
contacted me and said, well, why don't you come, come meet me and, you know, in our office,
I have something I want to ask you about, and I, that was great. I got to go down to Franz to
their office and, and talk, and what they wanted me to do was port Allegro Common Lisp to Mac,
because they decided they needed a Mac version, and I'd written a Mac version of Lisp, so I was
a good candidate for that. And so I made them some kind of a bid, like six months or something,
you know, I, no idea how long it was really going to take, but, and, and they never took me up on it.
I think they thought it was too expensive. I don't know, but in the process, they gave me a copy of,
of Allegro Common Lisp for Windows, and it was a big box with a ton of manuals and cost a thousand
dollars, like no way I could ever afford to buy that for myself. Again, you know, unless I was
professional, you know, for a professional purpose, but I was very happy. They just gave it to me,
and, and I, I love all the people at Franz are awesome, but when I played with it, this was,
it was crazy. It was like 16-bit fixed nums, which was, I think, a holdover from DOS,
you know, and, and this was, this was, you know, Windows 95 was out and Windows NT, and they had
preemptive multi-threading and tons of memory, all these features in here. They were marketing this
system that 16-bit fixed nums and eight character limit on filenames. I mean, they were like way
behind the times, and they knew that. I mean, they were like, yeah, well, they had bought it from
some other company, so that they'd have a Windows version. What? Okay, um, that, anyway, that was
the state of that, and I, and I was kind of like, well, why don't you guys have your main system
running on it? And they, well, we don't really think Windows is a serious platform, you know, we've
got Unix, you know, our Unix systems is what all the people are really seriously using, and
the Mac one will just be kind of like, you know, we'll see if anybody wants it. But Windows was not
of interest. And then I started going around and going, well, what Lisp's are running on Windows?
Because at that point in the late 90s, it looked like Windows was the platform. I mean, Windows was
taken off like crazy, and it really was powerful. It was at that point, it got a lot better than the
Mac for some years before OS X. And so Lispworks wasn't running on Windows. They'd see Lisp,
but that was kind of just a port of, you know, something, you know, bytecode compiler.
And it was like, I, you know, there's a market for here for like a serious Lisp on Windows.
I remember Gold Hill, but I don't think that it got, I think it might have been on
like Windows 3, but not on like Windows 90. It could have been that that still existed,
I never played with it. In any case, it looked at me like there was a big opportunity for a good
Lisp. So that was when I just decided to take, to get serious and do something like as good as,
trying to do something as good as Mac common Lisp in terms of performance. And, and, and it had to
be a complete rewrite because I was never going to be able to do it with the code that I built
already. And I knew it needed like a good garbage collector, like generational garbage
collector with taking advantage of real, you know, hardware virtual memory and, you know,
fully tagged. I, I kind of modeled it on, in all ways pretty on, on what I'd read in Mac
common Lisp's manuals, which are extensive and, and very enlightening. And again, you know,
it wasn't that easy to find stuff like this, like finding stuff about garbage collectors had been
impossible in the 80s. And in the 90s, it was possible, but mostly it was just some papers
that like Henry Baker had written, like you could find a few papers here and there, but
figuring out how to build some of this. And, and, and I wanted to do stew 100% compiled,
native compiled with no interpreter, because actually that's one of the things I hate about
common Lisp is the interpreter slash compiled dual execution path.
So to step back, everybody's got their own ideas, what they like. And I'd say my favorite things
about Lisp, these are the things I think that make Lisp amazing.
S expressions. Is there still not another language that has anything like S expressions,
you know, where the code and the data are the same format?
XSLT. Okay, yeah. Not as elegant, but okay. Yeah. The Conzing engine, by Conzing engine, I mean
generates conzes fast, doesn't, you know, you've got a garbage collector to match. It's like every
other language can't can't compete with a good Lisp in terms of like allocating heap stuff,
little heap objects. I found Java VMs usually orders of magnitude slower at generating tons of
little things. And so the fact that a Lisp can like generate conzes, you know, these eight byte
or 16 byte little pieces or small, you know, tag floats or whatever can generate those, you know,
generate the heap for those and then clean them up when you throw them away, you know,
really fast is a really key element of Lisp to me. And the fact that you could build any data
structure out of conzes is great. I know, I know. Yeah, yeah, that's one of the things I don't like
about Clojure, I guess I'd say. But macros, well, I've said, I guess there's been so many great
stuff written about macros and so many great macros written. Everybody's got garbage collection
these days, that's not such a big deal anymore, but a really efficient one, as I said, for getting rid
of tons of tons of people worry about people were always worrying about conzing, you know,
when we're Lisp people were writing, oh, don't do this, it'll generate too much cons. Well,
a good garbage collector is, you know, the amount of time it takes is relative to the
amount of live stuff that you're keeping around, if it's written right. It's not based on how much
stuff you're throwing away. In fact, the one I wrote takes zero time to throw away to delete
something. I mean, it's all it does is pick up the live objects and defragment them and move
them to a different heap and zero time is spent, you know, deleting the dead objects. So you could
generate millions of cons and not worry about garbage collection times. And I really like the
multi-paradigm aspects of common Lisp anyway. You can get, you know, objects if you want,
object-oriented paradigms, there it's great, it's like one of the best, but you don't have to use
them. You can use functional, you can use just simple, you know, procedural programming, works
great, whatever you want to do, declarative, you could make your domain-specific language.
Just a lot of different, I love that, you know, I just love not being constricted and having that
flexibility. Some things I don't like about common Lisp, and I'm bringing these up because if you
ever used my system, you'll notice, you'll probably notice, but I don't like the interpreted
compile or compiled model. Oh, I'm a Lisp programmer and I write a function. Oh, and I run it.
Oh, well, but it's not as fast as it should be. Oh, well, that's because I didn't compile it.
Okay, I compile it. Now it does the same thing, but faster. Well, why didn't it compile it in the
first place? I didn't get the, oh, because then you wouldn't be able to step through it or so,
you wouldn't be able to trace. You know, to me, that's just crazy. You know, all the common Lisp,
all any language should do is it should make your program run as best it can, you know, when you
enter it. Yeah. And don't have an interpreter. I don't know why you say that because I'm,
because Franz doesn't do, I mean, Franz Allegro interprets and compiles. I mean,
I spent three years working with that recently. So, I mean,
you could change the default.
Yeah.
All right. I'm glad you told me that. So, I'll remember that next time I'm using it.
And, and, and as I said, Franz is like everybody, I love everybody at Franz and
they have such a great compiler. It's the class in common Lisp compilers, my opinion,
and I did end up spending $5,000 on a copy of it when I needed it for a job.
I think that the declaration in syntax is atrocious unless I just hate declarations.
Like they just look ugly. They get in the way of perfectly good code.
It's a bunch of stuff that the compiler writer, I mean, that the programmer is supposed to do
to, to, to give hints, but you know, what is, what does they know? What does the programmer
know about that? I don't know. It's like, why, why should they have to worry about that?
It's such a huge language that it's almost impossible to implement. I mean,
having worked 10 years implementing when I say that, it's like ridiculously big.
Anybody who has, it's ridiculously big. I mean, I would never, ever take it on again,
but I got it, you know, at first, common Lisp, the language one wasn't that bad,
right? But I mean, I'd barely finished implementing that in common Lisp, the language
two came out and it was twice as big suddenly. And then I started implementing that and then
the ANSI spec comes out and it's like twice as big again. I mean, I like, I'm a guy who likes
a big toolbox. I like a programming language. I don't mind that there's different ways to do
things. I think that's all fine, but it almost is impossible to implement, you know. I mean,
it takes a big organization now. You know, one person cannot do it.
Is this because it was, like before there was this idea of a standard library,
so you had to put all the stuff in the language and that kind of
comes out of this? I think partly that. You know, you would say it's,
nowadays, tons of common Lisp standard would be in libraries and not just, you know,
and so they would be more optional. Like if there was a commonly agreed on subset,
that was useful, implementers wouldn't have a problem. They could start with that, right?
And then they'd have a useful language and then, you know, that could be like the first phase
and they could have, then they could go on and implement the rest. But as it stands now,
if you don't implement like pretty close to 100%, tons of people are going to have, you know,
tons of common Lisp packages that are out there aren't going to work because they use everything.
There's not a commonly agreed on set of things. I mean, it's, all gets used. You know, I mean,
that's a tribute to the people that are doing it. They know their language and they know what
they found these tools, they use them. But it's, I don't like the loop syntax. We were talking
about that a little, you know, to me, it's not Lispy. So I put in, I just, I think I stole it from
from Steel Bank, Common Lisp or something, you know, like it's like public domain. So I didn't
write it myself. But, and the whole, the implementation is smart enough to do X, you know.
It's kind of like the people that designed the language weren't really sure how something would
work and they realized that it was awfully complicated and it looked like it was going to
perform pretty badly. But those compiler writers in the future, they'll be smart. They'll figure
out some way to implement around that. You know, it's like, oh, so many times I bump into that
in this spec. Oh, yeah, well, C loss. I think it's the greatest object system there is.
And it's also, in some ways, one of the worst. I mean, it does everything and that's awesome. And
you can hold it up in anything anybody else's object system can do. It can do better multi
methods and everything else. But partly, again, that implementing it and making it efficient is
like so hard. And then being a programmer and trying to like get your, you know, understand
how to make it work and to make it work well is so complex. And honestly, I'm most of the time,
I just use def struct. Like structures are great. They're efficient. I know exactly what they're
doing. There's nothing complicated about them. I don't know. I'm just not a big fan.
It took, it took a long time to get the garbage collector working. At least a year, you know,
reading everything I could find about garbage collectors and all the different ways that
you make them work and copy in collectors. And there was Henry Baker wrote a ton of papers that
were really cool, you know, really useful. And finally, I mean, I built basically built an IDE,
built the code generator, the assembler for Intel, built an FFI based on kind of the API that
fonts used in the old version of the old bad version of Allegro for Windows, which they
kind of spanked before when they found out that I'd done it because they have a better one.
But it works and release this around 2000. My son, when he was young, he was somewhat of an artist
and he, he made this picture. I thought that that was appropriate, that sort of looked like,
you know, my, I've been on the keyboard too long and my fingers are kind of filed down to points.
And my hair is spikier than usual.
So when I'm about ready to like do a first release, and this is a single threaded lisp and
Reiner Yoswig, you know, is emailing me and going, you should do multiple threads. You have to do
multiple threads if you're coming out with a new lisp. And I immediately knew he was right,
like, because I was already using threads in my C++ programs and stuff. And I said,
yeah, but all the lisp's I knew you were using lisp threads, you know, sort of cooperative
threads that you could start and stop easily and, you know, manage in the lisp system. It was
really one big thread by the operating system. And I think that was not, not right. I mean, I,
I did, I knew enough about working with threads at that point that I knew you had to use the OS
threads. You had to support that. Windows supports it very well. And, you know, the lisp threads had
to be real OS threads or was not competitive with something like C++. And that way, if you've got a
multi-processor machine, which are not too common back then, but coming along, you have parallel
execution, you know, real parallel execution. And that was awesome. I mean, I, I decided to do that.
And it changed practically everything. I mean, it really had to redo almost everything to add thread
support. But I may have been the first or one of the first to actually have, you know, OS level
thread, threading in common lisp. I hate without interrupts. That's one of those things that you
see in a lot of lisp code that uses these lisp threads. It's like always, oh, well, my synchronization
method is that I'm going to basically turn off all the other threads when I change this variable.
You know, and in Windows, in OS threads, you just can't do that. You know, you can't just like
arbitrarily keep switching threads on and off. You know, you've got to send them a message or
you've got to kill them or something, but they, they're running on their own. You know, you don't
have control, that kind of control. So I think CormidLisp has a without interrupts in it, but it
kind of not probably really that good to use, not that safe or efficient.
Oh, so just to give you an example of all the things had to be changed to support threads,
special variable bindings. You know, I love special variables. I think they're an awesome
feature of lisp. I should add that to that list actually, that list actually. But according to
the CormidLisp spec, it, you know, the CormidLisp spec doesn't address threads. And if you read it
precisely, it even basically, you know, says that, you know, when you bind a variable,
it says all programs. It says all lisp programs will now see the new binding.
And, and that, I just realized that can't work. That can't work in a threaded system. Each thread
has to have its own bindings. And I, and I don't, I haven't looked into what other lisp's do with
this, but I just implemented every thread has its own bindings. You know, I think that's the only
way to do it personally. But, but once you do it, it's like magic. It's like once you do it now,
you can have local per thread global variables, special variables, as easy as using let. I mean,
it's like, you use them, it makes writing thread safe code actually very easy. Just rebind the
variables using let to themselves or whatever, rebind them at the beginning of your thread.
And then you're good. You have your own copies that only your thread knows about. It's very
efficient and very easy. Heep design and access is affected. You know, you've got to make sure that
you different threads or any thread can trigger garbage collection, right? Could make a memory
call, triggers garbage collection. That means any other thread between any two instructions can,
can have the whole world swept away. That's how the code, that's how the code is. Eating any
two instructions, hardware instructions, the garbage collector can try to like,
and when the garbage collector runs, it moves everything and every address changes. Every
function address changes, the whole world changes. Stack updated, you know. So it's kind of hard to
write code, to generate code that's safe in that kind of an environment when that can happen at any
moment. And a lot of stuff had to be figured out to make that, make that work. Foreign calls and
callbacks. Threads make that more complicated. Your foreign libraries might not support threads. Who
knows? Code generation. Can you generate code in parallel on multiple threads or should you?
You know, they might be trying to update the same functions.
Debugging, of course. Multiple threads. Lisp image saving and loading. If you got a bunch of threads
running and you save the image, you know, are you going to try to restart all those processes when
you load the image? And, you know, thread can come in and call your Lisp code that's not even a thread
you've seen before. It's like one that, you know, where that come from. Now, now you have to initialize
yourself with your Lisp variable bindings or something. You can have problems with machine
architectures that when you're doing a set that they, a simultaneous read can get a different set
of bytes out of the same variable. There's problems where that happens. Yeah, I can imagine.
And this is just a simple, or this is just an example of special variable binding.
In, in Corman Lisp, but, you know, if you've got, instead of just having a slot in a symbol that has
the global variable binding, really those have to be sort of compiled into a table. They're not on
the symbol anymore. They're in their own table. And you don't want to put every symbol in this table
because, you know, if you've got, you might have hundreds of thousands of symbols in your, in your
running system. And most of them, 94% of them, you're never going to rebind. So we only care
about the ones that you're going to bind dynamically. So the compiler and the runtime system needs to
maintain a symbol table that can grow, that only has the variable bindings that you need.
And then every thread has to have its own copy of this table. So that it can, you know, this,
this thread can have its own local bindings of those variables, whereas these are still
referring to the global bindings. So that, that's how Corman Lisp dynamic variable binding works
basically. I like getting down and dirty in code. I like just like, as you could tell from what I've
told you, I mean, I like disassembling stuff. I like seeing how things work. I like if somebody
has some new API or some new DLLs or some library, I'd like to be able to just write
some Lisp code that calls it. And it's just really important to me to be able to do anything I could
do in like C. And so Corman Lisp really brought out the sort of the low level ability. The fact
that it's only on Windows made this possible, right? If I was trying to support a bunch of platforms,
like say Franz does, I probably wouldn't have done this, but we, I basically built a lot of
different ways to call assembly language, extend the assembler, you know, to make new operations,
which generated code in different ways. A lot of like hacky kind of things that you could do. So
you could easily use it to use, deal with comm objects in, in, in the Windows world comms a big
deal. These certain kind of a, you know, built-in object model in Windows. Def Code Gen lets you
define your own sort of code generators, almost like it, what a special form would be.
And so here's an example of a, just a function, Def ASM, that's all written in assembler,
but right in the Lisp file. And the curly braces are used to designate an assembler block. And as
you can see, it looks like, just like Intel assembler basically, other than the, you know, the macro,
the args offset macro, and a couple of things like that. And if you want to mix assembler right into
a Lisp function, you can do that too. I use the double curly braces in that case, but that's
just an example that, that it's a normal Lisp function, it does some Lisp stuff, and then it
does some assembler. So you can probably imagine, if you know what you're doing, you could do almost
anything with this. The assembler has macros, so this is how you define an assembly macro.
This particular macro is the call. It's the way that most Lisp functions get called
through callf, and callf pushes a, pushes the environment, you know, lexical environment on,
well, it actually puts it in the DI register, and then calls the function. So it's a,
and then we added a way to just basically take big chunks of C headers, and bring them in,
you know, so that instead of having to, you know, write Lisp code to, to make the wrappers and
bindings on all of the C functions, or the, you know, and this, this thing, it doesn't show it
here, but it handles structures, and a lot of macros defines lots of things, you could just
bring direct .h files over, and it will, you know, pull them in, and let you call the functions in
Lisp, it's very easy. So I think between all of those things, you know, it makes it easy to use
it for, I can make a Photoshop plugin out of, out of Cormand Lisp, you know, I could generate a DLL
that, that could be used as a plugin. What's that? I haven't tried, but probably.
I don't know. We can talk about it.
I give a little quick demo here of Cormand Lisp running, nothing too, you know, it's kind of
hard to show this kind of thing, but I have to stop my slideshow, otherwise you won't be able to see
it. Oh, what's Cormand, what's that complaining about? Remind me later. When you have Windows,
when you have a Windows running on a Mac, it never stops trying to update stuff,
you know, it's like, it just drives you crazy.
So, I'm going to put this down for a minute. It's not showing up over there. Oh, it's not.
Okay, so why did it not show up? Just a second. Let's go to displays, and let's say
arrangement, mirror displays. Okay, and it's showing up now, right? Okay.
Now, okay. I just made a program called mines. I took a program that I found online that's kind
of interesting, and a lot of people have sort of ported it to different languages,
and it's called minesweepers, and so I just, for fun recently, ported it from C++ to Lisp,
and we've got, if you look in here, it's got some interesting
bunch of matrix math stuff that we carried over, and it's got neural nets,
so the idea is these minesweepers are really dumb when you first turn them on,
and then as they successfully find the mines, they get smarter, and then there's a genetic
algorithm in it too that sort of has a random bit pattern that allows it to, you combine those
between different successful minesweepers and they create new ones, so it combines neural net
and genetic algorithm, and it's kind of cool, so I'm just going to execute this and run that in a
separate thread, TH create thread, and you see you've got these little guys,
I can change the size, and it'll start to put mines in the other part of the window,
they're really stupidly just going around randomly now, they're not going toward the
mines particularly, but once in a while they accidentally get one, and they'll go through
generations, and you can go hundreds of generations, and after hundreds of generations it will start
to, they'll start to be smarter, it seems like it takes too long to me, I think the
algorithms could be tweaked to have them learn faster, but kind of interesting if we took,
if we wanted to just make some little dynamic changes, let's see where we are,
I've got a couple of variables here, if we just take,
let's see, sweeper scale, I'll take like the sweeper scale for controlling the size,
and we just, we wanted to just make it a little bigger on the fly,
you see they got bigger,
they have senses, they know where the closest mine is,
but what they're not programmed to do is go toward it, like they have to learn how to go
toward it, so you see a lot of them just like turning in circles,
but just show you, I mean again, we've got fully compiled code and we can modify it on the fly,
and that's pretty cool, and I can save it as an application, and if I was developing a game,
I think it'd be pretty nice, pretty nice interactive development environment,
and if we up at the top, you see the little animation gen zero, gen one,
that's the garbage collector, it's mostly just generation zero, it's just the ephemeral collector,
it's got one generation that keeps getting collected, generation one fills up a lot slower,
and then the main one, you'll hardly notice it growing at all, because most of this stuff gets
thrown away, by the way on the threads, each thread has its own heap, because otherwise we
have to synchronize around cons, and we don't like to do that, so cons is really fast because
each thread gets its own sort of mini heap, that it keeps refilling from the ephemeral thread,
and so they can allocate small things really quickly, if they allocate something too big,
then they have to synchronize.
I'm having a hard time hearing.
Can a thread be handed a cons or anything else that belongs to another thread?
It can be handed a cons, yes.
For example, if you didn't rebind a special variable, those special variable bindings are
shared by all the threads, and so they're certainly sharing can occur, and since common
list doesn't define what things like packages and hash tables, and stuff, it doesn't define whether
those are thread safe, because there's no concept of threads, I had to make a lot of just decisions
about what to do. I made hash tables synchronized so that threads couldn't share a hash table,
and they wouldn't have problems, but there's a little overhead there, and probably packages
should be synchronized, and I'm not sure if there's more things that could be synchronized,
but generally, I think you all know you don't want to do that. Generally, we try to avoid sharing
between threads, and if you get an enclosure, they enforce that a lot better.
I don't have too much more anything interesting to demo there, but I will
you know, it's got some nice features where you can, if I wanted to find, you know, the code do
anything, and you see I see I get a little tooltip there, and I can also right click on it and look
up the source, since we include the source to everything, you know, you can always look up
the source to every list function, so here I just looked up the source to def var, right, just a macro,
and for me I really like, I like to be able to look up the source to all the list operators
in any list I use. You could disassemble it too, but with a macro that would disassembly wouldn't
do you any good. Any questions on Corman Lisp?
What are we here? Cool. I have a few more slides I could get to, but I know we've gone pretty long
here, so I definitely have gone.
So people complain about Corman Lisp and it hasn't been for years, and I was wondering if you think that
people complaining about the language is fundamentally missing the point of what this is about.
If you disagree with how the language works, if you disagree with Desh, Desh's director, Declaspy,
just tell the trees to rewrite themselves, create them in the first format, it's trivial,
and everyone should be able to do it, even if you're an agent. Well, I agree with you on that.
I certainly feel like Lisp is so easy to modify and so easy to make it be whatever you want it to
be. That's why I say it's a family of languages. It's like, if it doesn't have a particular feature
of another Lisp you like, and Corman Lisp, especially Corman Lisp, is just easy to implement
anything. I feel like Closure has a lot of nice features, say, but I could implement those
Closure features in Corman Lisp pretty easily, I think. I know there's some parts of language
that wouldn't be as easy, and the Java VM interface, obviously.
Two questions. In creating your own Lisp, did you make any innovations around IDE in your
own development environment? Did you use Slime and standard stuff for your own thing?
Well, I built this IDE on the Windows Rich Text Edit, which is built into Windows,
so the text windows themselves are all rich text edits, which are calm objects, which
Corman Lisp then calls. When we see a lot of these editor features, those are all written in Lisp,
running in Lisp, the syntax, coloring, and stuff like that. Then it's a C++
Microsoft Foundation Class' MFC framework. I'm not a very much of a UI programmer, and my UIs
kind of always suck. They're not really beautiful, and that's not where I spend my time. There's
nothing like really fancy here, and it looks outdated now. I would like to rewrite the whole
editor in Lisp, and that would not be hard to do. It's like I needed this to build it. I needed
something that it worked with nicely to build it. When I started, I started with C++ editor.
I have a follow-up question. The same question is about around the database.
Did you make something like your own version of Lego Cache, or did you just make a
connector to a relational database? I've tended to use SQL databases for most of
my projects. To me, SQL is kind of like Lisp. I mean, SQL relational database is kind of like
Lisp in that they started off really amazing and theoretical, but they sucked performance-wise,
and then people just kept putting more and more and more and more effort into them, and they got
better and better. Now you have these amazing relational databases, and you have these amazing
Lisp systems that so many man-hours and effort went into. They're kind of getting a bad wrap
now with everybody wanting to go to object databases and no SQL and whatever, but generally I'm still
content with modern relational databases, and that's what I've used. I'd like to do something
with the Lego Cache. I think that's an awesome system too.
I definitely recommend a Lego Cache for any system where you didn't need SQL, you didn't want it.
Yeah.
So, what's next? Are you going to sort of just keep proliferating different kinds of
formulas? Are you going to do more DSLs?
Okay, so I might have an interesting slide or two left, so I'm going to go ahead and restart this here.
I kind of stopped working on it. Well, I did stop working on it for a long time because the
recession hit and our business was, you know, I have a software company, and sort of we've been an
independent software company for 15 years, but it just, you know, all our clients and contracts
went away in 2008, 2009, 2010, and we were kind of hanging on for dear life, and I just couldn't
justify these, you know, nights and effort spent enhancing Corman Lisp, which I knew was never going
to be profitable. And, you know, we just, I just said, you know, and I couldn't keep, you know,
and, you know, the certain amount of client support calls you get and stuff, I just, so
I just, I just stopped doing it. And, you know, a couple, you know, people started letting me know,
yeah, what's going on with Corman Lisp? And I would always say, well, it's on hold for now,
you know, I don't know. And that's kind of where it stayed for, for like six years, until Zach
Bean came along and said, well, why don't you just open source it? So, you know, people, other people
can work on it. And I said, that sounds great, because I'd rather other people worked on it,
and it continued to evolve. I think the code's written well, you know, but on the other hand,
it's incredibly complex. And I always kind of wondered, could anybody else work on it? You know,
could anybody figure it out to like fix the bugs that come up in the garbage collector or whatever?
I can barely remember or figure some of that stuff out. But once I open sourced it, it's an
MIT license now. Zach did the work kind of like, you know, getting rid of the licensing stuff that
was in there and everything. And one Artem Bolder of, I think that's his name, right? He's a Ukrainian
guy, a young guy. He, he's been taking kind of taken over, done a lot of stuff on it. And he fixed
a really significant bug that I had with 64-bit windows. Like, like, mostly every program out
there that runs on 32-bit windows will run on 64-bit windows as a six, as a 32-bit process.
This was one that had some problems. And it turns out it has to do with the thunking between the
34 and 60, the 32 and 64-bit layers. When you are calling out to a OS call, it's actually always
thunking the 64 bits and then coming back to 32 bits. And structured exceptions don't propagate
properly across that thunk. And we were relying on structured exceptions to implement the right
barrier that the, that the garbage collector use. You know, probably don't know what a right
barrier is, but that's something that has to do with the virtual memory system and makes the
garbage collector really fast. And that was like crashing when you were in a 64-bit OS and you
were doing, you know, FFI calls and callbacks, you know, and he tracked that down. And I was very
impressed. He tracked it down and figured out how to catch the exceptions and how to, you know,
repropagate them and got it working again, you know, in 64-bit. And so then I said,
okay, I want to work on it now. And I've been, I've kind of gotten back into it in the last,
in the last year. Having someone else to work on it with is really great. And of course, that's
the beauty of open source. You can find, I've got somebody from Ukraine. A lot of the work was done
by Vasily Baikov in the earlier days, who's Russian. And he did tons of, he did tons of
stuff in Common Lisp. And it kind of sounds like I'm colluding with Russians, which I am.
But then basically since then, I've started to use closure. I like Common Lisp, but I'm,
I will, I'll never, as much as I love Lisp and I evangelize about it, I won't,
I won't promote it to a client if I don't think it's the right tool for the job.
And a lot of times I think closure is a better tool for the job these days. More,
more people are getting into it. The fact that it runs on the Java VM and can run everywhere.
The fact that it's simpler, it's a simpler language. The fact that it can run on the
.NET, which is a platform I've been heavily involved with, the .NET version of closure.
And there's a lot to like about it. Macros are good enough for me, the way they did macros,
it's, you could do what you want with them. As we said, the S expressions are a little different,
everything's like, there's no dot pairs. You can't put whatever you want on the right side of a
con's call, which is kind of weird. Weird and good, depending on your perspective.
I've never really used scheme very much, but I'm really interested in it.
And I don't know why, I just, I got into Common Lisp early and I just, you know,
but one of my favorite books in the world is Christian Kennecks Lisp and Small Pieces,
which if you've ever read, you know, he's a scheme guy and he writes like 30 different scheme,
Lisp and scheme interpreters and compilers that he presents in this book. And it's,
for me, it was nice, nice bedside reading. So I'm building another version of my Arden
syntax compiler, this one, one that I own, using closure. And it was kind of neat because we could
take, if you, if you know how you build a compiler, you have these Bacchus Naur forms. And I just took
what you see in comments there is straight out of the spec for Arden syntax, the BNF,
straight out of the spec. And then I sort of just made the, you know, made a, made macro,
a macro parse rule that takes the language as it shows up in the spec and turns it into a
code generator, a parser and a code generator. And the next expression there is the generated
code basically in, in closure. And so this parser actually works just off of these. And it makes it,
you know, pretty easy to write the parser and pretty easy to update it. And if the spec changes,
it's pretty easy to see when it diverges or when, you know, where you have to change.
It's slower than my hand coded C++ parser that I wrote before, but I, you know,
probably could still make it faster. But it was a nice project to do in closure. And I,
I've liked it for that. I could do it in common Lisp too. It would probably be even,
you know, could probably have written it more easily. But again, I wanted to run a .NET. And
so if I said what I like about closure, all those things that I liked about Lisp,
S expressions, conzing engine, macros, garbage collection, functional paradigm,
not so much on the object oriented paradigm or the procedural, the fact that it has such a user base
and very good implementation and runs on practically every platform.
I, as I say, I think all Lisp, Lisp languages are Lisp family. They're all part of the same family.
And to me, if I love one, I love them all. And, and I could, I could, with macros, I could make
closure do what I want. I'm, you know, I'm sure of that. It does take some getting used to not
being able to mutate variables, getting rid of set queue, basically. And you could get around it
by just creating single cell Java arrays and using those for your mutatable variables that
can be shared. You know, if you have to, it's pretty easy to get around. So it's not like,
but of course it's kind of ugly. And the whole Java interface part of it's,
you know, it's, you have to use it too many places, I think. But,
and the last thing I had here, I was going to mention is I'm getting into continuations now,
which I never did. You know, all these years of doing common Lisp, I never used continuations.
And I, all I thought was like, I'd never wanted to try to implement them because it seemed too
hard or too inefficient to be saving and restoring the stack all the time. And so
now I have a language I'm implementing that uses fuzzy logic, new version of Arden, 2.9.
And the control flow is fuzzy. So if you have an if statement and the condition is
somewhere between true and false, it splits into two paths of execution. And so this kind
of an execution model requires backtracking and something that continuations can do. And I don't
know what else could, you know, and the only other alternative that I've come up with is
to do in continuations is to use continuation passing style, you know, where you every function
gets a continuation passed in and returns a continuation. And then you use a trampoline,
which is just a little loop that just keeps calls, gets a continuation back, calls that
continuation back, gets a continuation back. In the end, you have no stack. So your continuations
don't have to worry about saving and restoring the stack. I mean, you avoid that problem,
which is what's hard about continuations, saving and restoring the stack. And you cannot do them on
you cannot do that on Java VM or CLR, to my knowledge, business and support that.
Have you considered rewriting some of those algorithms and say using a star or some other
form other than continuation? You know, I don't know exactly the details of the problem you're
trying to solve, but maybe you're out with the wrong algorithm. Yeah, and I bet you could help
me with that. It's actually one of the first problems I've ever bumped into that I didn't,
you know, I still haven't solved. I mean, I'm having a tough time with it. So I'm open to
suggestions and advice. Okay, so I know all of you have like great stories about what you've
done with Lisp. And I'd love to know more about them. And I'm really, really happy I got to be the
one up here today talking about my story, not that it's any better than anybody else's story,
but I'm thankful that I had the chance because normally I never find anybody that wants to
talk about this stuff or hear about it, right? In my regular life, right? So getting to meet,
you know, be with this many Lisp programmers at once is really unusual and just great for me.
So thanks a lot. And I hope you've learned something of value today.
Thanks.