[Gd-hackers] Dylan, C, and me

David Rushby davidrushby at yahoo.com
Fri Oct 14 17:38:13 UTC 2005

I'd welcome advice and corrections from anyone on this
list.  I'm not afraid to recognize my mistakes and
learn from them, so fire away.

Why I'm Here

I'm interested in using Dylan as an extension language
for Python.

Although Dylan and Python are seldom compared, the
similarities between the two languages are rather
extensive [insert snide remark about both Python and
Dylan being poor imitators of Our Father who art in
hibernation, yea, even The One True Lisp].  One key
difference is that the designers of Python have
prioritized simplicity and consistency over
sophistication at nearly every turn.  For example,
Python has no equivalent of Dylan's optional static
typing, no macros, an ultra-simple module mechanism
that reduces optimization potential, no "sealing", and
so forth.  Python also suffers from the intimate
association between its designers and a particular
implementation, so implementation details have leaked
into the language, precluding many optimization

The result is that it's impossible to write
high-performance code in pure Python, and those who
undertake to make it possible inevitably end up
defining a "restricted Python" subset that is
considerably more static.  At that point, most of
these efforts seem to fizzle, never moving beyond the
alpha stage.  If they do progress to become reliable
tools instead of mere curiosities, it is typically
only after their mission has been redefined to
de-emphasize execution efficiency.  Even then,
integration between authentic Python and the
restricted dialect can be a problem.

The Python developers plan a backward-incompatible
overhaul of the language in the medium term,
code-named "Python 3000".  That's a rare breach of
compatibility; one might think they'd sieze on it as
an opportunity to make the language less
implementation-bound and more amenable to
optimization, even if they lack the manpower to
exploit the potential optimizations immediately. 
Unfortunately, I've found no evidence that execution
efficiency is any higher a priority in Python 3000
than in previous versions.  So it appears that pure
Python will remain an unsuitable basis for
high-performance code.

In making these observations, I've probably come
across as a whiner who wants to have his cake and eat
it too.  Actually, I find Python an immensely
productive and maintainable language, and I salute its
developers for propelling Python to its current level
of visibility and penetration with very little
corporate funding.  Countless tough decisions had to
be made correctly for Python to reach its current
position.  But every single Python implementation is
slow, because the language makes no provision for
trading dynamism for efficiency.

My Choices

The most common means of optimizing a Python subsystem
is to rewrite it in C or C++.  Python's C API is well
designed, and I've used it successfully to write
numerous C modules of moderate complexity and one C++
module of high complexity.  I appreciate--even
enjoy--C's unapologetic focus on the low level, but
its minimal facilities for abstraction become a
problem in larger programs.  My grasp of C++ is no
better than intermediate, but I have done extensive
bug-fixing in a medium-sized C++ code base.  I find
that the tension between abstraction and efficiency
creates an unpleasant disharmony in C++, which
surfaces jarringly and sometimes unexpectedly.

Programming in C feels like debating a political
opponent whose views are sharply opposed to mine, but
who is entirely forthright about his beliefs and
integrates them into a consistent philosophical
framework.  Programming in C++ is like facing off with
a brutally effective demagogue who has defeated every
challenger the opposition can throw at him over a long
career, but has never really taken an honest position
on anything, and has only keen opportunism where a
philosophy should be.

It's possible that my distaste for C++ is rooted in
ignorance, but I don't think so.  If there were a
redeeming epiphany waiting at the next level of C++
mastery, I'd have at least a vague sense of it by now.

The need to implement high-performance and occasionaly
also high-complexity extensions to Python remains, so
I've been searching for "a better C++".  The two
primary contenders at present are Dylan and OCaml.

Although I appreciate the sophistication of ML-style
type systems, I've never subscribed to the "static
typing is the alpha and omega of program correctness"
school of thought.  Most bugs in my programs arise
from an incomplete understanding of the problem
domain; errors of the sort that a static type system
could prevent are rare by comparison.  Also, my
preferred programming style is "imperative, with
functional features where they seem natural."  OCaml
allows such a style, but it's clearly intended for
programs in which the functional paradigm dominates,
and imperative features are resorted to when maximal
performance is required.

It appears to me that Dylan is "a Lisp in sheep's
clothing".  I like Lisp; I'd actually prefer to jump
straight to Lisp instead of using Python plus an
extension language.  But I can't seriously consider
Lisp as a general-purpose language because it suffers

  1) A sociopathic community.  Even in the absence of
other problems, the extraordinary incivility of
comp.lang.lisp would be an effective suppressant of a
Lisp resurgence.
     (See the sadly representative thread at
, where a party who poses a reasonable question about
the future of Lisp is promptly awarded "a membership
in the Complete Fucking Morons of Programming Club".)

  2) A standard library that's poised to address the
oh-so-demanding requirements of 1985 (no sockets, no
threads, no GUI, ...).

  3) Not popular enough to have a good selection of
prefabricated libraries.

  4) Fragmented implementations.  If the ease of
parsing programs represented as s-expressions didn't
ensure the fragmentation of Lisp into a multitude of
half-finished implementations, the behavior of the
community would.

  5) A total disregard for Windows support among the
more sophisticated open source implementations. 
Windows is the dominant operating system in my work
environment, so my primary language must integrate
well with it.

I can circumvent all of those problems by using Python
as the intermediary between an extension language and
the outside world.

As I've read about Dylan's design over the past few
days, the language has become quite appealing in its
own right.  It's obvious that Dylan's designers
anticipated at the outset many of the challenges that
Python is unsuccessfully wrestling with today.  I can
imagine Dylan creeping from its original role as an
extension language into areas of my code base that
Python used to occupy unchallenged.  But Dylan must
prove itself in the more limited role first.

Specific Questions

- d2c versus the former OpenDylan/Harlequin compiler

  In the limited benchmarking I've done so far, code
generated by d2c (and compiled with GCC 3.4.4) has
typically been several times faster than that
generated by OpenDylan.
  This "limited benchmarking" amounts to little more
than testing programs from the Great Computer Language
Shootout.  Since d2c is included in that competition
but OpenDylan is not, it's possible that the programs
have been written to play to d2c's unique strengths,
and that the performance advantage d2c enjoys over
OpenDylan would not be as pronounced in general code.
  However, my initial goals for Dylan require
performance not hugely worse than that of C++, so *if
we were to consider performance in isolation from
other factors*, it appears that d2c would be the right
choice for me.  Do you agree?

- Calling d2c-generated Dylan code from C

  Presumably the common ground on which Python and
Dylan meet would be C.  Since Python is "in the
driver's seat" in my case, my initial goal is to call
d2c-generated Dylan code from a C extension to Python.
 For the sake of simplicity, I'm first trying to
figure out how to call Dylan from C alone.

  As far as I can tell, Melange is intended to allow
Dylan to access C libraries, but doesn't provide any
support for calling Dylan from C code.  The "Known
Limitations" section of the Melange documentation
claims it doesn't support "callbacks", and it seems to
me that callback support would be a necessary

  d2c appears not to implement the OpenDylan macro
C-callable-wrapper (there is what appears to be
nothing more than a stub in
d2c/runtime/c-ffi/c-ffi.dylan).  The access layers for
Carbon and GTK seem to define their own ad hoc
implementations of C-callable-wrapper.

  Is there a d2c-specific variant of
C-callable-wrapper that I'm missing?  Does Gwydion
provide *any standard support at all* for calling
Dylan from C, or are individual libraries expected to
roll their own, as the Carbon and GTK layers
apparently do?

  If d2c doesn't provide any support for calling Dylan
from C, why not?  Is it a deficiency that I, who have
no familiarity with the d2c code base, only minimal
familiarity with the Dylan language itself, and only
the most trivial experience with compiler
implementation, could hope to correct without
investing months in preparatory study?

- d2c on Windows

  In order for d2c to be useful to me, it'd need to be
Windows-friendly.  Are there fundamental issues
preventing it from being so, or just grunt work that
I, a newbie, could realistically do?

Yahoo! Mail - PC Magazine Editors' Choice 2005 

More information about the Gd-hackers mailing list