D Day Two: Conclusion of “Future of D”

Walt and Andrei closed the second day and last formal of the conference (Saturday was more open discussion). This post is a continuation of the first day’s notes. Also note that the slides for all talks have been uploaded by Brad, and videos will appear in the future. Again, disclaimer — any errors are mine, not the speaker’s.

Jumping right into Walt and Andrei’s talk…

Object Model Improvements

  • struct ctor/dtor/opCopy/opAssign – Enable the creation of ref counting template wrappers for objects. When copying value objects: 1. member-wise copy 2. call the opCopy method (if supplied)
  • opImplicitCastTo – enable implicit casting of one user defined type to another
    1
    2
    3
    4
    
    struct S {
      int opImplicitCastTo() {}
      float opImplicitCastTo() {}
    }

    A very useful feature for library designers to take a burden off library users.
    As Walt presented it, Andrei expressed some distaste for this opImplicitCastTo feature (perhaps distaste is too weak a word?). Or maybe it was one of the many other topics where Walt and Andrei sparred. Throughout the presentation, there was good-natured banter between Andrei and Walter that was fun to watch.

  • opImplicitCastFrom – enable implicit casting of objects which you don’t own the definition
    1
    2
    3
    
    struct S {
      static S opImplicitCastFrom(F f)
    }
  • Polysemous Values – lazy, late determination of type when intent is not clear.
    Quiz: What is the type of X?

    1
    
    int i; uint u; auto x = i + u;

    Answer: there is no good answer until you see how the value is used.

  • Arrays and Slices
    Arrays will be separated into resizable Arrays and non-resizable. Resizables can be converted into a slice, but a slice can only be converted into a non-resizable. There was some concern about backwards compatibility here.
  • struct “inheritance”
    Based on C++ concepts
    How can we be sure a struct had a list of functions? Answer: if a struct inherits from an interface, the compiler will require that the struct implements all the features of the interface.
  • Immutability – still don’t like the solutions out there. There’s no consistency among languages in how to treat consting a chain of pointers/references. With D, going to take another look at giving programmers the ability to state their intent clearly.
    final – means “const” of the head
    const – means “const” of the tail
    final can be combined with const or invariant
    In Q&A, the room suggested keywords that matched Walt’s verbal word usage “head const” and “tail const” (with head/tail being context-sensitive keywords only — too common to be reserved).

Simplifying Code

  • Static Function Parameters – lets us specialize functions on static arguments
  • Order of function argument evaluation – new rule: always evaluated left to right
  • Unambiguous enum names will resolve without having to qualify the scope

Generic Programming

Unfortunately, they ran out of time and had to skip some slides in order to get to the important proposed macro functionality. They skipped to this section. See presentation for some smaller items that were missed.

  • AST Macros -
    1
    
    macro foo(e) { e = 3; }

    invoking with(*q);
    will replace foo(*q) with *q = 3; (THEN it does the semantic analysis)
    C preprocessor macros are widely considered to be evil — so why introduce similar macros in D? Because they’re only similar in goal — the very different implementation solves many of the C issues. In D, the semantic analysis is done after the macro is expanded (Walt noted that a similar implementation would be possible in C, but not be possible in full C++, because C++ parsing has semantic dependencies). And then the substitution is not at the character/text level — rather it’s at the abstract syntax tree level. So rather than being a preprocessing trick, invisible to the compiler, it is better integrated — so the compiler can provide more help and information at compile time.

  • Pattern matching in macros – Macro arguments an be pattern matches analogously to template type parameters
    1
    2
    3
    4
    5
    6
    7
    
    macro foo(e) {}  // (1)
    macro foo(e: e+1) {} // (2)
    foo(3); // matches (1)
    foo(3+1); // matches (2), e aliased to 3
    foo(x+y+1); // matches (2) -- e gets aliased to (x+y)
    foo(1+x); // matches (1) because matching predates semantic analysis
    foo(x+(3-2)); // matches (1) because matching predates constant folding

    This is one of the new pieces of functionality that potentially could help clean up the creation of domain specific languages in D.

  • Hygiene
    Symbols defined inside the macro are invisible outside. Symbols resolved inside are in the context of the expansion of the macro (not the invocation)
    Proposal is to define an unclean extension — $symbol — in the macro expands

  • string literals – 3 new syntaxes proposed
    1
    2
    3
    
    q"(foo(xxx))" // "foo(xxx)"
    q{foo}		// "foo" has to be a sequence of D tokens 
    // (e.g. q { 67QQ} error, not a valid D token)

    And Heredoc strings

    1
    2
    3
    4
    
    		q"EOS
    		This is a multi-line
    		heredoc string
    		EOS"

    Coming from the Ruby/Perl, I appreciated these block string literals. Walt noted that this was one of the few features already with an implementation in the private D branch.

  • return Storage class

Standard Library

They had no time to speak in length, but mentioned to a top feature was adding STL-style containers. No comment on Tango.

So there it is. Every one of the talks was very interactive. Perhaps because it focused on the future of the language, this one particularly so. Lots of questions interrupting each slide. So keep an eye out for the videos to be posted in upcoming weeks on Brad’s conference page. And check out Planet D for other commentary on the conference. As a D outsider, it’s great to see the unfolding of this attempt to tackle the language design challenges that have kept much of the world from moving past the known warts of C.

D Day Two: Other Talk Highlights

Here’s some of what we were treated to on Day 2, before the conclusion of Walt’s “Future of D” talk, which will be another post. These are just some of my notes and thoughts. The presentation slides have been put up by Brad Roberts (who did just an awesome job organizing this first conference!) at the conference slides page.

Kirk McDonaldConnecting D and Python

Kirk walked through the capabilities of Pyd, from the straight-forward features (calling D code from python), to the advanced (python callbacks, subclassing in the other language, etc.). As with all the speakers, he fielded a ton of questions, and Kirk handled them particularly well — showing deep knowledge of python, D, and characteristics of various other languages and systems which have cross-bindings.

One interesting part of Kirk’s presentation was looking over at Walter Bright’s reaction — and that was of rapt attention and knowing glances between him and Andrei. Why? Because the mission of D is to make those things that can be done by the compiler, easy for the programmer. And connecting D to the universe of other languages and runtimes (Python, Perl, Ruby, .NET, JVM, etc.) is one of those essential common cases. Kirk has already had an influence on D’s compile-time reflection features, and clearly this talk got Walt thinking about more that can and should be done.

I got a chance to sit at a table with Kirk on Thursday at F.X. McRory’s. Kirk mentioned then that’s he’s down in Portland, only with a two year degree, looking for a job. I hadn’t seen the presentation yet, didn’t know what his skills were — so gave all kinds of probably simplistic advice on the value of a full degree and how to land a position. After seeing the presentation and his depth of knowledge in the Q&A period — ok, forget all that — just hire him already!

Don ClugstonGenerating Truly Optimal Code with a Metaprogramming Library

As with Kirk’s, Walt just lit up during this presentation. Here you have a domain (vector and matrix algorithms) where, in order to archive the highest possible performance, people are still using Fortran (the horror! ;) ) and optimizing it with assembler (true horror!).

It’s the perfect opportunity for D to show its stuff, and Don has done just that.

The best line of the presentation was one where the room was having a discussion of language features to make this stuff easier. Don took a shot at Walt — “New D features keep making all my best code obsolete! I write 500 beautiful lines of code, and you go make it possible in 3″. Deadpan funny.

Cristian VlasceanuAnatomy of a Debugger

Debuggers are tough beasts to write. Cristian took it on himself, and has done quite an amazing job. Cristian talked much about the general case (non-D code), but also some D-specific issues, including how the DWARF symbol file format doesn’t support some D constructs, which creates some limitations (but there’s an opportunity to create standard extensions). Here are the bullets on his one “D language support” slide:

  • Demangler, thanks to Thomas Kuhne
  • Support for dynamic arrays
  • (Some) support for associative arrays
    –repurposed DW_AT_containing_type to be the
    key type, and DW_AT_data_type gives the
    element type.
  • Requires the ZERO_D_SUPPORT
    environment variable to be set

Benjamin ShropshireCompile Time Abstraction Techniques for the D Programming Language

Ben walked through some template tricks in D, leading up to some examples of compile-time parsing of domain-specific languages with D. D’s metaprogramming features are wonderful, yet this seemed like it had pushed the language too far (which is only a credit to Benjamin). Some of the examples could segfault the D compiler with anything other than smallish inputs. Walt and Andrei mentioned during the Q&A that the upcoming macro support would achieve some of these same goals very cleanly and concisely — better support for creating domain specific languages is coming.

Bartosz MilewskiSoftware Transactional Memory in D

Like many of the speakers, Bartosz had a fun, dry sense of humor — and brought it to what otherwise might have been a dry topic. Bartosz walked though the multi-threaded conundrum of synchronizing access to resources and avoiding deadlocks. Bartosz then made the pitch for a new set of language features to support software transactional memory, a sophisticated strategy for attacking this problem at a higher level than using traditional semaphores and locks at the application level.

When you have an interesting language with momentum — but that’s still evolving as D is — it’s a great opportunity for groups to pitch problems that are hard to solve without compiler support for their solution. STM is one of those. On the face of it, I suspect it’s not worth complicating the language to include support. But luckily, D has a single focal point for these kinds of decisions (Walt) who can be the unfriendly gatekeeper for these simplicity vs. functionality decisions. Like Linus does with Linux, saying “no” so often can be difficult, but it’s the secret sauce that will make a project like these succeed vs. its committee-designed or corporate-designed cousins.

Day 1: Other talk highlights

Kris Bell – Array slicing fo shizzle!

Some of us missed the pop culture reference in title, but this was a effective talk on how the array slicing feature of D is a key performance tool.

Kris had struggled for several years in scaling server-side Java code. He talked through a few of these cases. Repeatedly, the key bottleneck was allocation, working set, and the garbage collector. And Java’s design unfortunately makes many of these allocations nearly unavoidable, especially for the all-too-common task of string processing and web serving. Kris gave some numbers on the dramatic performance increases he saw with various designs that minimized those allocations.

This led into D’s array slicing feature, which lets you easily reference a sub-array without having to allocate a new object. Kris walked through design of a HTTP server with little or no memory allocation after startup and pre-allocation of buffers. This is the design used for D’s Tango HTTP server, and results in its high performance.

Brad Anderson & Gregor Richards – DSSS and DSource

The auxiliary tools around a language — build, test, Internet library repository, and library install & dependency management — are as critical as the language itself to enable a community to collaborate on a universe of effective, loosely coupled libraries, and make easy use of those libraries.

D is just getting started on the tools and web resources of the sort that Perl, Python, Ruby, and other dynamic languages have relied on for while. But it has a real opportunity to be first among the systems languages in this kind of support.

This talk covered how Brad (the creator of dsource), Gregor (the creator of DSSS), and others are working towards this end for D.

DSource

Went live in Feb 2004. Announced to newsgroup in March 2004. First 5 projects: DWT, Mango, eclipseD, DigDug, docoa

Current stats:

  • Operating for 3.5 years
  • 143 projects
  • 60ish active
  • 30 very active users
  • 1000 registered users
  • 15000+ forum posts
  • 560+ tutorials (many by Justin)

Brad is now working on the third generation dsource site

  • Written in Django (Python web framework), interfacing with Trac, Django, and SVN (was Trac+dsource plugins)
  • Central login, logout, registration
  • nicer projects list & search
  • project voting mechanism
  • TracForums – Eric Anderton (as phpBB)
  • Multiple forums, all within each project
  • Consistent look and feel, syntax highlighting
  • wiki page comments

DSSS

Gregor treated us to a insightful, funny walk-though of state of the art of building posix software. Look for the video when it comes out.

He described what DSSS is capable of today, how it solves those build problems in the simplest way possible for the library designer (in the best and common case, nothing extra from the programmer other than the ‘includes’ they are already doing in their D code).

Gregor showed some nice, early integration between dsource and DSSS — this is a rich area for features which would really help library users and writers.

The room had some discussion on other features which would require more configuration, such as library versioning (which is not yet supported).

Sean Kelly – runtime library design

Check out the slides when they come out for more on this. You can also read a nice interview from Sean here.

Day 1: Future of D (1/2)

The Future of D talk by Walter Bright and Andrei Alexandrescu was spread into two parts: kicking off the first day, and closing out the second. A brilliant ploy to keep everyone at the conference in their seats through both the days. So this post is notes just from the “Day 1″ potion of the talk.

The features discussed in this talk are not in D 1.0, and they’re just under consideration for D 2.0. In many cases, it was clear there was ongoing discussion on the details. I believe slides will be posted for all talks, and possibly video. Links will be provided in future posts.

The notes for these posts were taken live, but the posting came a day or two after. And I’m just a part-time language junkie, so I may encode some of my own misunderstandings or bad vocabulary here — those errors are mine, not any of the speakers! And apologies in advance for any errors, as the notes were raw. Now back to the talk …

D 2.0′s Design Objectives

While D obviously has its roots in being a “better C++”, the goal is to be non-religious and do what’s simplest and what works. That includes providing features to support other programming paradigms.

Someone from the crowd asked, “What paradigms are you missing?” Walt answered, “For example, more support for functional programming” — especially for the purposes of compiler-supported parallel programming. And promised more info through the talk.

D’s Design: Principled Approach

  • “No issue left behind”
  • Leave as few dark corners as possible
  • Have good engineering rationale for those that do exist (e.g. what’s the type of adding a signed and unsigned integer?)
  • Avoid the “onion in the varnish” syndrome (loosing reasons why things were done) — hope to move beyond old mistakes that were encoded by inertia
  • Avoid the “what the heck we were thinking?” question

The Rule of Least Astonishment – when you learn a rule, it’s generalizable

One funny term showed up on a slide. “polysemous values” – Some confusion, then laughter from the crowd when walt and andrei admitted this was a newly created word (in this case, to describe values with late determination of type). Must be meaningful in Romanian. [actually, as pointed out in comments, it is a real word, and an apropos one at that]

Features for Modular Development

  • Unified Function/Template Overloading
  • Uniform Function Call Syntax

    The definition of foo(a, args …) allows the function to be called as a.foo(args, …)

    Enables more of a model where the class designer implements the minimal amount that works. Extra sugar can be implemented outside of the class. (e.g. import matrix; import eigens;)

  • Function Overload Sets
  • nothrow

    Java does static checking of exception specifications. C++ does mix of static/dynamic. People hate both. nothrow decorates that a function does not throw exceptions.

  • Pure Functions – “pure” keyword

    Can’t read any non-const global data. Can’t write any global data, can’t modify anything reachable through its arguments, may modify only its own variables and stack. May throw exception.

    What can we do with Pure Functions: common subexpression elimination (don’t need temporary variables to call same fn with same params multiple times), can be reordered and scheduled by the compiler, can be automatically parallelized, can be memoized (prior calls stored in hash cache), can be tabulated (all possible values pre-calculated at compile time for lookup)

Part 2 (upcoming)

Object Model Improvements, Simplifying Code, Generic Programming, and Standard Library will be covered in part 2 of the talk.

Day 1: D Conference Intro

The opening talk of the first D Programming Language was given by Walter Bright, the designer of D, and Andrei Alexandrescu, the C++ guru who has been collaborating with Walter on some upcoming language features.

When Walt started working on D, everyone said “you can’t do a new language”, “it’s too big a project”, “the world is filled with half-implemented, poorly adopted langugages.”. He didn’t want this to become another one.

Walt felt he was in a good position to advance the state of the art, because he had been involved with C++ compiler construction for years, already had an optimizing C++ compiler, linker, etc. to build upon — and could focus work on making a front end that fixed the frustrations so many had with C++. After just getting a start on the project, its existence got outed on slashdot. And thus Walter was committed to make it into a real product.

This spring D saw its 1.0 release. And now, six years after the outing on slashdot, we have a a first conference with people from all over the world (well, 3 from outside the US) focused on the language.

Walt opened his talk by observing that with the momentum that has built — Simply, “D is here”. And proceeded into a great talk on upcoming language features. Highlights and notes coming in a future post.

Tomorrow: The First Annual D Language Conference

This week Amazon is hosting the first annual D Programming Language Conference here in Seattle. Looks like the conference will have just over 50 people attending. There’s a single track of speakers, so I hope to attend most/all the sessions, and blog some notes and thoughts here.

There is so much rich thinking going on with dynamic languages like Ruby, etc. This can be a much more expressive, productive way to program, compared to the more verbose and brittle static-typed language world. And with the web at the forefront, they have attracted a great community that is pushing the envelope in many areas.

But with Ruby, I often I feel like I’m building a house on shifting sands. And there is much that C/C++ can do to improve. Where Ruby is flexible and concise, but loose and slow — C/C++ is fast and strong but verbose and brittle. [That seeming contradiction of being both strong and brittle is a truth, but one that's sometimes not immediately obvious]

D is a tight, elegant evolution of C/C++ that retains the strength, but attacks the verbosity and brittleness from several angles, especially with its metaprogramming features. It is C++ completely rethought, simplified, and done right.

As a device driver developer, the language has features that I often wished I had (like static ifs, better templatized functions, and compile-time code execution). And they are syntactically elegant, unlike C++.

The first talk of the morning starts it off right, given by Walter Bright, D’s designer and mind behind the Zortech C++ compiler and others; and Andre Alexandrescu, the C++ guru and author of the brilliant, dense Modern C++ Design: Generic Programming and Design Patterns Applied, among many other things. D already has the signs of attracting its own great community. These are two minds to follow, and I’m looking forward to this talk and others.