Elmord's Magic Valley

Software, lingüística e rock'n'roll. Sometimes in English.

Why (not) boxes?

2018-08-28 23:28 -0300. Tags: comp, prog, pldesign, lisp, in-english

In the previous post I discussed an idea for dealing with mutable data in a Lisp-like programming language by using mutable boxes and immutable everything-else, with a bunch of optimizations. One of the usual suspects asked me what was the advantage of this scheme over just declaring things const as one would in a language like C/C++. At first I did not have an answer ready. This is one of those situations where you are so stuck in your own perspective that some questions don't even occur to you.1 The immediate answer was that I was thinking in the context of a dynamically-typed language, so an immutability declaration like const2 was out of the picture. But there is more to it.

If I were to give a really complete answer, I would have to begin answering why I want dynamic rather than static typing. I started this post originally by trying to explain exactly that, but there is way more to say about it than I have the energy to do right now. For now let's just take for granted that I'm designing this framework in the context of a dynamically-typed languaged.

But I could still have a dynamically-typed equivalent of the const declaration: just shift the constness to the dynamic type of the object. So vectors and other composite objects would have a flag indicating whether they are mutable or not. I discussed this possibility at the end of the previous post, but I also commented I didn't find that solution as satisfying. But why not?

Semantic clarity

One thing I like about the mutability-as-boxes model is that it seems to makes it easier to think "equationally" about mutability: instead of mutability being an inherent property of vectors (and other data structures), mutability is an 'embellishment' which can be added to any data structure (by putting it into a box), and it seems more or less obvious (to me, anyway) what will be the expected behavior when adding or removing mutability from something (or rather, when adding or removing something from mutability). For example, if vectors were inherently mutable or immutable, then I have to know what operations exist to convert one type into the other, and what happens to the original (if I make an immutable vector out of the mutable one, will the new vector reflect further changes in the original (like a C const reference), or is it an independent, never-changing copy?). Of course, when you learned the programming language you would learn about those details and be done with it, but the boxes model seems to suggest the answers by itself: if I extract a vector (immutable, like all vectors) from a mutable box, I would expect that further changes to the box contents won't affect the vector I just extracted (because changing the box contents means replacing one vector with another, not changing the vector itself); and if I have an (immutable) vector v around and I put it inside a box, I would expect that further changes to the box contents won't affect my original v (for the same reason: if I change the box contents, I'm replacing v with a new vector, so it's not v anymore). In fact, in the previous post I have mostly wondered about how to implement the model efficiently, rather than what the correct behavior of each operation should be, because that part did not seem to raise any questions.

There is a flip side to this: although it is easier (to me, anyway) to think about the semantics of the mutability operations, the optimizations required to make it work well make it harder to think about the performance of the written code. That's the sufficiently smart compiler problem: a sufficiently smart compiler (or runtime) can turn something that would in principle be expensive into something fast, but then you change a small thing in your code in a way that the optimization cannot handle, and suddenly the performance of your program drastically changes. You end up having to know which cases the implementation can optimize, which makes up for the semantic simplicity. Unless you can make sure the optimization will handle all 'reasonable' cases (for varying values of 'reasonable'), this can be a problem.

Equality

Object equality is a more complicated concept than one might expect. There are multiple notions of equality around – some languages have multiple operators for different kinds of equality (for example, == and === in JavaScript, or eq?, eqv? and equal? in Scheme). One type of equality that's given particular prominence in Scheme is the idea of object equivalence, embodied in the eqv? predicate: two objects are equivalent iff no operation (other than the equality predicates themselves) can tell them apart. Mutability is particularly important for object equivalence: two mutable objects (say, two vectors [1 2 3] and [1 2 3]), unless they are one and the same object in memory, are never equivalent, even if they have the same contents, because you can tell them apart by modifying one and seeing if the other changes as well (i.e., they might cease from having the same contents in the future). On the other hand, two immutable objects of the same type and with the same contents are equivalent, because there is in principle no operation that can tell them apart. (Of course the implementation might provide a function to return the address of each object in memory, which would allow us to tell the objects apart. But let's not concern ourselves with that.) Another notion of equality is that of equality of contents, embodied in the equal? predicate: two objects are equal if they are of the same type and have the same contents, even if they are mutable.

When you have a lookup data structure such as a dictionary, you have to decide which kind of equality you will use to compare the keys in the data structure with the key being looked up. Scheme hash table implementations typically require one to specify the equality operator explicitly, because strings are mutable, so you want equal? if your keys are strings, but in other cases you may want to distinguish objects that are not equivalent in the above sense, so you want eqv?.

But if you make strings and vectors immutable, you can compare them with eqv?, and the cases where you want to actually use equal? for hash table lookup mostly go away. And you generally don't want mutable keys in your hash tables anyway (because if you mutate the object that was the original key, typically your hash table stops working because now the key changed but is still hashed under the old key's hash); we tolerate that in Scheme only because strings (and lists, and vectors) are mutable and we want to be able to use them as keys. So if mutability is isolated by boxes, now we can make hash table lookup use object equivalence (eqv?) by default and not worry about explicitly choosing the right predicate for hash table lookup. (Having a sensible default predicate for hash table lookup is important, among other cases, if you want to have literal syntax for hash tables, i.e., if you want to be able to write a literal hash table like {"foo": 1, "bar": 2} in your code without having to say "hey, by the way, the keys are compared by equal? in this case".)

You can still use boxes as keys in a hash table. But since boxes are mutable, a box is only eqv? to the very same object in memory, so you have to use the same box object as the key when you store a value in the table and when you look the value up. This is actually useful if you want to store information about the box itself rather than the contents. But what if I want to look up based on the box contents? Well, then you unbox the contents and look it up! Which expresses intent far better, if you ask me. (This is not entirely equivalent to an equal?-based hash table lookup because you may have boxes inside boxes which would all have to be unboxed to achieve the same effect. Not that this is a very common use case for hash table keys.)

Could we not do the same thing with the mutability flag model? In that model, eqv? would check the mutability flag; objects with the mutability flag on would only be equivalent if they were one and the same, and objects with the mutability flag off would be compared for contents. It would work, but would not be as pretty, if you ask me. However, as long as mutability is easily visible (for example, mutable objects would be printed differently, say like ~[1 2 3] if the mutability flag is on), it could work fine.

Mutable boxes are useful on their own

In Scheme, variables are mutable: you can use (set! var value) on them to change their values. The problem is, variables are not first-class entities in Scheme, so you cannot pass them around directly. So if you want to share mutable state across functions, you have to put the variable in some place where all the interested functions can see it; I remember once having moved subfunctions into another function just so they could all share the same mutable variable. Alternatively, you can create a mutable data structure and pass it around to the relevant functions – and the box is the simplest mutable data structure you can use, if all you want is to share one single mutable cell around. So mutable boxes are useful even if you don't intend to make them the one single source of mutation in the language. And since they are already there, why not just go ahead and do just that? (I am aware that "why not?" is not exactly the most compelling argument out there.)

Another case: cons-cell based lists are somewhat annoying to use with mutation. Suppose you have a list in a variable x, and you pass it around and it ends up in a variable y in another part of the program. If you append things to the tail of x by mutating the tail, both x and y will see the new items, because the tail is reachable from both x and y. But if you append things to the front of x, y won't see the new items in the front, because the new elements are not reachable from the old list tail. If you put the whole mutable list inside a box and passed the box around, both x and y would have the same view of the mutable object. And if you took the list out of the box and put it on another variable z, it would become immutable, so either you see the same changes to the list as everyone else, or you isolate yourself from all subsequent mutations, but it will not happen that you will see some changes to the (tail of the) list and not others (to the front).

Conclusion

I hope I have been able to show why I find the mutability-as-boxes model appealing. I'm not saying it does not have problems (on the contrary, I have already said it has problems), I'm just trying to show what is the point of the whole thing.

_____

1 This is kinda disturbing when you think about it. How many other questions am I not asking?

2 Well, const is not really about the immutability of the data, it's more about the permission to modify a piece of data from a given reference. That is, if a function is declared as taking a const char * argument, that means that the function is not supposed to modify the data pointed to, but it does not mean that the region will not be changed through other references. In other words, it's about requiring something from the user of the reference, but not about providing a guarantee to the user of the reference. A true immutability declaration would both forbid the user from modifying the data and ensure to them that the data will not change during use. Immutability in a language like Rust works like this (except immutable is the default, and mutability is explicitly declared).

Comentários

An approach to mutation

2018-08-01 22:19 -0300. Tags: comp, prog, lisp, pldesign, in-english

I've been around lately with an idea for handling mutation in a new Lisp-like programming language. Most of these ideas are probably not new – in fact, while doing a little research I've found out about Clojure's transients, which embody some of the same ideas – and the parts that are possibly new are not necessary good. But I want to write this down for future reference, so there we go.

DISCLAIMER: This is one of those programming language design posts exploring a bunch of ideas and reaching no conclusion. Read at your own peril.

Some context

The problem with mutation is when the mutable data is shared with other parts of the program – especially when you don't know what parts of the program share the same data. For example, suppose you call a method blog_post.get_tags(), and it returns you a list ["comp", "prog", "lisp"] – can you mutate this list? For example, if I were to sort it, or remove elements from it, can I do it in-place, or I would be mutating a list used internally by the blog_post object and thus inadvertently affecting other parts of the program? Without looking at the method's source code, we don't know. If we wanted to be sure not to break anything, we would have to make a copy of the list and change the copy instead.

What if I am the person writing the get_tags() method? Should I always return a new copy of the list, wasting some memory and cycles but ensuring that whoever calls my function won't be able to affect the internal fields of the blog_post? Or should I always return the same list object, thus avoiding a new allocation but relying on the caller to do the right thing?

Now consider the strings inside that list. If I were to convert them to upper-case, should I do it in-place, or copy them first? In a language like C, this is the same problem as before: you have to know whether get_tags() gives you a copy of the original strings (which you can freely modify), or the internal strings used by blog_post (which you should not modify). But in a language like Java or Python, this problem does not come up: since strings are immutable in those languages, the only way to 'change' them is by making a new string, so modifying them in-place is not an option. On the other hand, the writer of the get_tags() method can now happily return the internal strings of the blog_post object, since they can be sure the strings cannot be modified by external code.

If you make all data structures immutable, you eliminate this problem – and that's the purely-functional approach, taken by languages like Haskell. Clojure is similar in making the core data types (such as lists, vectors and hashmaps) immutable, and having controlled forms of mutability. In traditional Lisps like Scheme and Common Lisp, on the other hand, most composite data types (including lists, vectors and strings) are mutable. The standards of those languages are careful in describing which functions always return freshly allocated data and which return values that may share parts with the function's arguments. This is basically part of the contract of those functions, which you have to know whenever you want to mutate values generated by them. The situation in traditional Lisps is somewhat aggravated by the fact that linked lists may share a tail: two lists (1 3 7 5) and (2 7 5) may actually share the same cons cells for the (7 5) part. In a mostly functional setting, this is okay, but if we want to mutate anything, we have to be extra careful not to be inadvertently changing something else. In this example, sorting the second list in place may end up messing the first list.

What I'm interested in is finding a middle ground between full immutability and full mutability. I want to be able to return immutable data from functions, so I can know the consumers of that data won't inadvertently change it, and I also want to be able to create mutable data which can be modified in place. It would also be nice to be able to use mutable data for temporary processing and make it immutable after we are ready. And I want to be able to tell at a glance if I'm dealing with mutable or immutable data.

So here is the idea…

The idea

First, we make all basic composite data types (lists, vectors, dictionaries, strings, etc.) immutable. Then we add a single mutable box type. Values of the box type have a single mutable field. This idea is at least as old as ML's ref type, so nothing new so far. I will use the notation &val to mean a box containing val, and the expression (set! box val) changes the contents of the box box to val. I will also use @ (read at) for the indexing function, so (@ vec idx) means the idxth element (starting at 0) from vector vec. (@ box) with no indices means to extract the box's contents.

So now we can make a mutable cell with an immutable vector inside, e.g., &[1 2 3]. We cannot mutate the vector directly, but we can replace the whole vector with another immutable vector. That may be elegant and all, but it's not as convenient as a mutable array, nor as efficient. There are some tricks we can play here, though.

The first trick is to make the assignment operator (set!) recognize vector indexing as its first argument, so if v is the vector-containing box &[1 2 3], we can write (set! (@ v 0) 42) to replace the vector [1 2 3] with the vector [42 2 3] inside the box. It looks like we are mutating the vector's first element, but actually we are replacing the whole original immutable vector with a new immutable vector with a different element at position 0.

This gets us convenience, but it's still inefficient: if I write a loop to mutate all elements of the vector, it will generate a fresh new vector on each iteration. But then comes the second trick: how can we tell the difference between a mutable cell with an immutable vector inside from an actual mutable vector? If we make the difference invisible to the programmer, then we can mutate the vector in-place as an optimization. So (set! (@ v 0) 42) syntactically looks like mutating a vector element, semantically means replacing the whole vector with a new one, but implementationally actually works by mutating the vector element anyway. I'm not sure about the wisdom of this double layer of self-cancelling illusions, but let's explore this idea further.

Let's call the naive implementation using a mutable box with an immutable vector inside, well, the naive implementation. And let's call the implementation which underlyingly uses a mutable vector to represent the box+vector combination the smart implementation (with the full understanding that it may actually be too smart for its own good, or maybe not smart enough to make this idea work well).

The most basic operation you can do with a box is extracting the contents. In the naive implementation, that's just returning the value inside. In the smart implementation, we must simulate this by copying the current contents of the mutable vector into a freshly allocated immutable vector and returning that. A user can then observe the difference between the two implementations by taking the contents of the same box twice and checking whether the results are eq? to each other, i.e., whether they are the same object in memory.

It seems to me that the solution to this problem is to ditch object identity for immutable objects from the language, i.e., get rid of the lower-level eq? operation (Python's is), or at least relegate it to a library of lower-level operations. Immutable objects should only be compared by its contents, not by identity: if I compare [1 2 3] with [1 2 3], it should not matter whether they are separate objects in memory or not: they have the same contents and that's what matters. The only way to tell two distinct objects with the same contents from each other (apart from eq?) is by mutating one of them and seeing if the other changes as well; but if the objects are immutable, this distinction disappears.

A possible optimization to reduce the amount of copying when extracting a box's contents is to return the actual underlying vector, but mark the box as copy-on-write, i.e., we postpone the copy to the next time we need to mutate the vector inside the box. If the box is not mutated afterwards, the vector is never copied. The problem with this may be performance: we need to check the copy-on-write flag before every change to the vector, and the whole point of these optimizations is performance. Sure, we avoid a copy, but we slow down every write to the vector. This is aggravated by the fact that this flag must be synchronized across threads, lest we end up with two threads making new copies and clobbering each others' view of the box.

Speaking of which, doesn't thread synchronization throw this whole idea out of the window anyway? Extracting the contents of a box must be an atomic operation, but someone might be mutating the underlying vector while we are copying it into a new immutable vector to return it. This is okay as long as we can guarantee that the resulting copy represents one possible atomic state of the box at the time of the extraction, but consider the following scenario:

To avoid this problem, the implementation would have to acquire a lock (or use some other form of thread synchronization) when extracting the contents of a box, thus slowing down what should be a cheap operation. The copy-on-write solution avoids the copy incoherence problem, because the copying happens from the now-immutable vector to the new mutable one, and not the opposite, so we know that the origin will not change mid-copy. But as we have seen, we need to ensure synchronization of the copy-on-write flag, so it's pretty much the same.

Maybe this synchronization requirement is a good thing: maybe we want copies to be atomic anyway, and this way the semantics of the language guarantees that. But maybe this is an unnecessary overhead most of the time.

Even if we go with the copy-on-extract solution, we can avoid copying in the case of extracting the object from the box and then discarding the box (e.g., if we want to create a mutable vector, do a bunch of mutating operations on it, and then make it immutable) by providing an (unbox! b) operation which returns the contents and sets the box contents to nil (or whatever other value to indicate that the box is "empty"). Because we know the vector will not be mutated again, we can just return the underlying vector and call it immutable. This is basically what Clojure's persistent! operation does (though I didn't know that when I had this idea).

Let's consider some other problems and optimizations.

Putting it back in the box

What about sequences of transformations? For example, suppose I implement a filter function, which takes a predicate function and a vector and returns a new vector containing only the elements for which the predicate function returns true, like this:

(def (filter pred vec)
  (let ([result &[]])
    ;; Collect satisfying items in the mutable result vector...
    (for [item in vec]
      (when (pred item)
        (push! item result)))
    ;; And then return the contents as an immutable vector.
    (unbox! result)))

What if I want to, say, filter a vector and then reverse it? If filter is written like this, I get an immutable vector back, so I would have to copy it into a mutable vector again just so I can reverse it. If only filter had not called unbox! at the end, I could have reversed it in-place without a new allocation! But if I don't unbox!, I will have to always manually unbox the result when I want to, and most of the time I do want an immutable result.

There is a possible trick to help us here: if we unbox a value just to immediately box it back again, we can actually keep using the same underlying storage with no copying. The problems with this optimization are: (1) We must be able to know that no other references to the object have been created between the unboxing and the re-boxing, and basically the only way to do this is with some sort of reference counting. Reference counting has its share of problems (cyclic data structures never reach count zero, we need to synchronize count updates across threads), so relying on an optimization which requires us to use reference counting is not good. (2) We need to make sure the reference count does not inadvertently rise above 1, which would preclude the optimization. Since there may be more going on under the scenes in the compiler/interpreter than reaches the eye, this would be an unreliable optimization, that sometimes does not trigger for non-obvious reasons.

An alternative to use reference counting would be to do this analysis at compile-time, either through some form of escape analysis (which is hard to do across functions), or some crazy type system with uniqueness types (like Rust's borrow checker), which does not mesh well with my goal of a dynamically typed language.

Nested data structures and sharing

What if I put nested data inside a box, like &[[1 2] [3 4]]? Should the sub-vectors become underlyingly mutable too? If so, when should we stop recursing through the structure, which could contain other composite data types in it? Should we do it lazily, using copy-on-write as we mutate the inner vectors? The implementation of this can get tricky. If I access (@ b 0), should I get an immutable [1 2], or should I get a mutable &[1 2] which shares the same memory as the original element, so that mutations on the &[1 2] vector are reflected on the &[[1 2] [3 4]] one?

I'm too tired to even analyse the possibilities right now.

Refactoring code to change data mutability

Suppose I have a data structure with a bunch of immutable fields, say, (Person name age) and I decide I want to make the age field mutable. In our conceptual framework, this means wrapping the value of the field in a mutable box, e.g., (Person "Hildur" &22), i.e., the field itself remains immutable, but its value is a mutable cell. That looks nice, and makes all mutability readily visible, but it also means that we have to change all code using the age field to extract the value from the box, even code that does not mutate the value.

Maybe this is a good thing: if the code was written under the assumption that the value does not change, maybe it is good that we have to revise everything when we turn it mutable. On the other hand, this makes it harder to try things out in code and run it to see what happens, and for I long time I have defended the ability to run incomplete (and even wrong) programs while prototyping. However, I also want to be able to run optional static checks, and it's easier to do so when code is explicit about its intentions. So there we go.

An alternative: mutability as a flag

An alternative to the mutability-as-boxes approach is just to make all composite data structures carry a 'mutable' flag. We can still use the notation &[1 2 3] to mean a vector with the mutable flag on. We can provide an operation like Clojure's persistent! which turns the mutability flag of an object off. An operation to turn it back on would be more dangerous, but might be useful for debugging purposes (though it's the kind of thing you can be certain will be abused if added to the language (though Lisps have traditionally always preferred to empower the user rather than impose decisions on them)).

In this scenario, the semantics of (set! (@ v 0) 42) is to actually modify the vector in-place, so we don't need the double illusion trick. If we want to return an immutable version of a mutable data structure without losing the mutability, we have to explicitly copy it. Perhaps more descriptive of intention, we may have a non-destructive persistent operation which returns an immutable version of a value, which may or may not actually involve a copy (it may actually use copy-on-write behind the scenes). Thread synchronization has to be done explicitly, otherwise you assume the risks of getting a partially-modified copy. This is somewhat unsatisfying, but inconsistencies across threads could happen with boxes anyway whenever you had to work with more than one box, so a better solution to synchronization is needed anyway.

(In)conclusion

This idea of using mutable boxes + immutable data structures + optimization tricks had been haunting me for a week and seemed really appealing at first, but thinking more deeply about it, it does have its share of problems. Maybe it's a cool idea anyway, maybe not. I have to think more about it. Said research will require more and abundant funding.

1 comentário

On Scheme's minimalism

2017-09-14 19:34 -0300. Tags: comp, prog, lisp, scheme, pldesign, ramble, in-english

[This post started as a toot, but grew slightly larger than 500 characters.]

I just realized something about Scheme.

There are dozens, maybe hundreds, of Scheme implementations out there. It's probably one of the languages with the largest number of implementations. People write Schemes for fun, and/or to learn more about language implementations, or whatever. The thing is, if Scheme did not exist, those people would probably still be writing small Lisps, they would just not be Scheme. The fact that Scheme is so minimal means that the jump from implementing an ad-hoc small Lisp to implementing Scheme is not that much (continuations notwithstanding). So even though Scheme is so minimal that almost everything beyond the basics is different in each implementation, if there were not Scheme, those Lisps would probably still exist and not have even that core in common. From this perspective, Scheme's minimalism is its strength, and possibly one of the reasons it's still relevant today and not some forgotten Lisp dialect from the 1970s. It's also maybe one of the reasons R6RS, which departed from the minimalist philosophy, was so contentious.

Plus, that core is pretty powerful and well-designed. It spares each Lisp implementor from part of the work of designing a new language, by providing a solid basis (lexical scoping, proper closures, hygienic macros, etc.) from which to grow a Lisp. I'm not one hundred percent sold on the idea of first class continuations and multiple values as part of this core*, and I'm not arguing that every new Lisp created should be based on Scheme, but even if you are going to depart from that core, the core itself is a good starting point to depart from.

[* Though much of the async/coroutine stuff that is appearing in modern languages can be implemented on the top of continuations, so maybe their placement in that core is warranted.]

1 comentário

Lisp without cons cells

2016-05-28 13:14 -0300. Tags: comp, prog, pldesign, lisp, ramble, in-english

Okay, I'm gonna write this down now to distract myself for a while before I get back to Master's stuff.

In a recent post I talked about the problem of cross-process garbage collection, and suggested wrapping objects in a reference-counted container when crossing process boundaries as a possible solution, but I remarked that this would have a large overhead when passing many small objects. The prime example would be passing a linked list, as (at least naively) every node of the list would get wrapped as the elements of the list are accessed.

Now, I particularly cared about this case because the linked list (based on cons cells) is a very prominent data structure in Lisp. And although they have some nice properties (they are conceptually simple, you can insert and remove elements into the middle/end of a list by mutating the cdrs), they also are not exactly the most efficient data structure in the world: half the memory they use is just for storing the "next" pointer (which fills processor cache), whereas in a vector you just need a header of constant size (indicating the vector size and other metadata) and the rest of the memory used is all payload. Also, vectors have better locality. On the other hand, "consing" (i.e., nondestructively inserting) an element into a vector is O(n), because you have to copy the whole vector, and even destructive insertion may require a whole copy every once in a while (when you exceed the current capacity of the vector). I've been wondering for a long time: could you make a Lisp based on a data structure that is halfway between a linked list and a vector?

If we are to allow the common Lisp idioms with this new kind of list, it has to support consing and taking the tail of the list efficiently. (Another possibility is to replace the common idioms with something else. That is much more open-ended and requires more thought.)

What I've been thinking of as of late is roughly a linked list of vectors, with some bells and whistles; each vector would be a chunk of the list. Each vector/chunk would have a header containing: (1) the number of elements in the chunk; (2) a link to the next chunk; (3) an index into the next chunk. Then comes the payload. So, for example, if you have the list (w x y z), and you want to append the list (a b c) on the front of it, you'd get a structure like this (the | separates graphically the header from the payload; it does not represent anything in memory):

[3 * 0 | a b c]
   |
   `->[4 * 0 | w x y z]
         |
         `-> ø

The reason for the index is that now you can return the tail of a list lst without the first n elements by returning a vector chunk with 0 length and a pointer into lst with index n: [0 lst n | ]. If the n is greater than the size of the first chunk (e.g., if you want to drop 5 elements from the (a b c w x y z) list above), we must follow the "next" pointers until we find the chunk where the desired tail begins. This is likely to be more efficient than the cons cell case, because instead of following n "next" pointers, you follow the number of chunks, subtracting the length of the skipped chunk from n each time. In the worst case, where there is one chunk for each element, the performance is the same as for cons cells, at least in number of pointers traversals. (We must only allow empty chunks, like the [0 lst n | ] example, at the beginning of a list, never in the middle of a chunk sequence. This ensures worst-case cons-like behavior. If we allowed empty chunks anywhere, reaching the nth element of a list could require arbitrarily many chunk traversals.)

One problem with this is that now (cdr lst) allocates memory (it creates a [0 lst 1 | ] chunk and returns it), unlike the cons cell case, where cdr never allocates memory (it just returns the value of the cell's "next" pointer). One possible solution is to try to make the result of cdr go in the stack rather than being heap-allocated, to reduce the overhead (the compiler could special-case cdr somehow to make it return multiple values rather than a new chunk, and build the chunk on the fly in the caller if it turns out to be necessary.) Another way around this would be to return a pointer into the middle of a chunk instead of a new chunk. I see two ways of achieving this:

All these have drawbacks. First, you need to know that the pointer you have is a pointer to a cons cell to be able to safely do the pointer arithmetic. (The fixed-size chunks case is simpler to solve: you zero out the pointer and see if it points to a chunk type tag.) Also, pointers into the middle of objects complicate garbage collection (and even more reference counting, I think). Finally, if you fix the size of chunks some of the advantages of using chunks in first place go away; if I allocate a 1000-element list at once, that should get me a single 1000-element chunk.

Or should it? Another problem here is that now garbage collection / reference counting can only collect whole chunks. If you choose your chunks badly, you may end up holding memory for longer than necessary. For instance, if you have a 1000-element list and at some point your program takes tails until it only remains with a reference to the last three elements, and the list was made out of a single 1000-element chunk, now you're stuck with a huge chunk most of which is unused – and more, all the elements in it are held from being collected too. Maybe we'd need a heuristic: if the tail size you want is less than some threshold size of the chunk, the system would return a copy of the tail rather than the tail. This would mess with mutability (you'd never know if the tail list you got shares storage with the original), but maybe immutable lists are the way to go anyway.

The other problem to solve is how to make cons efficient: the classical Lisp cons adds (non-destructively) one element to the front of an existing list, and we don't want to create a new chunk per cons invocation, otherwise the chunks just degenerate into cons cells. One idea I had is to allocate chunks with a least a certain amount of elements. For example, if you create a list with just a, you'd get a chunk with a few blank spaces (and enough metadata to know what is blank and what isn't; this could be an extra header element, or just a distinguished value meaning "blank"): [4 ø 0 | _ _ _ a]. Now, when you cons a new element x into that list, cons would check if there is a space immediately before the a in the existing chunk, and mutate it in place: [4 ø 0 | _ _ x a]. This won't mess with the program's view of the list because so far it only had references to the already filled part of the list. The problem with this is if you have multiple threads wanting to cons onto the same list at the same time: we must ensure only one of them gets to mutate the chunk. For example, say one thread want to cons x onto the list (a), and another thread wants to cons y onto the same list (a). We must make sure that only one gets to mutate the chunk in place ([4 ø 0 | _ _ x a]), and the other one will fail and fall back to either by copying the chunk and then mutating the copy, or by creating a new chunk that points to the old one ([4 [4 ø _ _ x a] 3 | _ _ _ y]; note that outer chunk points into the inner chunk with an index 3, skipping the first 3 elements, including the x added by the other thread). This could have a synchronization overhead. I'm not sure if it would be significant, though, because all you need is a compare-and-swap: "try to write into this space if it is blank". You don't need a lock because you don't need to wait anyone: if this first try fails (i.e., if the other thread got the space first), the space won't be available anymore, so you must immediately fall back to creating a new chunk rather than waiting for anything.

A possible side-effect of all of this is that now vectors as a separate data structure may not be necessary: you just allocate an n-element list at once, and it will largely have the same performance as an n-element vector. Well, unless we make lists immutable, then we may need (mutable) vectors. And lists still have some arithmetic overhead to find the position of the element (because in general we don't know that the list is a single chunk when performing an access, we have to find that out), so vectors may still be advantageous in many circumstances.

Now, back to (trying to) work.

[Update: Apparently I reinvented a half-hearted version of VLists. Also, I didn't mention that, but the Lisp Machine had a feature similar in spirit (but not in implementation) called CDR coding, which used a special tag in cons cells to mean that the rest of the list itself rather than a pointer to it was stored at the cdr place, thus saving one pointer and gaining locality. In the Lisp Machine, every memory object was tagged, so this special tag came more or less for free, which is generally not the case for modern architectures.]

Comentários

C is in an identity crisis, and some thoughts on undefined behavior

2016-05-19 23:11 -0300. Tags: comp, prog, c, pldesign, ramble, in-english

So, stories about undefined behavior have been making rounds again in my Twitter and RSS feeds (two things I was supposed not to be using, but anyway), which brought me some new thoughts and some other thoughts I meant to blog about ages ago but forgot about them.

The most recent one was this comment on Hacker News (via @pcwalton, via @jamesiry), which presents the following code, which is supposed to take a circular linked list, take note of the head of the list, and walk around the list freeing each node until it finds the node that points back to the head (and thus the end of the list):

void free_circularly_linked_list(struct node *head) {
  struct node *tmp = head;
  do {
    struct node *next = tmp->next;
    free(tmp);
    tmp = next;
  } while (tmp != head);
}

This looks (to me) as good C code as it gets. However, this code triggers undefined behavior: after the first iteration of the loop frees the node pointed to by head, it is undefined behavior to perform the tmp != head comparison, even though head is not dereferenced.

I don't know what is the rationale behind this. Maybe that would make it possible to run C in a garbage-collected environment where as soon as an object is freed, all references to it are zeroed out. (The fact that no one has ever done this (as far as I know) is mere detail. The fact that in a garbage-collected environment free would likely be a no-op is a mere detail too.)

The feeling I had after I read this is that C is in a kind of identity crisis: C allows you to do all sorts of unsafe operations because (I'd assume) it's supposed to let you do the kind of bit-bashing you often want to do in low-level code; at the same time, modern standards forbid that very bit-bashing. What is the point of programming in C anymore?

[Addendum: To be more clear, what is the purported goal of the C language? The feeling I have is that it has moved from its original function as a "higher-level assembly" that is good for systems programming, and is trying to serve a wider audience more preoccupied with performance, but in doing so it is not serving either audience very well.]

And the standards forbid these operations in the worst possible way: by claiming that the behavior is undefined, i.e., claiming that compilers are free to do whatever the hell they please with code perfoming such operations. Compilers keep becoming better and better at exploiting this sort of undefinedness to better "optimize" code (for speed, anyway). Meanwhile, they keep breaking existing code, and opening new and shiny security vulnerabilities in programs. The NSA probably loves this.

At this point, I'm beginning to think that C does not serve its purpose well anymore. The problem is that there seems to be no real alternative available. Maybe Rust can be it, although I don't really know how well Rust does in the bit-twiddling camp (e.g., can you easily perform bitwise and/or with a pointer in Rust? Well, come to think of it, even C does not allow that; you have to cast to an integer first.)

* * *

The other undefined behavior I've been reading about lately is signed overflow. In C, signed overflow is undefined, which means that code like:

if (value + increment < value) {
    printf("Overflow occurred! Aborting!\n");
    exit(1);
}
else {
    printf("No overflow; proceeding normally\n");
    value += increment;
}

is broken, because the compiler is likely to optimize the overflow check and the then branch away and just leave the else branch. I have seen two rationales given for that:

Pointer arithmetic. In the good old times, an int and a pointer used to have the same size. People happily used ints as array indices. Array indexing is just pointer arithmetic, and in some architectures (like x86), you can often perform the pointer arithmetic plus load in a single instruction.

Then came 64-bit architectures. For reasons I don't really get (compatibility?), on x86-64 and other 64-bit architectures ints remained 32-bit even though pointers became 64-bit. The problem now is that transformations that assumed integers and pointers to be the same size don't work anymore, because now their point of overflow is different. For example, suppose you had code like:

void walk_string(char *s) {
    for (int i=0; s[i]; i++) {
        do_something(s[i]);
    }
}

Usually, the compiler would be able to replace this with:

void walk_string(char *s) {
    for (; *s; s++) {
        do_something(*s);
    }
}

which is potentially more efficient. If ints and pointers have the same size, then this transformation is okay regardless of overflow, because the int would only overflow at the same point the pointer would anyway. Now, if ints are supposed to wrap at 32-bits but pointers wrap at 64-bits, then this transformation is not valid anymore, because the pointer version does not preserve the overflow behavior of the original. By making signed overflow undefined, the problem is sidestepped entirely, because now at the point of overflow the compiler is free to do whatever the hell it pleases, so the fact that the overflow behavior of the original is not preserved does not matter.

Now, there is a number of things wrong in this scenario:

Optimizations based on "real math". The other reason I am aware of for making signed overflow undefined is to enable optimizations based on the mathematical properties of actual mathematical integers. An example is assuming that x+1 > x, for instance (which is what breaks the overflow test mentioned before). Another example is assuming that in a loop like:

for (i=0; i<=limit; i++) { ... }

the halting condition i<=limit will eventually be true, and therefore the loop will finish; if i were defined to overflow, then this loop would be infinite when limit == INT_MAX. Knowing that a loop terminates enables some optimizations. The linked article mentions enabling use of loop-specific instructions which assume termination in some architectures. Another advantage of knowing that a loop terminates is enabling moving code around, because non-termination is an externally-visible effect and you may not be able to move code across the boundaries of an externally-visible event [please clarify]. Now, something that occurred to me back when I read that post is that it assumes a dichotomy between either treating overflow as undefined, or defining it to wrap around. But there are other possibilities not explored here. I don't necessarily claim that they are feasible or better, but it's interesting to think on what optimizations they would enable or preclude. For instance:

Now, the problem with the trapping semantics is that you have to check for overflow on every operation. This could be costly, but there are people working on making it more efficient. Seriously, this is the kind of thing that would be trivial if only architectures would help a little bit. Having the processor trap on overflow (either by having special trapping arithmetic instructions, or by having a special mode/flag which would enable trapping) would make this essentially costless, I think. Another nice-to-have would be a set of arithmetic instructions which treated the lower bits of words specially as flags, and trapped when the flags were not, say, all zeros. This could drastically reduce the cost of having fixnums and bignums in the language; the instructions would trap on non-fixnums and invoke a handler to perform the bignum arithmetic (or even find out that the operands are not numbers at all, and signal a dynamic type error), and perform the fast integer arithmetic when the operands had the fixnum flag. Alas, unfortunately we cannot just invent our own instructions, as we typically want to be able to use our programming languages on existing platforms. We could try to lobby Intel/AMD, though (as if).

(At this point I'm not really thinking about semantics for C anymore, just about the possible semantics for integers in a new programming language. Even if, say, Clang incorporated an efficient mechanism for trapping integer overflow, the standard would still say that signed overflow is undefined, so I'm not sure there is much hope for C in this situation.)

2 comentários

Random project ideas

2016-05-13 00:00 -0300. Tags: comp, prog, pldesign, ramble, in-english

A few days ago someone asked me what kind of projects I would like to work on. I have some ideas that have been wandering my mind for a long time, so I decided to write some of them down. (I'm sure someone will come around, steal some of them, and make millions of dollars, but there we go.) I have been meaning to write a post like this one for quite a while, but never got around it. For the next days I will be working on finishing my Master's monograph, so I decided write them down now. The text is kind of a mess, and especially towards the end it is more of a "thinking out loud" sort than a clear exposition of ideas, but it's probably a good idea (for me, anyway) to register these ideas at once before I forget them. As Zhuangzi would say, "Let me say a few careless words to you and you listen carelessly, all right?"

A programming language

Although there are plenty of nice programming languages around (and plenty of un-nice ones as well), I don't know any language I'm fully satisfied with, and I certainly would like to try creating one. I don't think there can be such a thing as one perfect language to rule them all, but I'd like to create one that is better suited to the way I like to do things and to the kinds of things I like to do. Some features of that hypothetical language would be:

Optional types

After writing some larger programs in Scheme, I realized that some static type analysis would go a long way in catching many programming mistakes. Moreover, while working (for a short time) with the Clang/LLVM codebase, I realized how types can be useful in finding other places in a program which need to be changed after a change in one part. However, I strongly believe in the ability to run incomplete (and incorrect) programs, both for the sake of debugging (it is often convenient to be able to run an incorrect program with some concrete data to see what is the state at the point of the error, rather than relying solely on static type errors), and for the sake of exploratory programming (when you want to try out stuff and figure out what program do you want to write after all).

(Some static-types people, when confronted with this notion, give replies like: "but static types help me with exploratory programming!". Well, dynamic typing helps me with exploratory programming. As I said before, I don't believe in a single programming language to rule them all, and neither do I believe that either static typing or dynamic typing is inherently superior in all circumstances and for all people. Indeed, I would actually like not to have to choose one or another, and that's kind of the point of this section.)

So, I would like to alternate between dynamic and static types according to convenience. First of all, I don't think static type mismatches should preclude compilation/execution; rather, these should emit warnings, and the compiler should emit code that runs as far as possible before hitting the error.

In dynamically-typed languages, it is relatively easy to call functions with arguments of the wrong type and keep running, because all values carry enough metadata (tags of some kind) to enable checking the type at any given moment. Statically-typed languages, on the other hand, usually employ "unboxed" representations for data (e.g., a 32-bit integer in memory is just 32 bits, with no extra tag indicating that it is an integer; a pointer is just another 32 (or 64) bits in memory, with no way of distinguishing it from an integer of the same size), therefore calling a function with a value of the wrong type (e.g., passing an integer to a function expecting a pointer) must be blocked; otherwise, there would be a violation of safety (e.g., the function would try to use the integer as a pointer, probably with disastrous consequences). Conventional statically-typed languages block execution of such unsafe function calls by refusing to compile the program, but that's not necessary: one might instead emit code that evaluates the program until the point where an unsafe function call would be performed, and abort execution just then. For instance, if f is a function taking integers, and g(x) returns a string, and the program contains the expression f(g(x)), you can still emit code that evaluates g(x) (and f), and then interrupts execution instead of calling f with the string. (I've written about this before, but anyway.)

So, you could have a language where (1) type declarations are optional; (2) if function parameter/return types are not declared, dynamic types and boxed representations are used by default; (3) if types are declared, static type checking is performed on function calls, and where the compiler cannot prove that the types match (e.g., because they don't match, or because a function with statically-typed parameters is called with a dynamically-typed value), it emits code to interrupt execution (and provide a decent diagnostic message, and/or trigger the debugger) just before the function is called. I wonder if interrupting execution at the function call wouldn't be too early (as ideally I'd like to be able run an incomplete/incorrect program as far as is reasonable). Perhaps there could be a compiler option/pragma to emit code for the entire module as if everything were dynamically typed, even when type declarations are provided (but still emit warnings upon detected type mismatches).

This mix of static and dynamic types brings some implementation problems. Because statically-typed functions can't just be called with any random argument due to representation mismatches, if a function expects another function as an argument (let's say, a function f expects a function taking an integer and returning an integer), you cannot pass it a dynamically-typed function g that happens to return integers when given integers, because the representation of the values it returns would be different (g returns boxed integers, but f expects a function that returns raw integers). Likewise, if a function f expects a dynamically-typed function, it cannot be given a function g that returns raw integers. One solution would be to generate a dynamically-typed wrapper around statically-typed functions when they are used in a context that expects a dynamic function, and vice-versa.

To research: It'd probably be good to see how Haskell deals with the situation where an Int -> Int function is used where an a -> a one is expected. Although now that I wrote this I realize that that's not the same problem, because in Haskell the concrete a is always statically known at the point where the integer value is used. Probably more promising would be to look at how Java and C# deal with this. (In Java, the difference between a raw integer and an Integer object is directly visible to the programmer, whereas I think that's not the case in C#.)

There has also been lots of work on the subject of gradual types, both recent and old, and I have to check it out.

Language interoperability

I'm probably beginning to sound repetitive, but as I said before, I don't believe in a single language for all circumstances, and not only I want to be able to use other existing programming languages, I'm probably going to create more than one in my lifetime. Still, I don't like the idea of having to reimplement code already available in a library just because the library is not available in the language I want to use. I'd like to be able to easily integrate pieces of code written in different programming languages, without having to manually write wrappers/bindings/etc. Ideally, there would be a "universal" way of exchanging data / invoking procedures across programming languages, and the "bridging" work would only have to be done once for each programming language (to make it support the "universal" way), rather than for each library and each language you want to use the library in.

This already kinda exists: the JVM and CLR do this – for languages that are implemented on the top of the JVM or CLR. The problem is that you are constrained to implement your language on the top of these virtual machines, which might not be a good match for the programming language in question (e.g., the VM may not support tail call optimization, or multiple inheritance, or you want to use the stack in a really strange way, or you want your strings to be UTF-8 instead of UTF-16, or you want direct access to operating system features, or whatever), or you may simply not want to use a bulky VM, or one whose evolution is in the hands of a single company and cramped by API copyright claims and/or patents.

An idea that occurred to me in the past is that of an extensible virtual machine: a machine where you could load modules that implemented new opcodes/features that would make it better suited to various programming languages. It is possible to do this without opcode conflicts between the different extensions by giving opcodes (fully-qualified) names instead of fixed numbers; the mapping from concrete opcode numbers to opcode names would be given in a header in the bytecode file, and would not be fixed. (I'm pretty sure I got this idea from WebAssembly, but I haven't been able to find a source right now.) So, for instance, you could have a Python extension for the VM, with Python-specific features and opcodes, and a Scheme extension, and so on. Of course, making all extensions work together is not just a matter of avoiding opcode conflicts, but having the freedom to extend the VM with features more convienient to each language to be supported sounds more promising than being forced to work with a fixed instruction set which did not have the language of interest in mind.

VMs are not the only possibility (although VMs have a number of other possible advantages, which I may write about in another post). Another way would be to let each language implementation be completely independent, but make it possible somehow for them to share data and call code from each other. This has the advantage of (potentially) making it easier to accomodate existing language implementations (rather than having to reimplement them as VM extensions), but of course brings with it a number of challenges to make interoperability work, such as:

Another possibility, which got into my mind after reading this, would be a model in which no memory is shared between languages: everything is passed by copy, and things don't even have to be in the same process. (This brings up another problem that has been in my mind since forever: the fact that Unix streams, files, etc., are limited to bytes rather than structured data. Here we would be defininig a mechanism for exchanging higher-level data across processes.) This solves the data exchange problem, but not the code invocation problem. This could be done by message passing, but I wonder how that would work. For instance, suppose language A wants to use a regex library from language B.

To research: There is an infinite number of things to research here. One that I realized recently is GObject, GNOME/Gtk's way of doing objects in C which was created with the goal of easily supporting bindings to other languages in mind. Maybe GObject solves all the problems and there's nothing to do (other than making everything support GObject for exchange of arbitrary objects), or maybe GObject can be used as an inspiration. I should also look up other inter-process communication mechanisms (especially D-Bus, and Erlang/OTP). Other people have been working on this problem from other angles.

EOF

Sorry if this post looks like a mess of random ideas thrown around, because that's exactly what it is. As always, writing about stuff helps me organize my ideas about them and realize problems I hadn't thought about before (though how organized things are here is up to question). Feel free to comment.

Comentários

Lisp stares at PHP

2015-09-02 22:11 -0300. Tags: comp, prog, pldesign, lisp, php, lows, in-english

[This post is also available in English.]

No último post, eu falei sobre o lows, um Lisp que compila para PHP 5.2 que eu comecei a desenvolver, e o que eu fiz até agora nele. Neste post, eu pretendo discutir algumas questões de design da linguagem, mais ou menos na mesma idéia da série Blueprints for a shell (só que bem menor (or so I hope)). Como já mencionei, não sei quando vou mexer no lows de novo (afinal, eu tenho certos afazeres mundanos, tais como um mestrado para terminar), mas deixo aqui documentadas algumas idéias. Comentários são sempre bem-vindos.

Tipos de dados

Os tipos de dados do PHP não casam muito bem (leia-se: não casam praticamente nada) com os tipos convencionais do mundo Lisp. O plano é tentar projetar a linguagem para permitir um estilo de programação suficientemente Lisp-like com os tipos existentes, ao invés de tentar recriar os tipos líspicos tradicionais, até porque isso não seria muito viável em termos de performance em PHP.

Arrays

Basicamente o único tipo de coleção que o PHP possui é o array, que faz tanto o papel de lista quanto de dicionário (e mesmo como dicionário, ele preserva a ordem das inserções). Eu não vejo nenhuma maneira eficiente de implementar um cons em PHP (adiciona um elemento a uma lista, criando uma lista nova, em espaço/tempo O(1)). Daria para criar uma classe Cons, mas o overhead seria muito grande, e além do mais o objetivo da linguagem é interagir facilmente com código PHP. So, arrays.

As operações map, filter, reduce e afins são independentes da representação da coleção e funcionam igualmente bem com arrays. O idioma de criar listas usando cons e recursão, por outro lado, cai por terra, mas ao mesmo tempo ele já não seria uma boa escolha em PHP porque chamadas de função são mais custosas, e não há tail call optimization. Na verdade tail call optimization não nos ajudaria nesse caso anyway, porque a recursão não fica in tail position; nessa situação, quando a performance é relevante, a galera costuma acumular os elementos da lista como um argumento da chamada recursiva e chamar reverse! no final do loop, mas se é para fazer isso, podemos passar um array como argumento e adicionar elementos com push. Anyway.

No futuro, map e afins podem ser inlined, o que torna essas construções basicamente equivalentes a loops. Até mesmo a closure passada como argumento para o map não precisa ser construída, e ao invés disso o corpo do lambda pode ser inserido diretamente no loop.

Há que se pensar na sintaxe equivalente ao Array(...) do PHP. Para listas simples, (array 1 2 3) é suficiente. O problema é o equivalente da forma Array("foo"=>1, "bar"=>2). Quando a chave é uma string literal simples, uma possibilidade seria usar keywords do Chicken: (array foo: 1 bar: 2). Quando a chave não é literal, algo como (array (=> key1 val1) (=> key2 val2)) poderia ser usado, mas isso é meio verboso. I don't know.

Uma bizarrice de arrays do PHP é que elas são passadas por cópia. Ao que parece, simplesmente não existe uma maneira de passar um array "by sharing", como os valores costumam ser passados em Lisp ou em Python. É possível criar uma referência (que na verdade é um alias) para uma variável que contém um array, mas a referência é à variável, não ao array; se outro valor é atribuído à variável, a referência/alias reflete a modificação. Acho que teremos que conviver com isso.

Símbolos

Não existe nada equivalente a símbolos em PHP. A principal vantagem de símbolos sobre strings é o fato de que eles podem ser comparados em tempo constante, mas o PHP não possui nada realmente equivalente ao operador eq dos Lisps, então acho que não teria muito propósito implementar símbolos em lows. 'foo poderia ser usado como abreviação para "foo", talvez.

Pharen tem um "operador" #foo, que gera a string correspondente ao nome da função foo. A razão desse operador é que Pharen converte hífens em nomes de função para underlines (o que é uma boa idéia em um Lisp→PHP), e o operador # produz uma string com o nome convertido. Talvez ' pudesse ser usado com o mesmo propósito em lows.

By the way, lows não tem um operador quote de verdade, pois o código de um programa lows não é representado por uma estrutura de dados de lows (i.e., do PHP). Yeah, lows não é uma linguagem homoicônica, e talvez nem merecesse o título de Lisp por conta disso. Macros em lows não seriam escritas em lows, mas sim em (Chicken) Scheme, mas veremos isso mais adiante.

(Qual seria então o significado de '(+ 1 2) em lows? Erro de sintaxe? Uma abreviação de (array '+ '1 '2)?)

Miscelânea

Não há muito o que dizer sobre os demais tipos (números, strings, booleanos, e NULL). PHP converte loucamente entre tipos, o que é meio desagradável, mas não há muito o que fazer, a menos que queiramos inserir checks antes do uso de qualquer operação. (Isso poderia ser uma opção de compilação, útil para debugging, talvez.)

Operadores

Operações aritméticas e afins em Lisp costumam ser funções, e não operadores especiais. Porém, não queremos ter que chamar uma função em lows para realizar operações aritméticas, pois o custo de uma chamada de função em PHP é alto. (Mesmo Lisps costumam otimizar essas chamadas quando é possível determinar os tipos dos argumentos em tempo de compilação.) Por outro lado, seria bom podermos passar + e afins como argumento para outras funções. Uma possível solução seria expandir '+ para (lambda (x y) (+ x y)), mas isso fixa o número de argumentos do operador, enquanto o ideal é que (+ 1 2 3) seja possível e seja traduzido para 1+2+3. Outra possibilidade seria ter tanto uma função, definida na biblioteca padrão, quanto um operador especial, e o compilador decidiria qual usar dependendo do caso. Mas não tem por que incluir uma função para cada operador na biblioteca padrão, se o próprio compilador pode gerar o lambda com o código apropriado automaticamente quando necessário.

A maioria dos operadores do lows seriam equivalentes diretos, e teriam o mesmo nome, dos operadores do PHP. As exceções seriam:

Os operadores binários têm um detalhe extra: x || y em PHP sempre retorna um booleano, enquanto em Lisp (or x y) costuma retornar x se ele é verdadeiro, e y caso contrário. Daria para traduzir (or x y) para x? x : y (com o detalhe de que x não pode ser avaliado duas vezes), mas isso meio que estraga a idéia de tradução direta (os condicionais de ifs teriam uma cara bizarra no código resultante). Talvez o compilador pudesse detectar quando o and/or está sendo usado como condicional de um if e traduzi-lo para &&/|| nesses casos.

De qualquer forma, o que o PHP acha que é verdadeiro é diferente do que um Lisp normalmente acha que é verdadeiro: em Lisp, normalmente o valor booleano falso (ou a lista vazia) é considerada como falso, e tudo o mais é considerado verdadeiro (inclusive 0, a string vazia, etc.). Esse comportamento inclusive faria mais sentido em PHP, que possui funções como strpos que devolvem um índice numa string (que pode ser 0) ou false. Mas não adiantaria mudar só o and, teria que mudar o comportamento do if também. Tem que ver isso aí...

Funções

Funções em PHP não verificam se o número de argumentos passados corresponde ao número de parâmetros declarados. Poderia haver uma opção de compilação para inserir um check no começo de cada função.

While we are at it, poderíamos permitir declarar os tipos dos argumentos e adicionar checks no começo da função, ou verificar e emitir warnings em tempo de compilação quando possível. Mas isso ficaria para o futuro distante.

Muito me alegraria ter keyword arguments em lows. Eles poderiam ser passados como um array, i.e., (f x y a: 1 b: 2) seria equivalente a (f x y (array a: 1 b: 2)), e no corpo da função referências a a seriam traduzidas para referências à chave no array.

Variáveis globais e constantes

O plano original era que um acesso a uma variável não declarada localmente seria interpretado como um acesso à variável global de nome correspondente. Porém, estive pensando se não seria melhor usar um prefixo ou outra convenção para nomear variáveis globais (e.g., *foo*, a la Common Lisp, ou $foo, a la Ruby). A vantagem seria poder detectar erros de digitação em tempo de compilação, ao invés de assumir que o nome não declarado se trata de uma variável global.

Outro problema são as constantes globais. O Pharen aparentemente decide se um nome é uma constante ou uma variável exigindo que constantes sejam escritas em maiúsculas. ($_SERVER e afins também são escritos em maiúsculas, mas o Pharen usa um operador especial $ para acessar essas variáveis.) Nada no PHP exige que o nome de uma constante seja totalmente em maiúsculas, entretanto. Uma possibilidade seria usar +nome+, uma convenção usada por alguns programadores Common Lisp. I don't know. (error-reporting +E-ALL+)? Talvez (error-reporting #E_ALL)? (error-reporting <E_ALL>)?

Namespaces

Um dialeto de Lisp é classificado como Lisp-1 ou Lisp-2 dependendo de se nomes de funções e de variáveis vivem em um mesmo namespace (Lisp-1), ou se dois namespaces separados são usados (Lisp-2). PHP é mais próximo de um Lisp-2 do que de um Lisp-1, mas a distinção não é bem a mesma, pois em PHP variáveis possuem um prefixo que as identifica ($), enquanto em Lisp a posição do nome em uma chamada determina o namespace a ser utilizado (se o nome aparece como primeiro elemento em uma lista entre parênteses, ele é um nome de função; caso contrário, é uma variável).

Para o lows ser um Lisp-1, seria necessário determinar se (f x y) deveria ser traduzido para f($x, $y) ou $f($x, $y). No geral, isso não é possível (o compilador lows não tem conhecimento dos nomes definidos fora do arquivo que está compilando). Assim, um Lisp-2 é uma escolha mais natural: (f x y) sempre traduz para f($x, $y). Para obter o equivalente de $f($x, $y), i.e., para usar uma variável como a função a ser chamada, deve-se usar um operador especial, normalmente chamado funcall nos Lisps: (funcall f x y).

Em PHP 5.2, $f() só funciona se $f é uma string contendo o nome da função a ser chamada; não é possível chamar closures criadas com (lambda ...) dessa maneira. Por outro lado, call_user_func($f, ...) funciona tanto quando $f é uma string quanto com outros valores chamáveis. Como queremos que funcall funcione em ambos os casos, funcall deve ser traduzido para call_user_func.

A situação contrária ao funcall, i.e., quando se quer usar uma função nomeada como argumento para outra, é coberta pelo nosso operador 'foo, que produz uma string com o nome (convertido) da função foo. Como PHP não possui funções de primeira classe nem declarações locais de função, a string com o nome da função é suficiente para identificar a que função está se referindo.

Classes, métodos e bagulheiras

Definições de classes e métodos são relativamente straightforward. (defclass Foo atributos/métodos...) não tem por que ser muito diferente da construção equivalente em PHP. Podemos adicionar algumas abreviações, e.g., permitir definir um construtor padrão ao invés do costumeiro $this->x = $x; $this->y = $y; … do PHP.

A bagunça começa na hora de escolher uma sintaxe para chamada de métodos e acesso a atributos. Pharen usa algo como (-> objeto (método args...)), mas isso é meio verboso. Uma solução seria permitir (objeto->método args...) como uma abreviação, no caso mais comum em que objeto é uma variável, mas eu prefiro . para isso ao invés de ->. By the way, e se eu quiser um método cujo nome está em uma variável, i.e., $objeto->$método(args...)? Suponho que em Pharen seja possível escrever (-> objeto ($método args...)), que é como eles resolvem o problema do funcall, mas meh, não é assim que lows funciona – e se eu quiser uma expressão ao invés de uma variável para computar o nome do método? Que tal (-> objeto (funcall método args...))? Mas agora eu vejo que toda essa sintaxe conflita com a sintaxe para acesso de atributos. Em Pharen, (-> objeto nome) é equivalente a $objeto->nome. Novamente, para usar um nome vindo de uma variável, pode-se usar $nome, mas, novamente, isso não permite usar uma expressão para calcular o nome do atributo; (-> objeto (computa-nome)) seria interpretado como $objeto->computa_nome().

So. Estou com a sensação de que o adequado em lows seria que $obj->nome fosse algo como (-> obj 'nome), com o nome quoted. Sem o quote, nome seria avaliado como uma expressão que produz um nome. Quanto a chamada de métodos, talvez o jeito seja usar um operador distinto do usado para atributos. Meh. Independentemente dos operadores escolhidos, obj.nome seria considerado uma abreviação para (-> obj 'nome), já que acredito que acesso com um nome fixo a um objeto em uma variável pré-determinada seja o caso mais comum. E (obj.método args...) seria uma abreviação para $obj->método(args...). Isso tudo requer mais pensação.

Analogamente, o argumento do operador new também deveria ser quoted: (new 'Foo), não (new Foo). Nesse sentido, as operações do lows seriam mais parecidas com o make-instance e slot-value do CLOS do que com o Pharen. Now, o CLOS usa (método objeto args...) para chamadas de métodos (que na verdade são multi-métodos; o objeto é um argumento como os demais, que tambem podem ser especializados para diferentes classes). O problema de adotar essa sintaxe em lows é que é necessário saber que método é um método e não uma função para decidir que código PHP gerar. Uma possiblidade seria usar (.método objeto args...). Tem que ver isso aí. Outra situação a ser considerada é quando queremos passar um método como argumento para uma função (i.e., o equivalente do Array($obj, "método") do PHP). Hmmrgh...

Falando de abreviações, acho que seria uma boa adotar @nome como uma abreviação de $this->nome, a la Ruby. (@método args...) poderia ser abreviação de $this->método(args...).

[Um problema de usar (. objeto 'nome) é que . é sintaxe especial em Scheme, e atualmente eu uso a função read do Scheme para ler o código-fonte lows de um arquivo. Mais adiante, é de se pensar escrever um reader próprio para o lows. Isso também teria a vantagem de permitir manter informação de linha e coluna nas formas lidas.]

Macros

Como o compilador é escrito em Chicken Scheme, e não PHP (e nem por sonho eu pensaria em escrevê-lo em PHP (embora talvez reescrevê-lo em lows no futuro não seja algo de se descartar)), e não possui um interpretador próprio, ele não é capaz de rodar código lows em tempo de compilação. Isso exclui a possibilidade de macros procedurais nativas em lows. O meu plano, por ora, é permitir que macros em Scheme possam ser escritas, pois essas podem ser executadas pelo compilador. Por um lado é bizarro (em um Lisp) escrever macros em uma linguagem diferente da linguagem de programação, mas por outro isso tem a vantagem de permitir usar todas as funções do Chicken na geração de código. Uma outra possibilidade (não-exclusiva) é ter um sistema de macros baseado em casamento de padrões, como o syntax-rules do Scheme, que não requer a execução de código lows em tempo de compilação.

Além das macros que recebem código lows e produzem código lows, também seria interessante ter uma construção para emitir código PHP diretamente, similar ao asm do GCC e afins. No caso mais simples, a construção seria usada com uma string constante, mas nada impediria que expressões arbitrárias em Scheme pudessem ser usadas para gerar o código PHP.

Enough talk

Acho que era isso por enquanto. Sugestões, opiniões, etc., são bem-vindos.


[Version in English follows.]

In the last post, I talked about lows, a Lisp which compiles to PHP 5.2 which I started to develop, and what I got done so far. In this post, I intend to discuss some design questions of the language, more or less in the same spirit of the Blueprints for a shell series (in Portuguese) (except much shorter (or so I hope)). As I mentioned, I don't know when I'm going to work on lows again (after all, I've got some mundane tasks to do, such as a Master's to finish), but I'll leave some ideas documented here. Comments are always welcome.

Data types

The PHP data types don't match very well (read: mostly don't match) the conventional types from the Lisp world. The plan is to try to design the language to enable a sufficiently Lisp-like programming style with the existing types, rather than trying to recreate the traditional Lispy types in PHP, even more because that would not be very feasible in terms of performance in PHP.

Arrays

Basically the only type of collection PHP has is the array, which fills the roles of both lists and dictionaries (and even as a dictionary, it preserves insertion order). I don't see any efficient way to implement cons in PHP (adds an element to a list, creating a new list, in space/time O(1)). It would be possible to create a Cons class, but the overhead would be too large, and moreover the goal of the language is to interact easily with PHP code. So, arrays.

The map, filter, reduce and similar operations are independent of the representation of the collection and work equally well with arrays. The idiom of building lists using cons and recursion, on the other hand, is right out, but at the same time it wouldn't be a good choice in PHP anyway because function calls are more costly, and there is no tail call optimization. Actually, tail call optimization wouldn't help in this case anyway, because the recursion is not in tail position; in this situation, when performance matters, people usually accumulate the elements of the list in an argument to the recursive call, and call reverse! at the end of the loop, but if we're going to do that, we can equally well pass an array as the argument and accumulate the elements with push. Anyway.

In the future, map and company can be inlined, which makes these constructions basically equivalent to loops. Even the closure passed as an argument to map does not need to be created, and instead the body of lambda can be inserted directly in the loop.

We have to think about the syntax equivalent to PHP's Array(...). For simple lists, (array 1 2 3) works. The problem is the equivalent of the form Array("foo"=>1, "bar"=>2). When the key is a simple literal string, one possibility would be to use Chicken keywords: (array foo: 1 bar: 2). When the key is not a literal, something like (array (=> key1 val1) (=> key2 val2)) could be used, but that is somewhat verbose. I don't know.

One oddity of PHP arrays is that they are passed by copy. Apparently, there simply is no way to pass an array "by sharing", as values are usually passed in Lisp or Python. It is possible to create a reference (actually an alias) to a variable containing an array, but the reference is to the variable, not to the array; if another value is assigned to the variable, the reference/alias will reflect the modification. I guess we'll have to live with that.

Symbols

There is nothing equivalent to symbols in PHP. The main advantage of symbols over strings is that they can be compared in constant time, but PHP doesn't really have anything equivalent to the Lisp eq operator, so I think there wouldn't be much to gain in implementing symbols in lows. 'foo could be used as an abbreviation for "foo", perhaps.

Pharen has an "operator" #foo, which yields the string corresponding to the name of the function foo. The reason for that operator is that Pharen converts hyphens in function names to underscores (which is a good idea in a Lisp→PHP), and the # operator yields a string with the converted name. Perhaps ' could be used to the same purpose in lows.

By the way, lows does not have a true quote operator, as the code of a lows program is not represented by a lows (i.e., PHP) data structure. Yeah, lows is not a homoiconic language, and perhaps doesn't even deserve the name "Lisp" because of that. Macros in lows wouldn't be written in lows, but rather in (Chicken) Scheme, as we'll see later.

(What would be the meaning of '(+ 1 2) in lows? Syntax error? Shorthand for (array '+ '1 '2)?)

Miscellaneous

There isn't much to say about the remaining types (numbers, strings, booleans, and NULL). PHP converts nilly-willy between types, which is somewhat annoying, but there is not much we can do, unless we want to insert checks before every operation. (This could be a compilation option, useful for debugging, perhaps).

Operators

Arithmetic operations and the like in Lisp are usually functions, not special operators. However, we don't want to have to call a function in lows to perform arithmetic operations, because the cost of a function call in PHP is high. (Even Lisps usually optimize these calls away when it is possible to determine the types of the operands at compile time.) On the other hand, it would be nice to be able to pass + and the like as arguments to other functions. One possibility would be to expand '+ to (lambda (x y) (+ x y)), but that would fix the number of arguments of the operator, whereas ideally (+ 1 2 3) should work and translate to 1+2+3. Another possibility would be to have both a function, defined in the standard library, and the special operator, and the compiler would decide which to use depending on the situation. But there is no reason to include a function for each operator in the standard library, when the compiler can automatically generate the lambda with the appropriate code as needed.

Most lows operators would be directly equivalent, and would have the same name, as the PHP operators. The exceptions would be:

The binary operators have an extra detail: x || y in PHP always returns a boolean, whereas in Lisp (or x y) usually returns x if it is true-ish, and y otherwise. It would be possible to translate (or x y) to x? x : y (taking extra care not to evaluate x twice), but that kinda messes with the idea of straightforward translation (the conditions of if blocks would look weird in the compiled code). Maybe the compiler could detect when and/or are being used as the condition of an if and translate them to &&/|| in those cases.

In any case, what PHP thinks is true is at odds with what Lisps usually think is true: in Lisp, usually the false boolean value (or the empty list) is considered false, and everything else is considered true (including 0, the empty string, etc.). This behaviour would actually make more sense in PHP, which has functions such as strpos which return an index into a string (which may be 0) or false. But then it wouldn't be enough to change the behavior of and, we'd have to change the behavior of if too. Gotta think about it.

Functions

PHP functions don't check if the number of arguments in a call match the number of declared parameters. There could be a compilation option to insert a check at the beginning of each function.

While we are at it, we could allow declaring the types of the parameters and add checks at the beginning of the function, or verify and emit warnings at compile time if possible. But that would be left for the distant future.

It would much gladden me to have keyword arguments in lows. They could be passed in an array, i.e., (f x y a: 1 b: 2) would be equivalent to (f x y (array a: 1 b: 2)), and in the function body references to a would be translated to array dereferences.

Global variables and constants

The original plan was that an access to a variable not declared locally would be interpreted as an access to the global variable with that name. However, I have been thinking if it wouldn't be better to use a prefix or some other convention to name global variables (e.g., *foo* as in Common Lisp, or $foo as in Ruby). The advantage would be to be able to detect mistyped variable names at compile time, rather than assuming that the undeclared name refers to a global.

Another problem is global constants. Pharen seems to decide if a name is a constant or a variable by requiring all constants to have all-uppercase names. ($_SERVER et al. are all-uppercase too, but Pharen uses a special operator $ to access those.) Nothing in PHP requires that the name of a constant be all-uppercase, though. A possibility would be to use +name+, a convention used by some Common Lisp programmers. I don't know. (error-reporting +E-ALL+)? Maybe (error-reporting #E_ALL)? (error-reporting <E_ALL>)?

Namespaces

Lisp dialects are usually classified as Lisp-1 or Lisp-2 depending on whether function and variable names live in a single namespace (Lisp-1), or if there are two separate namespaces for them (Lisp-2). PHP is closer to a Lisp-2 than a Lisp-1, but the distinction is not quite the same, because in PHP variable names have a prefix identifying them ($), whereas in Lisp the position of the name in a call determines which namespace is to be used (if the name appears as the first element of a parenthesized list, it is a function name; otherwise, it's a variable name).

For lows to be a Lisp-1, it would be necessary to determine if (f x y) should be translated to f($x, $y) or $f($x, $y). In general, this is not possible (the lows compiler has no knowledge of the global names defined outside the file being compiled). Therefore, a Lisp-2 is a more natural choice: (f x y) always translates to f($x, $y). To get the equivalent of $f($x, $y), i.e., to use a variable as a function to be called, a special operator must be used, usually named funcall in Lisps: (funcall f x y).

In PHP 5.2, $f() only works if $f is a string containing the name of the function to be called; it is not possible to call closures created with (lambda ...) in this way. On the other hand, call_user_func($f, ...) works equally well when $f is a string and with other callable values. Since we want funcall to work in both cases, funcall must be translated to call_user_func.

The opposite situation to funcall, i.e., when we want to pass a named function as an argument to another, is covered by our ' operator, which yields a string with the (converted) name of the function foo. Since PHP does not have first-class functions or local function declarations, a string with the function name is enough to identify which function is being referenced.

Classes, methods, and stuff

Class and method definitions are relatively straightforward. (defclass Foo attributes/methods...) doesn't have to be very different from the equivalent construction in PHP. We can add some shorthands, e.g., allow a default constructor instead of the usual $this->x = $x; $this->y = $y; … in PHP.

The mess begins when choosing a syntax for method calls and attribute access. Pharen uses something like (-> object (method args...)), but that is somewhat verbose. A solution would be to allow (object->method args...) as a shorthand for the most common case when object is a variable, but I prefer . for this instead of ->. By the way, what if I want to use a method whose name is in a variable, i.e., $object->$method(args...)? I suppose Pharen allows (-> object ($method args...)), which is how they solve the funcall problem, but meh, that's not how lows works – what if I want an expression instead of a variable to compute the method name? What about (-> object (funcall method args...))? But now I see that all this syntax conflicts with the syntax for attribute access. In Pharen, (-> object name) is equivalent to $object->name. Again, to use a name from a variable, it allows $name, but, again, this does not allow using an expression to compute the name of the attribute; (-> object (compute-name)) would be interpreted as $object->compute_name().

So. I have the feeling that the appropriate thing in lows would be that $obj->name should be something like (-> obj 'name), with the name quoted. Without the quote, name would be evaluated as an expression yielding a name. As for method calls, perhaps the solution is to use a distinct operator from that used for attribute access. Meh. Regardless of the operators chosen, obj.name would be considered shorthand for (-> obj 'name), as I think access with a fixed name to an object in a known variable is the most common case. And (obj.method args...) would be shorthand for $obj->method(args...). All of this requires more thinking.

Analogously, the argument for the new operator should also be quoted: (new 'Foo), not (new Foo). In this sense, lows's operations would be more similar to make-instance and slot-value from CLOS than to Pharen. Now, CLOS uses (method object args...) for method calls (which are actually multi-methods; object is an argument just like the others, which can also be specialized for different classes). The problem of adopting this syntax in lows is that it requires knowing that method is a method and not a function to decide which PHP code to emit. One possibility would be using (.method object args...). Gotta think about that. Another situation to be considered is when we want to pass a method as an argument to a function (i.e., the equivalent of PHP's Array($obj, "method")). Hmmrgh...

Speaking of shorthand, I think it would be a good idea to adopt @name as shorthand for $this->name, a la Ruby. (@method args...) could be shorthand for $this->method(args...).

[One problem of using (. object 'name) is that . is special syntax in Scheme, and currently I use Scheme's read function to read lows source code from a file. Later on, one might think about writing a proper reader for lows. That would also have the advantage of allowing the compiler to keep track of line and column information.]

Macros

Because the compiler is written in Chicken Scheme, and not PHP (and I wouldn't even dream of writing it in PHP (though perhaps rewriting it in lows someday is not entirely unthinkable)), and has no lows intepreter, it is not capable of running lows code at compile time. This excludes the possibility of native procedural macros in lows. My plan, for now, is to allow writing macros in Scheme, as those could be run by the compiler. On the one hand it is somewhat weird (for a Lisp) to write macros in a different language, but on the other hand this has the advantage of allowing one to use all Chicken functions in code generation. Another (non-mutually-exclusive) possibility is to have a macro system based on pattern matching, like Scheme's syntax-rules, which does not require running lows code at compile time.

Beside macros which take lows code and yield lows code, it would also be interesting to have a construction to emit PHP code directly, similar to asm in GCC and the like. In the simplest case, the construction would be used with a constant string, but nothing would exclude using arbitrary Scheme expressions to generate PHP code.

Enough talk

I think that's it for now. Suggestions, opinions, etc., are welcome.

21 comentários

Lisp meets PHP

2015-09-02 04:25 -0300. Tags: comp, prog, pldesign, php, lisp, lows, in-english

[This post is also available in English.]

Como eu andei comentando por aí, eu comecei a implementar uma linguagem Lisp-like que compila para PHP 5.2, chamada lows. Não sei quando vou mexer nesse projeto de novo, mas deixo aqui algumas notas para o meu eu futuro e para quem tiver interesse.

Prelúdio

Tudo começou no domingo retrasado, quando eu resolvi dar uma mexida no blog system, for a change. Mexer no blog sempre é uma experiência ambivalente, pois por um lado tem uma porção de idéias que eu gostaria de implementar nele, mas por outro lado eu tenho que fazer isso em PHP, porque é a única coisa que roda no inf.ufrgs.br (e eu não pretendo pagar por hospedagem any time soon). Eu comentei com uma pessoa que se eu tivesse a opção, eu já teria reescrito o blog em Scheme há muito tempo. Ela me perguntou por que eu não fazia algo para rodar Scheme em PHP, e eu comentei que já tinha pensado em fazer um compilador de Lisp para PHP, mas que achava que era muita mão só para poder escrever o blog em Lisp. Assim, eu segui meu domingo fuçando no blog em PHP.

Depois de uma porção de gambiarras e mais uma porção de concessões às bizarrices do PHP (e.g., existem os métodos mágicos __call, __callStatic e __get, mas não existe um __getStatic, sabe-se lá por quê), eu consegui reescrever a parte do blog responsável por mensagens multilíngües de uma maneira que me agradasse, e até já não estava mais achando tão horrível escrever o código em PHP.

No final do dia, depois de ter testado o código no meu servidor local, eu resolvi fazer upload da nova versão para o inf.ufrgs.br. Para minha surpresa, o PHP começou a reportar uma porção de erros no código. Turns out que a versão do PHP que roda no inf.ufrgs.br é a "5.2.6-1+lenny16". Para quem não sabe, lenny é o codinome do Debian 5. O Debian 5 foi lançado em 2009 e não recebe mais atualizações desde 2012. Três releases estáveis do Debian saíram desde então (as releases estáveis do Debian saem a cada mais ou menos dois anos). Meanwhile, eu estava rodando PHP 5.6.12 em um Debian testing em casa, e praticamente todo o código que eu tinha escrito usava features introduzidas no PHP 5.3.

Depois de tentar sem muito sucesso mudar um pouco o código para ver se conseguia fazê-lo rodar no PHP 5.2, eu resolvi largar de mão e deixar para mexer no código outro dia. Porém: (1) eu não estava a fim de enfeiar o código só para fazê-lo rodar em um PHP velho; (2) more generally, eu não estava a fim de mexer em PHP de novo; e (3) eu não estava conseguindo dormir aquele dia. Conclusão: comecei a escrever um tradutor Lisp→PHP, primariamente for the lol. Mais uma noite mal-dormida, e eis que eu tinha um tradutor (ou compilador, como preferir) Lisp→PHP que fazia pouca coisa, mas o suficiente para me convencer de que a idéia era pelo menos viável. Nasceu assim o lows, ou Lisp for Old Web Servers.

Idéia

A idéia do projeto é criar uma linguagem Lisp-like que satisfaça os seguintes objetivos:

Compilar para PHP 5.2. A idéia é eu poder rodar o código resultante no inf.ufrgs.br (e idealmente escrever a próxima versão do blog em lows), então eu preciso "targetar" especificamente PHP 5.2. Eu também podia tentar convencer a galera da admrede a atualizar o servidor, mas (1) acho pouco provável que isso aconteça any time soon, e eu não estava a fim de esperar; (2) a essa altura eu já tinha tomado a limitação a PHP 5.2 como um desafio (lembre-se de que eu comecei o projeto para matar tempo enquanto o sono não vinha); (3) já existe um projeto similar, chamado Pharen, que targeta PHP 5.5, e eu queria um diferencial (a.k.a. desculpa) para justificar o meu projeto.

Gerar código PHP relativamente straightforward. Tanto quanto possível, o código PHP resultante da compilação deve ser uma tradução mais ou menos direta do código lows original. A idéia é facilitar a depuração (e em particular a tarefa de encontrar o código lows correspondente a um erro reportado no código PHP), e também a esperança de que quanto mais direto for o código resultante, menor o impacto na performance de escrever o código em lows ao invés de diretamente em PHP.

Integrar facilmente com PHP. Deve ser possível usar funções, classes, etc. do PHP a partir de código lows e vice-versa, sem necessidade de conversões, anotações e afins.

Manter uma essência Lisp-like. A idéia não é simplesmente criar um redressing de PHP em S-expressions, mas sim uma linguagem que permita programar em um estilo semi-funcional e "Lispy" e evite as bizarrices do PHP na medida do possível (ao mesmo tempo em que introduz outras bizarrices).

Esse conjunto de objetivos influencia tanto a implementação (que deve gerar um PHP relativamente limpo/direto) quanto o design da linguagem (que não deve fugir muito do PHP para permitir a tradução relativamente direta e a compatibilidade com código PHP).

Transformando Lisp em PHP

Expressões e statements

Um desafio que eu encontrei logo no começo é o fato de que o PHP faz uma distinção entre expressões e statements, que (mostly) não existe em Lisp. Em particular, coisas como if, let (declaração de variáveis locais) e progn (executa uma seqüência de expressões e retorna o valor da última, mais ou menos análogo a um bloco entre chaves em PHP, mas que produz um valor) são expressões em lows. O if em princípio até poderia ser traduzido para o operador ternário (test? then : else), e o let poderia ser mais-ou-menos contornado já que atribuição é uma expressão em PHP. O problema é que PHP não tem um operador vírgula como em C. Coisas como:

(+ 1 (let ((x 23)
           (y 42))
       (* x y)))

não possuem uma tradução direta para PHP, pois não é possível escrever 1 + ($x=23, $y=42, $x*$y). Uma solução gambiarrenta seria gerar:

1 + ((($x=23) || TRUE) ? ((($y=42) || TRUE) ? ($x*$y)
                                            : whatever)
                       : whatever)

o que simula o operador vírgula usando só um branch do operador ternário, mas: (1) isso não funciona no caso geral (em particular, se uma das expressões é um progn contendo um echo ou alguma outra coisa statementosa); (2) isso vai totalmente contra a idéia de gerar código straightforward. A solução é mover as atribuições para antes da soma, mas, no caso geral, só mover qualquer coisa que não seja uma expressão em PHP para antes da expressão não é suficiente: se os branches de um condicional contêm expressões com efeitos colaterais, não é possível movê-las para fora do condicional. Por exemplo, em algo como:

(defun print-and-return (val)
  (echo "O valor é " val "\n")
  val)

(+ 1 (if (> x y)
         (let ((a (print-and-return (- x y))))
           (* a a))
       0))

não é possível traduzir a soma para:

$a = print_and_return($x-$y);
1 + (($x>$y)? ($a*$a) : 0)

pois print_and_return não pode ser chamada antes que o teste $x>$y seja realizado.

[A essa altura talvez lhe ocorra (como me ocorreu) o pensamento: "Ok, e por que a gente simplesmente não proíbe expressões complexas desse tipo aninhadas em outras expressões? Quando é que eu vou usar isso anyway?" Mas esse é justamente o tipo de limitação tosca de que nós estamos tentando fugir criando uma nova linguagem ao invés de programar em PHP! "Do not tell me “that’s what you get for doing weird things”. If two features exist, someday, someone will find a reason to use them together."]

A solução que eu encontrei foi traduzir o (if ...) para um bloco if em PHP, armazenar o valor do if em uma variável temporária, e usar a variável temporária na soma. O exemplo anterior fica algo como:

if ($x>$y) {
    $a = print_and_return($x-$y);
    $tmp = $a*$a;
} else {
    $tmp = 0;
}

1 + $tmp

Isso significa que para traduzir uma expressão como (+ ...), pode ser necessário emitir blocos de código antes da tradução da soma propriamente dita. Conseqüentemente, a função de tradução não pode ser simplesmente algo como:

translate[(+ lhs rhs)] = translate[lhs] + translate[rhs]

pois tanto a tradução de lhs quanto de rhs podem requerer a inserção de blocos de código antes da soma (e a soma, por sua vez, pode estar aninhada em outra expressão).

A solução que eu encontrei para esse problema foi fazer as funções de tradução retornarem dois valores: a expressão equivalente em PHP, e uma lista de "efeitos", que são basicamente (mas não necessariamente) instruções para emitir código nas redondezas da tradução. Por exemplo, a função de tradução aplicada ao if do exemplo gera a expressão $tmp (que pode ser inserida no meio de outra expressão que usa o valor do if), e o efeito (EmitBefore bloco-if-em-PHP), que indica que o bloco-if-em-PHP deve ser inserido antes da expressão que contém o if na geração do código PHP. Como a inserção só pode ser realizada fora de uma expressão, o efeito é propagado pelas funções de tradução de expressões, até que ele chega em uma função que emite statements (e.g., o corpo de um bloco if do PHP, ou o corpo de uma função) e pode então ser emitido. Pseudocodiciosamente (oops, hmm):

translate[(+ lhs rhs)] =
   let
       lhs-trans; lhs-effects = translate[lhs]
       rhs-trans; rhs-effects = translate[rhs]
   in
       lhs + rhs; lhs-effects ++ rhs-effects


translate-statement[item] =
   let
       item-trans; item-effects = translate[item]
   in
       (código correspondente aos EmitBefore em item-effects) ++ item-trans ;
       (efeitos em item-effects excluindo os EmitBefore já processados)

O mesmo mecanismo pode ser usado para emitir código em outras situações (e.g., no caso do lambda, como veremos adiante), ou para coletar e propagar informações durante a tradução. Por exemplo, quando uma variável x que não possui declaração visível é usada, é emitido um efeito (Global x). A função que traduz o corpo de uma função coleta esses efeitos para gerar declarações do tipo global $x; no começo da função.

lambda

O próximo desafio foi traduzir o lambda para PHP. PHP >=5.3 possui closures (meio toscas – é necessário declarar explicitamente que variáveis são capturadas pela closure – mas elas existem), mas PHP 5.2 não. A próxima coisa que eu pensei foi usar uma classe "callable" com um método mágico __invoke, mas turns out que classes chamáveis só foram introduzidas em PHP 5.3 também. Porém, as funções que aceitam coisas chamáveis em PHP, como call_user_func e usort, aceitam arrays da forma Array(objeto, nome-de-método) como chamáveis. Pois, aí está algo que o lambda pode retornar.

Capturar as variáveis em uma closure mostrou-se bem mais fácil do que eu antecipava, graças às referências do PHP. Uma closure em lows é representada por uma classe com um membro/slot/propriedade/atributo/whatever para cada variável capturada. Quando a classe é instanciada, as variáveis são passadas por referência para o construtor. Dentro do corpo do lambda, referências a variáveis capturadas x são traduzidas para $this->x; como $this->x foi inicializado com uma referência ao $x capturado, o corpo do lambda vê a mesma variável $x através do atributo, inclusive refletindo modificações à mesma.

Como exemplo, algo como:

(defun adder (x)
  (lambda (n)
    (+ x n)))

(defun main ()
  (let ((f (adder 10)))
    (call_user_func f 5)))

vira algo como:

class Closure1 {
    function __construct(&$x) {
        // Captura de variáveis.
        $this->x = &$x;
    }

    function __invoke($n) {
        // Corpo do lambda.
        return $this->x + $n;
    }
}

function adder($x) {
    // Cria a closure, passando a variável a ser capturada para o seu construtor,
    // e retorna um valor que, quando chamado, chama o método "__invoke" da closure.
    return Array(new Closure1($x), "__invoke");
}

function main() {
    $f = adder(10);
    return call_user_func($f, 5);
}

E assim, o PHP e suas referências nos surpreendem positivamente (o que é uma surpresa in itself).

By the way, o mecanismo de efeitos aqui é usado para duas coisas: (1) a geração da classe antes da função que contém o lambda é feita propagando um efeito (EmitBeforeToplevel definição-da-classe); (2) cada referência a uma variável externa ao lambda gera um efeito (CapturedVar x); esses efeitos são coletados pela função que traduz o lambda para saber que atributos devem ser inicializados na classe e que argumentos devem ser passados ao construtor. Quando eu criei a treta dos efeitos eu não tinha pensado em todas essas aplicações, então mui me alegrou descobrir que eu podia reusar o mecanismo para essas coisas.

Name clashes

Em PHP, variáveis locais têm como escopo a função inteira onde se encontram, não apenas o bloco onde foram declaradas. Conseqüentemente, em código como:

(let ((x 23))
  (echo "x aqui vale 23: " x "\n")
  (let ((x 42))
    (echo "x aqui dentro vale 42: " x "\n"))
  (echo "x aqui fora ainda vale 23: " x "\n"))

não se pode usar o mesmo nome para as duas variáveis x na tradução, pois a definição mais interna de x sobrescreveria a mais externa. A solução e renomear uma das (ou ambas as) variáveis. O ideal seria fazer o mínimo de renomeações possível, para facilitar a leitura e depuração do código resultante. Porém, a implementação atual simplesmente renomeia todas as variáveis (adicionando um prefixo _número_), já que testar quando uma variável deve ser renomeada não é muito simples. Essa decisão não é local: mesmo não havendo nenhuma variável x visivelmente declarada no ponto onde ocorre um (let ((x 23)) ...), ainda assim é necessário renomear o x se em um ponto posterior da função uma variável global x for referenciada.

O algoritmo de renomeação / geração de nomes temporários assume que nomes iniciados por _número_ são reservados para o compilador. Acredito que isso não seja um problema na prática. (Para o caso de variáveis locais, uma variável _42_ vai ser renomeada para algo como _1__42_ de qualquer forma.) Um problema mais sério dessa abordagem é no escopo global, em particular nos nomes gerados para as classes que implementam closures (e.g., _1_Closure), pois esses nomes podem conflitar com closures criadas em outros arquivos (e.g., quando os resultados da tradução de múltiplos arquivos são incluídos com include em um programa PHP). Talvez uma solução seja incluir o nome do arquivo no nome da classe, ou gerar um hash a partir do código da closure (mas isso ainda gera conflito se um lambda idêntico aparece em outro arquivo), or something. I don't know. Também seria bom se o nome da classe fosse informativo o suficiente para indicar de onde saiu a definição no código original (e.g., _1_Closure_arquivo_função_linha). [Side remark: namespaces não existem em PHP 5.2.]

Um conflito de variável mais sutil é quando um let é executado múltiplas vezes e um lambda captura uma variável definida pelo let. Por exemplo, supondo a existência de uma construção while:

(let ((array-of-lambdas (array))
      (i 0))
  (while (< i 5)
    (let ((n 0))
      (array_push array-of-lambdas
                  (lambda ()
                    (set! n (+ n 1))
                    (echo n))))
    (set! i (+ i 1))))

Isso seria traduzido para algo como:

Class _1_Closure {
    function __construct(&$n) {
        $this->n = &$n;
    }

    function __invoke() {
        $this->n = $this->n + 1;
        echo $this->n;
    }
}

$array_of_lambdas = Array();
$i = 0;
while ($i < 5) {
    $n = 0;
    array_push($array_of_lambdas,
               Array(new _1_Closure($n), "__invoke"));
    $i = $i + 1;
}

O problema é que todas as iterações do loop usam a mesma variável $n, que é passada por referência ao construtor da closure; o correto seria cada iteração capturar um $n diferente. A solução é emitir uma chamada a unset($n) no final do while, de maneira que cada iteração crie uma variável nova, mas eu ainda não implementei isso.

PHP formatado

Um dos objetivos do projeto é gerar PHP legível, e isso envolve gerar código com indentação adequada. Depois de alguns false starts (na versão inicial, as funções de tradução geravam strings de código PHP diretamente, e a minha idéia original era usar caracteres especiais do ASCII como indicadores de "increase indentation" e "decrease indentation" quando o código fosse impresso, mas eu me dei conta de que não dava para escolher caracteres para isso porque qualquer caractere pode aparecer em uma string; além disso, misturar geração de código e questões de formatação estava ficando um bocado desagradável), eu resolvi fazer as funções de tradução gerarem estruturas representando árvores de sintaxe abstrata (ASTs) de PHP. Depois da tradução, as árvores são passadas a uma função print-php que trata dos detalhes sórdidos de imprimir o código com quebras de linha, indentações, espaços e parênteses nos lugares apropriados. Separation of concerns FTW.

O futuro

Como o post ficou grande, e eu deveria ir dormir, ficaremos por aqui. Em um post futuro, pretendo falar de algumas features que falta implementar, tais como classes, chamadas de métodos e demais firulas orientadas a objetos, bem como as decisões de design mais tricky (que eram o objetivo inicial do post, mas enfim). Quem tiver interesse, pode dar uma olhada no código no GitHub.


[English version follows.]

As I have been talking about, I started implementing a Lisp-like language which compiles to PHP 5.2, called lows. I don't know when I'm going to work on this project again, but I'll leave here some notes for my future self and whoever might be interested.

Prelude

It all began last last Sunday, when I decided to play with my blog system, for a change. Working on the blog system is always an ambivalent experience, because on the one hand there is a bunch of ideas I would like to implement in it, but on the other hand I have to do it in PHP, as that is the only thing that runs at inf.ufrgs.br (and I don't plan to pay for hosting any time soon). I commented to a person that if I had the choice, I would have rewritten the blog in Scheme long ago. She asked my why I didn't make something to run Scheme in PHP, and I said I had already though of writing a compiler from Lisp to PHP, but that I thought it was too much work just to be able to write the blog in Lisp. So, I went on with my Sunday messing with the blog in PHP.

After a number of kludges and another number of concessions to the oddities of PHP (e.g., there are the __call, __callStatic and __get magic methods, but no __getStatic, who knows why), I succeeded in rewriting the part of the blog responsible for multilingual messages in a way that pleased me, and I was even not finding it so horrible to write the code in PHP.

At the end of the day, after having tested the code in my local server, I decided to upload the new version to inf.ufrgs.br. To my surprise, PHP started reporting lots of errors in the code. It turns out that the version of PHP running at inf.ufrgs.br is "5.2.6-1+lenny16". For those who don't know, lenny is the codename of Debian 5. Debian 5 was launched in 2009 and does not get updates since 2012. Three stable Debian releases have been out since then (the stable releases of Debian are launched more or less every two years). Meanwhile, I was running PHP 5.6.12 in a Debian testing at home, and practically all the code I had written used features introduced in PHP 5.3.

After trying without much success to change the code a bit to see if I got it to run on PHP 5.2, I decided to leave it alone and work on the code another day. However: (1) I was not willing to uglify my code just to make it run on an old PHP; (2) more generally, I wasn't willing to work with PHP again; and (3) I was having difficulties to sleep that day. Conclusion: I stared writing a Lisp→PHP translator, primarily for the lol. One more badly-slept night later, and so it was that I had a Lisp→PHP translator (or compiler, if you prefer) that did little, but enough to convince me that the idea was at least feasible. Thus lows, or Lisp for Old Web Servers, was born.

Idea

The idea of the project is to create a Lisp-like language which satisfies the following criteria:

Compile to PHP 5.2. The idea is for me to be able to run the resulting code at inf.ufrgs.br (and ideally write the next version of the blog in lows), so I need to target specifically PHP 5.2. I could also try to convince the admins at INF to upgrade the server, but (1) I don't think that's going to happen any time soon, and I was not willing to wait; (2) at this point I had already taken the limitation to PHP 5.2 as a challenge (remember that I started the project to kill time while I couldn't sleep); (3) there is already a similar project, called Pharen, which targets PHP 5.5, and I wanted a distinctive feature (a.k.a. excuse) to justify my project.

Emit relatively straightforward PHP code. As much as possible, the PHP code resulting from compilation should be a more or less direct translation of the lows source. The idea is to ease debugging (and in particular the task of finding the lows code corresponding to a PHP error message), and also the hope that the more direct the resulting code, the smaller the impact on performance of writing the code in lows rather than directly in PHP.

Integrate easily with PHP. It must be possible to use PHP functions, classes, etc. from lows code and vice-versa, without requiring conversions, annotations and the like.

Keep a Lisp-like essence. The idea is not simply to make a redressing of PHP in S-expressions, but rather a language which enables programming in a semi-functional and "Lispy" style and avoids the oddities of PHP as much as possible (while introducing new oddities of its own).

This set of goals influences both the implementation (which must emit relatively clean/direct PHP code) and the design of the language (which must not stray away too much from PHP to allow a relatively direct translation and compatibiltity with PHP code).

Transforming Lisp into PHP

Expressions and statements

A challenge I found right at the beginning is the fact that PHP makes a distinction between expressions and statements, which (mostly) does not exist in Lisp. In particular, things like if, let (local variable declaration) and progn (runs a sequence of expressions and returns the value of the last one, more or less like a block in braces in PHP, but yielding a value) are expressions in lows. if in principle could be translated to the ternary operator (test? then : else), and let could be more-or-less worked around because assignment is an expression in PHP. The problem is that PHP does not have a comma operator like that of C. Things like:

(+ 1 (let ((x 23)
           (y 42))
       (* x y)))

don't have a direct translation to PHP, because it is not possible to write 1 + ($x=23, $y=42, $x*$y). A kludgy solution would be to emit:

1 + ((($x=23) || TRUE) ? ((($y=42) || TRUE) ? ($x*$y)
                                            : whatever)
                       : whatever)

which emulates the comma operator by using only one branch of the ternary operator, but (1) that doesn't work in the general case (in particular, if one of the expressions is a progn containing an echo or some other statement-y thing); (2) that goes totally against the idea of emitting straightforward code. The solution is to move the assignments to before the addition, but, in general, just moving anything that is not an expression to before the expression is not enough: if the branches of a conditional contain expressions with side effects, they cannot be moved out of the conditional. For instance, in something like:

(defun print-and-return (val)
  (echo "The value is " val "\n")
  val)

(+ 1 (if (> x y)
         (let ((a (print-and-return (- x y))))
           (* a a))
       0))

it is not possible to translate the addition to:

$a = print_and_return($x-$y);
1 + (($x>$y)? ($a*$a) : 0)

because print_and_return cannot be called before the test $x>$y is performed.

[At this point, perhaps it ocurred to you (as ocurred to me) the thought: "Okay, why don't we just forbid complex expressions like these nested in other expressions? When will I use that anyway?" But that is exactly the kind of weird limitation which we are trying to escape from by creating a new language instead of programming in PHP! "Do not tell me “that’s what you get for doing weird things”. If two features exist, someday, someone will find a reason to use them together."]

The solution I found was to translate (if ...) to a PHP if block, store the value of the if expression into a temporary variable, and use the temporary in the addition. The previous example becomes something like:

if ($x>$y) {
    $a = print_and_return($x-$y);
    $tmp = $a*$a;
} else {
    $tmp = 0;
}

1 + $tmp

This means that to translate an expression like (+ ...), it may be necessary to emit blocks of code before the translation of the addition itself. As a consequence, the translation function cannot be just something like:

translate[(+ lhs rhs)] = translate[lhs] + translate[rhs]

because both lhs and rhs may require inserting blocks of code before the addition (and the addition itself may be nested in another expression).

The solution I found for this problem was to make the translation functions return two values: the equivalent expression in PHP, and a list of "effects", which are basically (but not necessarily) instructions to emit code in the surroundings of the translation. For example, the translation function, when applied to the example if, yields the expression $tmp (which can be inserted in the middle of another expression which uses the value of the if, and the effect (EmitBefore PHP-if-block), which indicates that PHP-if-block must be inserted before the expression containing the if when emitting the PHP code. Since the insertion can only be performed outside of an expression, the effect is propagated by the functions responsible for translating expressions, until it arrives at a function which emits statements (e.g., the body of a PHP if block, or a function body), where it can then be emitted. Pseudocodefully:

translate[(+ lhs rhs)] =
   let
       lhs-trans; lhs-effects = translate[lhs]
       rhs-trans; rhs-effects = translate[rhs]
   in
       lhs + rhs; lhs-effects ++ rhs-effects


translate-statement[item] =
   let
       item-trans; item-effects = translate[item]
   in
       (code corresponding to the EmitBefores in item-effects) ++ item-trans ;
       (effects in item-effects excluding those EmitBefores already processed)

The same mechanism can be used to emit code in other situations (e.g., in the case of lambda, as we'll see later), or to collect and propagate information during translation. For example, when a variable x which has no visible declaration is used, a (Global x) effect is generated. The function responsible for translating functions collects those effects to generate global $x; declarations at the beginning of the function.

lambda

The next challenge was to translate lambda to PHP. PHP >=5.3 has closures (somewhat crappy ones – one must declare explicitly which variables are to be captured by the closure – but they exist), but PHP 5.2 doesn't. The next thing I thought was to use a "callable" class with an __invoke magic method, but it turns out that callable classes were introduced only in PHP 5.3 too. However, the functions which accept callable things in PHP, such as call_user_func and usort, accept arrays of the form Array(object, method-name) as callables. So, this is something that lambda can return.

Capturing variables in a closure proved much easier than I anticipated, thanks to PHP references. A closure in lows is represented as a class with a member/slot/property/attribute/whatever for each captured variable. When the class is instantiated, the variables are passed by reference to the constructor. Inside the body of lambda, references to captured variables x are translated to $this->x; because $this->x was initialized with a reference to the captured $x, the lambda body sees the same variable $x through the attribute, even reflecting modifications to it.

As an example, something like:

(defun adder (x)
  (lambda (n)
    (+ x n)))

(defun main ()
  (let ((f (adder 10)))
    (call_user_func f 5)))

turns into something like:

class Closure1 {
    function __construct(&$x) {
        // Variable capture.
        $this->x = &$x;
    }

    function __invoke($n) {
        // lambda body.
        return $this->x + $n;
    }
}

function adder($x) {
    // Create the closure, passing the variables to be captured to the constructor,
    // and returns a value that, when called, calls the closures' "__invoke" method.
    return Array(new Closure1($x), "__invoke");
}

function main() {
    $f = adder(10);
    return call_user_func($f, 5);
}

And so, PHP and its references surprise us positively (which is a surprise in itself).

By the way, the effects mechanism is used here for two things: (1) emitting the class before the function containing the lambda is done by propagating an (EmitBeforeToplevel class-definition) effect; (2) each reference to a variable external to the lambda generates a (CapturedVar x) effect; these effects are collected by the function responsible for translating lambda to find out which attributes must be initialized in the class and which arguments must be passed to the constructor. When I came up with the effects idea I hadn't thought about all those applications, so it much gladdened me to find out I could use the mechanism for those things too.

Name clashes

In PHP, local variables have the scope of the entire function where they are created, not just the block where they were declared. As a consequence, in code like:

(let ((x 23))
  (echo "x here is 23: " x "\n")
  (let ((x 42))
    (echo "x here inside is 42: " x "\n"))
  (echo "x out here still is 23: " x "\n"))

we cannot use the same name for both x variables in the translation, because the innermost definition of x would overwrite the outermost one. The solution is to rename one of the (or both) variables. Ideally we should perform the minimum number of renames possible, to make it easier to read and debug the resulting code. However, the current implementation simply renames all variables (adding a _number_ prefix), since testing when a variable must be renamed is not very simple. This decision is non-local: even if there is no visible declaration of a variable x at the point where a (let ((x 23)) ...) occurs, it is still necessary to rename x if at some later point in the function a global variable x is referenced.

The renaming / temporary name generation algorithm assumes that names beginning with _number_ are reserved to the compiler. I think this is not a problem in practice. (In the case of local variables, a variable _42_ would be renamed to something like _1__42_ anyway.) A more serious problem of this approach is at the global scope, in particular in the names of generated classes which implemente closures (e.g., _1_Closure), because those names may conflict with closures created in other files (e.g., when the translation results of multiple files are included into a single PHP program). Perhaps a solution is to include the file name in the name of the class, or to compute a hash from the closure code (but this would still cause conflicts if an identical lambda appears in another file), or something. I don't know. It would also be nice if the class name were descriptive enough to indicate where the definition came from in the source code (e.g., _1_Closure_file_function_line). [Side remark: namespaces don't exist in PHP 5.2.]

A more subtle variable conflict occurs when a let is executed multiple times and a lambda captures a variable defined by the let. For example, supposing the existence of a while construction:

(let ((array-of-lambdas (array))
      (i 0))
  (while (< i 5)
    (let ((n 0))
      (array_push array-of-lambdas
                  (lambda ()
                    (set! n (+ n 1))
                    (echo n))))
    (set! i (+ i 1))))

This would be translated to:

Class _1_Closure {
    function __construct(&$n) {
        $this->n = &$n;
    }

    function __invoke() {
        $this->n = $this->n + 1;
        echo $this->n;
    }
}

$array_of_lambdas = Array();
$i = 0;
while ($i < 5) {
    $n = 0;
    array_push($array_of_lambdas,
               Array(new _1_Closure($n), "__invoke"));
    $i = $i + 1;
}

The problem is that all iterations of the loop use the same $n variable, which is passed by reference to the closure constructor; the correct would be for each iteration to capture a different $n. The solution is to emit an unset($n) at the end of the while body, so that each iteration would create a new variable, but I haven't implemented this yet.

Pretty-printed PHP

One of the goals of the project is to emit readable PHP code, and this involves emitting properly indented code. After some false starts (in the initial version, the translation functions emitted PHP code strings directly, and my original idea was to use some special ASCII characters to indicate "increase indentation" and "decrease indentation" when printing, but I realized that I could not choose any characters for that because any character can appear in a string; moreover, mixing code generation and formatting questions was becoming rather ugly), I decided to make the translation function emit structures representing PHP abastract syntax trees (ASTs). After translation, the trees are passed to a print-php function, which takes care of the gory details of printing the code with line breaks, indentation, spaces and parentheses at the proper places. Separation of concerns FTW.

The future

As this post turned quite long, and I should get some sleep, we'll finish here. In a future post, I intend to talk about some features that are still missing, such as classes, method calls and other stuff, as well as trickier design decisions (which were the initial goal of this post, but anyway). If you are interested, you can look at the code on GitHub.

1 comentário

lash status update

2015-05-14 23:42 -0300. Tags: comp, prog, pldesign, lash, life

Faz quase dois meses desde o primeiro commit do lash. O status do projeto é o seguinte:

No decorrer dessa última semana, o parser original, baseado na biblioteca Comparse de parser combinators, foi substituído por um parser descendente recursivo escrito à mão. Os motivos principais para a reescrita foram que a Comparse não mantém informação de linha e coluna dos elementos parseados, aparentemente não tem suporte nenhum a error reporting (o parser simplesmente "backtracka" quando se depara com um erro, até que o parser inicial backtracka e devolve #f), e o parser estava com uns comportamentos estranhos diante de algumas entradas (o que não é culpa da Comparse, mas não tinha por que eu perder tempo debugando se eu já teria que reescrever o parser uma hora ou outra pelos outros motivos citados). O handling de espaços e newlines no parser antigo também estava o caos, enquanto no atual aparentemente tudo funciona como deveria nesse quesito.

O parser novo reconhece quase toda a linguagem prevista para a "release inicial", lança exceções nos pontos certos do código ao encontrar erros de sintaxe (embora as exceções ainda não sejam muito descritiva, mas já é um começo), e armazena a linha e a coluna de cada construção nos nós da árvore sintática (com pequenos erros, mas nada difícil de resolver). O código ainda está meio crude, e tem muita coisa que ainda dá para refatorar (e.g., repetições que estão codificadas como loops explícitos ao invés de uma construção que abstraia a repetição), mas isso vai ir se resolvendo ao longo do tempo. De repente as partes mais abstratas do parser podem até virar uma biblioteca de parser combinators no futuro (com a diferença de que eu estou usando uma struct mutável e exceções ao invés de uma mônada para manter o estado do parser e indicar erros, o que seria meio unusual para uma biblioteca de parser combinators, mas whatever; ninguém disse que seriam parser combinators funcionais).

O parser novo reconhece mais construções do que o resto do shell é capaz de executar (por exemplo, pipelines, redirects, &, &&, ||, $()... em outras palavras, praticamente todas as funções do shell), o que de certa forma é bom, porque me compele a implementar as coisas que faltam. Nos últimos dias o desenvolvimento anda numa taxa mais ou menos ok (para mim), e acho que é mais ou menos realista prever uma release 0.1* mais ou menos funcional para julho. Pelo menos é (mais ou menos) isso que eu espero. Isso é bom, porque em um momento de otimismo em março eu submeti uma proposta de palestra para o FISL sobre o shell e, na vaga possibilidade de ela ser aceita, até lá seria bom o shell estar num estado usável. (Eu submeti a proposta sob a premissa de que se tudo desse errado com o shell e eu fosse aceito era só pedir para tirarem a palestra, mas a essa altura acho que isso não será mais necessário. Tudo isso assumindo que eu seja aceito, o que seria muito doido, na real.)

Comecei a usar a wiki do projeto no GitHub para fazer anotações. O plano que ela venha a conter:

Para editar a wiki é necessário criar uma conta no GitHub, aparentemente, mas acho que podemos conviver com isso. Contribuições são sempre bem-vindas.

Quanto à linguagem do shell, algumas coisas mudaram:

Durante o desenvolvimento do novo parser eu descobri um "bug" no Chicken que faz com que variáveis criadas com define dentro de um cond sejam declaradas como globais. A galera na mailing list parece ser da opinião de que isso não é um bug e sim uma feature, entretanto. Meanwhile, eu resolvi o problema no lash redefinindo o cond para wrappear as cláusulas em um (let () ...) implícito (o que cria um "scope boundary" que torna as definições locais), e de brinde ainda lançar uma exceção se nenhuma cláusula for verdadeira. Scheme, yay.

Enquanto o lash anda às mil maravilhas, o mestrado vai por água abaixo, mas isso é assunto possivelmente para outro post.

_____

* No momento eu não estou numerando as versões, mas pelo esquema de numeração previsto (<major>.<minor>.<número de commits desde o último update do minor>), estaríamos na versão 0.0.31. Parece bastante, mas é porque eu tenho o hábito de commitar loucamente enquanto estou mexendo no código.

1 comentário

Mind dump

2015-04-10 01:34 -0300. Tags: comp, prog, pldesign, lash, life, mind, ramble, music

Coloquei o lash no GitHub, for what it's worth. Eu me pergunto se foi uma coisa sensata publicar ele agora, mas já faz um tempo que eu vinha anunciando que ia publicar "em breve", então coloquei lá de uma vez. (Além disso, esses dias eu quis mexer nele fora de casa e não tinha o código.) O código está em um estágio bem inicial – vergonhosamente inicial, dado que já faz umas três semanas que comecei a trabalhar nele, e o que eu fiz até agora eu provavelmente poderia ter feito em uns três dias se tivesse tido a disciplina de trabalhar nele semi-diariamente. Por outro lado, o projeto está andando para frente, mesmo que devagar, o que já é melhor do que todos os anos anteriores em que eu disse "puxa, eu queria fazer um shell" e não escrevi uma linha de código. So, that's progress. Além disso, na atual conjuntura eu provavelmente deveria tentar relaxar um pouco a cuca e me preocupar menos com isso; afinal, isso é um projeto pessoal e eu não devo nada para ninguém. No final das contas, megalomanias de dominação mundial à parte, o principal afetado pelo bom sucesso do projeto sou eu mesmo.

Eu cheguei à brilhante e inaudita conclusão de que eu vou ter que reduzir bastante minha atividade twittereira e internética em geral se eu quiser começar a fazer alguma coisa produtiva com a minha vida (onde escrever um shell e estudar línguas obscuras contam como coisas produtivas). Por outro lado, a Internet atualmente é responsável por uns 95% das minhas interações sociais, especialmente agora que eu não tenho mais aulas, coisa que faz uma certa falta, a despeito da minha fama de anti-social. A solução provavelmente é (shudder) sair de casa e falar com pessoas.

Eu também cheguei à conclusão (igualmente brilhante) de que muita coisa nessa vida é questão de criar hábitos. Por exemplo, até algumas semanas atrás eu costumava usar toda a louça da casa até não ter mais louça limpa, momento em que eu aplicava o garbage collector e lavava tudo (ou, dependendo da preguiça, só o que eu precisasse na hora). Eu me dei conta de que isso não estava sendo muito conveniente e resolvi começar a lavar as coisas logo depois que uso, ou antes de ir dormir. No começo era meio ruim ter que me "obrigar" a fazer isso, mas agora eu já me habituei e isso não me incomoda mais tanto (além do que, como a louça não acumula, normalmente o esforço de lavar é pequeno). Talvez seja uma questão de criar o hábito de sentar uma hora do dia para programar/estudar/whatever. O flip side disso é que a gente também se habitua ao longo da vida a uma porção de coisas que a gente deveria questionar e/ou atirar pela janela, não só hábitos acionais como também (e principalmente) hábitos mentais. Estas são minhas (brilhantes e inauditas) palavras de sabedoria do dia. (Tecnicamente todo hábito é mental, mas deu pra entender. Acho.)

Escrever o lash em Chicken Scheme tem sido uma experiência bastante agradável. Eu estou aprendendo (a parte não-R5RS d)a linguagem "as I go", mas até agora a linguagem não me deixou na mão, a implementação é estável e gera executáveis pequenos e razoavelmente rápidos, e o sistema de pacotes funciona. (Rodar chicken-install pacote e consistentemente ver o pacote ser baixado e compilado sem erros era quase chocante no começo. O fato de que as bibliotecas são shared objects (a.k.a. DLLs) de verdade e carregam instantaneamente também muito alegrou o espírito, especialmente dada minha experiência anterior com bibliotecas em Common Lisp.) A única coisa que deixa um pouco a desejar é o error reporting, mas nada "deal-breaking".

Eu me dei conta de que uma das coisas que eu mais gosto em linguagens "dinâmicas" é a habilidade de rodar um programa incompleto. Eu já meio que escrevi sobre isso antes, mas eu já não lembrava mais quão deeply satisfying é poder rodar um programa pela metade e ver a parte que foi escrita até o momento funcionando. Por outro lado, é bastante incômodo errar um nome de função ou os argumentos e só descobrir o erro em tempo de execução. Faz muito tempo que eu acho que o ideal seria uma linguagem com análise estática de tipos, mas em que erros de tipo gerassem warnings ao invés de impedir a compilação, e que permitisse a declaração opcional dos tipos de variáveis e funções. Uma dificuldade que eu via nisso até agora é que enquanto em uma linguagem dinâmica os dados costumam ter uma representação uniforme em memória que carrega consigo alguma tag indicando o tipo do dado, e portanto é possível chamar uma função com argumentos do tipo errado e detectar isso em tempo de execução, em uma linguagem estaticamente tipada convencional os dados costumam ter uma representação untagged/unboxed e de tamanho variável, o que tornaria impossível compilar um programa com um erro de tipo sem violar a segurança da linguagem (e.g., se uma função f recebe um vetor e eu a chamo com um int, ou eu rejeito o programa em tempo de compilação, o que atrapalha minha habilidade de rodar programas incompletos/incorretos, ou eu gero um programa que interpreta o meu int como um vetor sem que isso seja detectado em tempo de execução, o que provavelmente vai causar um segfault ou algo pior). Porém, esses dias eu me dei conta de que ao invés de compilar uma chamada (insegura) a f(some_int), poder-se-ia simplesmente (além de gerar o warning) compilar uma chamada a error(f, some_int), onde error é uma função que avalia os argumentos e lança uma exceção descrevendo o erro de tipo. O resultado prático é que o executável gerado roda até o ponto em que é seguro rodar (inclusive avaliando a função e os argumentos) e interrompe a execução no ponto em que seria necessário chamar a função com um argumento de tipo/representação incompatível. Melhor dos dois mundos, não? Vai para o meu caderninho de idéias para a Linguagem Perfeita™.

Eu ia escrever mais umas notas sobre a vida, mas ultimamente eu ando mui receoso quanto a publicar coisas da minha vida pessoal – o que provavelmente é uma boa idéia. É fácil esquecer que qualquer um, no presente ou no futuro, pode ler o que a gente escreve nessa tal de Internet. Eu também já falei sobre isso mil vezes antes, which makes it all the more surprising que eu ainda tenha que me relembrar disso ocasionalmente. Anyway.

Unrelated com qualquer coisa, conheci recentemente uma bandinha chamada Clannad, na qual eu me encontro totalmente viciado no momento. Também conheci uma coisa totalmente excelente chamada Galandum Galundaina, uma banda mirandesa com certeza.

12 comentários

Main menu

Posts recentes

Comentários recentes

Tags

comp (114) prog (51) life (44) unix (32) random (27) lang (27) about (24) mind (22) mundane (21) pldesign (20) in-english (19) lisp (17) web (17) ramble (15) img (13) rant (12) privacy (10) scheme (8) freedom (8) lash (7) music (7) esperanto (7) bash (7) academia (7) home (6) mestrado (6) shell (6) conlang (5) copyright (5) misc (5) worldly (4) book (4) php (4) latex (4) editor (4) politics (4) etymology (3) wrong (3) android (3) film (3) tour-de-scheme (3) kbd (3) c (3) security (3) emacs (3) network (3) poem (2) cook (2) physics (2) comic (2) llvm (2) treta (2) lows (2) audio (1) wm (1) philosophy (1) kindle (1) pointless (1) perl (1)

Elsewhere

Quod vide


Copyright © 2010-2018 Vítor De Araújo
O conteúdo deste blog, a menos que de outra forma especificado, pode ser utilizado segundo os termos da licença Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

Powered by Blognir.