[Larceny-users] Heaps in err5rs: giant heaps, errors loading, extra slow compile with -stopcopy

[William D Clinger]

Ryan Newton wrote:
> I've got a large R6RS project.  I'm having problems with compile-
> stale and also with trying to build my libraries incrementally using
> explicit dependencies represented in a makefile.  (I can elaborate on
> that if anyone is curious.)

Using compile-stale on Linux, I am able to reproduce
the "Client was expanded against a different build of
this library" bug but not the "unable to open file"
bug.  I can't seem to reproduce either bug under
Solaris, which is of course a clue.

> I *could* load my project all at once from source (in about 5 min).

That's about 260 lines per second, which is in line
with the compilation speed we're getting on our Linux
machine.

> They wouldn't give me incremental builds, I was thinking that heaps
> might save me.  Load from source (5 min), dump heap, and then have a
> very fast startup and execution time.

That should have been a reasonable workaround until
we can fix and then improve compile-stale.

> Alas, I'm running into some problems.  First, adding -stopcopy causes
> my system to take not five minutes to load but 43!!  (It takes 43 min
> in either -err5rs or -r6rs mode.)  When it's done it saves a 73
> megabyte heap file.  (Chez saves a 10mb heap file.)

Something must be going wrong here.  Quite possibly
there are at least two different things going wrong,
one to cause the poor load time and the other to
cause the bloated heap file.

At the moment, I can only speculate.

Some possibilities for the poor load time:

    bug in compiler (such as using eq? hash tables,
        which are much slower with the stop-and-copy
        collector than with the generational)

    bug in Larceny's stop-and-copy garbage collector
        (unlikely)

    stop-and-copy collection really is that much
        slower when Twobit is compiling a large
        program

Some possibilities for the bloated heap file:

    bug causing the compiler to preserve source code
        or variable names

    grotesquely inefficient code generation

    R6RS bloat (but that should be independent of the collector)

> Neither of those problems are necessarily total show stoppers.  But
> then I get an error on load:
>
> [newton at honor ~/wavescript/src]  $ larceny -heap larc.heap
> Larceny v0.961 "Fluoridation" (May  8 2008 16:17:32,
> precise:Linux:unified)
> Larceny Panic: Sys/alloc.c;368: Assertion failed.
> Aborted
>
> When I try with smaller examples, not loading so much stuff, then
> loading from heap works.  But it dies with the 73mb heap file.
>
> Looking at the code, here is the assert line:
>
>    assert( dest < slots &&
>            dest+data.descriptor_slots-1 < slots );

Ouch!  We haven't seen any trouble with heaps that
grow as large as a gigabyte, but we don't have any
experience with loading such large heap files and
there might be some interaction with heap loading.
Another possibility is that the heap dumper might
have a bug that causes it to create a malformed heap
file.

It would be interesting to know whether you get the
same error with

    $ larceny -stopcopy -heap larc.heap

Will