Compiler CLI and Scalability #131

This document proposes to replace/(enhance) the racks tool currently used to compile RacketScript programs.

Motivation

racks is slow

• It compiles each file serially, which is particularly frustrating during first compilation.
• We did some work to avoid recompiling unchanged modules, but it still not foolproof and may require full recompile under certain circumstances. Eg. If a module with macro changes, any other module using that macro must be recompiled.
• The identifier graph used to follow the bindings is recomputed in each run, which takes noticeable number of seconds. This graph can potentially be cached, and only affected modules should be processed to update this graph.

racks is impractical

It simply compiles what it gets. It has no notion of package.json, info.rkt which is often used in practical applications. It overwrites package.json with the default one.

Some common examples:

• User can't specify JS dependencies at all.
• No control over how to produce the distribution JavaScript file, and limited to what we allow to use e.g. Gulp, Babel, Minifiers ...
• Plain JavaScript files has to be copied manually to distribution directory.

Proposal

racks should be replaced with a tool that more or less is like what raco is to Racket or lein is to ClojureScript. Here's a rough workflow I see:

• Create a new project:

    racks init my-new-project
  

• Install all dependencies:

    racks install
  

• Create a new project using some a template project structure. However, unlike current situation, this is used just to create project. The config can be edited to whatever.

    racks init my-new-project -t babel-webpack
  

• Builds the project:

    racks build
  

• Watch the project and build automatically:

    racks build --watch
  

• Start a REPL

    racks repl
  

All these command could be simple raco extensions, like raco racketscript <subcommand>, and racks can just be alias for raco racketscript or something like that.

Secondly, compilation of each module should be independent of other module as much as possible. There is only much we can do here as mentioned earlier, but there is still scope.

info.rkt can act as combined project configuration, i.e. for both Racket and JavaScript. Since, package.json is simply a JSON, it can just be a S-expression in info.rkt. The project config can also specify the main entry point, any additional JavaScript/static files that needs to copied, or pretty much anything user may want to do pre/post operations.

Conclusion

The documents proposes much more than what we have currently have time and resources for. However I believe it is certainly something we should start thinking about. I will keep on updating this document based on later discussions.

This shouldn't require any changes to existing compiler or runtime, and can be built as a new script. So, while we slowly work on it, it shouldn't block us with any other tasks in hand.