Back to all discussions

  •   over 9 years ago

Adding Clojure support

I and a teammate (@rajatkhanduja) are going to try and create a Clojure parser in JavaScript. We're starting out by reading up on the Mozilla Parser API and Acorn, followed by a look at how the ClojureScript compiler works. Let's see how this pans out :)

  • 47 comments

  • Manager   •   over 9 years ago

    Awesome, Clojure will be great! Good luck!

  •   •   over 9 years ago

    Hey, Nick, I have a query regarding Aether. So I'm planning to start out by implementing a Mozilla AST-compatible parser for the bare minimum of Clojure syntax. My question is: how do I plug it in to Aether? Is there a config file where I can register my parser, and everything else will be taken care of? Thanks.

  • Manager   •   over 9 years ago

    We don't have very good config/plugins/modules for different languages yet, having just added the second. You can see here part of how the CoffeeScript Redux mode was added: https://github.com/codecombat/aether/blob/master/src/aether.coffee#L274-L279

    We'll be cleaning that up in the coming weeks as we want to plug in more languages.

  •   •   over 9 years ago

    Thanks for that info. I'll continue working on my parser without worrying about integrating it just yet, then. Would you please post an update here on ChallengePost (or on the GitHub Wiki), once the parser framework is ready? That would help. I could even help out with the framework documentation. Thanks again!

  • Manager   •   over 9 years ago

    Will do–just watch the Aether repository on GitHub for any important updates.

  •   •   over 9 years ago

    Hi Nick,

    I am considering leveraging Jison (a Bison clone in JS): http://zaach.github.io/jison/docs/ to write the parser. Do you think it makes sense to use it? From a cursory glance it seems to me that such a parser would be easier to write and more maintainable in the long-run.

    A number of well-known projects use Jison already, including CoffeeScript, handlebars.js, Roy, and jsonlint (a comprehensive list is available here: https://github.com/zaach/jison/wiki/ProjectsUsingJison).

  • Manager   •   over 9 years ago

    Hmm, it might work–I'm not sure. It will just have to be able to preserve the original source range information and be able to create a Mozilla-compatible AST out of the Jison parser output.

  •   •   over 9 years ago

    It should work, as there's already a Mozilla API-compatible JavaScript parser written using Jison: https://github.com/zaach/reflect.js :)

  •   •   over 9 years ago

    Nick, I have a concern regarding code style for CodeCombat solutions in Clojure. Ideally, a JS call like this.moveXY(47, 60) would translate to (moveXY 47 60) in idiomatic Clojure (or better yet, (move-xy 47 60), following Clojure naming conventions). Do you have an idea for how this might work?

    Maybe, for every language we can have a mapping from idiomatic identifiers in that language to actual JavaScript identifiers. In the case of Clojure this might be something like, { 'move-xy' : 'this.moveXY' }, and then Aether can replace instances of 'move-xy' with 'this.moveXY' in the transpiled code. I thought I should point this out as you guys work on the parser integration framework, it might help inform some decisions. Thoughts?

  • Manager   •   over 9 years ago

    We still need to be able to allow OOP, because you might need to call these methods on something other than `this`:

    this.moveXY(this.enemy.pos.x, this.enemy.pos.y);
    var panicDistance = this.enemy.distance(base);

    I have been reading a bit about Clojure just now, but it's still unclear to me the best way to do this. Thoughts?

    It would be easy to convert JavaScript mixed camelCase identifiers to Clojure-style with Underscore String's dasherize method:

    _.string.dasherize('moveXY'); // "move-x-y"
    _.string.camelize('move-x-y'); // "moveXY"

  •   •   over 9 years ago

    Hmm, you're right, I hadn't thought about that. There are a few alternatives:

    #1. Usual way to do OOP:

    (let [pos (.pos (.enemy this))] // var pos = this.enemy.pos
    (.move-x-y this (.x pos) (.y pos))) // this.moveXY(pos.x, pos.y)

    Note the period at the beginning of each property name.

    #2. Alternative OOP syntax using .. macro (see http://blog.simplefluous.com/2010/03/quick-clojure-understanding-dot-dot-macro.html):

    (let [pos (.. this enemy pos)]
    (.. this (move-x-y (.. pos x) (.. pos y))))

    But explaining to beginners how .. works would be a challenge.

    #3. Treat JS objects as Clojure hash-maps:

    (let [pos ((this :enemy) :pos)]
    ((this :move-x-y) (pos :x) (pos :y)))

    #4. Try to create our own light-weight OOP syntax:

    (let [pos (this.enemy.pos)]
    (this.move-x-y (pos.x) (pos.y)))

    This will probably be simpler for beginners to understand, but might annoy Clojure veterans.

  • Manager   •   over 9 years ago

    Hmm, I don't think I'm qualified to evaluate this, not being a Clojure expert. From an outsider's view, I kind of like #3.

  •   •   over 9 years ago

    Hi Nick, any updates on the integration framework for parsers? I've been watching the Aether repository, but haven't gotten any updates through that channel.

    I'm at the point where most of the parser and core library is in place, so I want to proceed with actually testing it on CodeCombat and resolve any issues that may arise. :)

  • Manager   •   over 9 years ago

    Wow, almost ready for testing? Awesome! I'll start working on a plugin system for using alternate-language parsers this week as part of a big Aether refactoring that's long overdue.

  • Manager   •   over 9 years ago

    I've just pushed Aether 0.2.0, which has a standardized language module interface. You can see the Clojure stub here:

    https://github.com/codecombat/aether/blob/master/src/languages/clojure.coffee

    The module interface is here:

    https://github.com/codecombat/aether/blob/master/src/languages/language.coffee

    Basically, you'll want to implement the parse() method (and probably the wrap() method) at the minimum to hook into your parser. Let me know if you have any questions or want some help getting it integrated.

  •   •   over 9 years ago

    Great, thanks! I'll work on the integration over the weekend, and will let you know if I need any help.

  •   •   over 9 years ago

    I've got a fair idea of how to implement the parse() method. But the core library is in a separate module, and that module needs to be loaded at the point where the transpiled code is executed. Also, we need to detect which function calls from the user's code refer to functions from the core library, and accordingly re-wire them (func(args) -> clojure.core.func(args)). How do we achieve that?

    For my testing so far I was using estraverse to go over all the identifiers in the AST, and if any of them is in the core library, replace the node with a corresponding MemberExpression.

  • Manager   •   over 9 years ago

    I can add a runtime-library-inclusion step to Aether. Can you give me either a small example or a link to the runtime library so I can try it out as I go?

    I think the way that you're currently replacing AST MemberExpressions will be good for hooking up the core library–that can be part of the parse() step. We'll just make sure that "clojure" is in the global scope for the method.

  •   •   over 9 years ago

    There seems to be some confusion here. The core library needs to be in scope when the code is *executed*, not when we're traversing the AST to hook it up in the parse() step -- that part doesn't actually use the library. See https://github.com/vickychijwani/closer.js/blob/master/src/repl.coffee#L6-L16, for example.

    I made a command-line based REPL for my implementation: https://github.com/vickychijwani/closer.js/blob/master/src/start-repl.coffee. It doesn't do much - just generates JS code using escodegen and executes it in a sandbox (lines 19-21). See line #7 that assigns to global.core -- that is needed because the core library needs to be available in the sandbox at execution time (the "sandbox" being the global scope of the module), but the parse step doesn't need it at all.

    I hope that gets the point across.

  •   •   over 9 years ago

    Correction: the parse() step also needs the library -- to figure out if the current Identifier node refers to one of the library functions (https://github.com/vickychijwani/closer.js/blob/master/src/repl.coffee#L9). But the question remains: how do we ensure the library is in scope when executing the transpiled code? (as far as I understand the code execution occurs outside of Aether, is that correct?)

  • Manager   •   over 9 years ago

    The simple, janky thing to do is to prepend the library code to the transpiled user code. If we get it working that way, we can figure out something more performant as a next step, like having the parser export a runtime and then the consumer depend on both Aether and that runtime. Regenerator does something like that: http://facebook.github.io/regenerator/

  •   •   over 9 years ago

    Ok, that should work as a first step. I'll try it out.

  •   •   over 9 years ago

    Good news: I was able to complete a few levels in Clojure! :)

    Specifically: Rescue Mission, Cowardly Taunt, Emphasis on Aim. I've pushed these changes to my fork of Aether (diff here: https://github.com/vickychijwani/aether/compare/clojure).

    There are two broad categories of todo items to tackle next, as I see it:

    1. Complete all the levels
    2. Refactor the integration code, right now it is really ugly

    I'm working on #1 right now.

    For #2, we'll have to work together because the current language framework doesn't support easy integration of runtime libraries. As of now I'm using fs.readFileSync and BRFS (a Browserify transform) to read the entire library and prepending it to the transpiled code, as you suggested (https://github.com/vickychijwani/aether/compare/clojure#diff-b67fa2dfedc85f4d26e45beaf0e6748eR223).

    I'll keep you posted on my progress with the levels. Let me know your thoughts on #2.

  •   •   over 9 years ago

    Also, I'm facing an issue running CodeCombat locally. Often, while loading a level, the browser just hangs and I have to close / kill the tab and try again. I inspected the network calls made by the browser, and it seems that it never receives a response when requesting aether or lodash. Is this a known issue? I'm running Chromium 34 on Linux.

    Another query I have is: at what point does the transpiled code get executed on the user's browser? I want to be able to set breakpoints in it. Thanks!

  •   •   over 9 years ago

    Some of the levels seem to have uneditable JavaScript code hard-wired into their solutions, causing the parser to throw errors. For instance, Molotov Medic has some read-only JS for the dying soldiers. Similarly for Zone of Danger and Break the Prison. How do I get around this?

Comments are closed.