Thursday, June 5, 2014

automatic synchronous to async transform


Asynchronous programming in JavaScript can quickly become a mess of callbacks, sometimes it is called `callback hell`. This hell gets worse when you need to move some logic to WebWorkers and pass data back and forth. Your once simple synchronous function must be rewritten into many callbacks that shuffle data around using postMessage and onmessage, and trigger the next callback.

PythonJS allows you to code in a synchronous style, and when your code it translated to JavaScript, it will also be transformed into async callbacks. You can call the sleep function to stop a function for a set amount of time while other things take place, the function will resume after the timeout.

These commits: [1], [2], [3], allow the webworker to directly call functions in the main thread, passing it data as normal function arguments. Under the hood, PythonJS will call postMessage with the function name and arguments. In the main thread this triggers the requested function with the result sent back to the worker. The function in the worker halts until it gets this response from the main thread. What ends up being alot of async code, can be expressed in just a few lines of sync code.

python input

import threading
from time import sleep

shared = []

def blocking_func(x,y):
 shared.append( x )
 shared.append( y )
 return x+y

def async_func( a ):
 shared.append( a )

def main():
 w = threading.start_webworker( worker, [] )
 sleep(1.0)
 assert len(shared)==3
 assert shared[0]==10
 assert shared[1]==20
 assert shared[2]==30
 print('main exit')

## marks this block of code as within the webworker
with webworker:

 def worker():
  ## blocks because the result is assigned to `v`
  v = blocking_func( 10, 20 )
  ## non-blocking because result is not used
  async_func( v )
  self.terminate()

javascript output - main

shared = [];

blocking_func = function(x, y) {
  shared.append(x);
  shared.append(y);
  var __left16, __right17;
  __left16 = x;
  __right17 = y;
  return ((( typeof(__left16) ) == "number") ? (__left16 + __right17) : __add_op(__left16, __right17));
}

async_func = function(a) {
  shared.append(a);
}

main = function() {
  var w;
  w = __start_new_thread(worker, []);
  __run__ = true;
  var __callback0 = function() {
    __run__ = true;
    var __callback1 = function() {
      console.log("main exit");
    }

    if (__run__) {
      setTimeout(__callback1, 1000.0);
    } else {
      if (__continue__) {
        setTimeout(__callback2, 1000.0);
      }
    }
  }

  if (__run__) {
    setTimeout(__callback0, 200.0);
  } else {
    if (__continue__) {
      setTimeout(__callback1, 200.0);
    }
  }
}
worker = "/tmp/worker.js";

javascript output - webworker


onmessage = function(e) {
  if (( e.data.type ) == "execute") {
    worker.apply(self, e.data.args);
    if (! (threading._blocking_callback)) {
      self.postMessage({ "type":"terminate" });
    }
  } else {
    if (( e.data.type ) == "append") {
      __wargs__[e.data.argindex].push(e.data.value);
    } else {
      if (( e.data.type ) == "__setitem__") {
        __wargs__[e.data.argindex][e.data.key] = e.data.value;
      } else {
        if (( e.data.type ) == "return_to_blocking_callback") {
          threading._blocking_callback(e.data.result);
        }
      }
    }
  }
}
self.onmessage = onmessage;

worker = function() {
  var v;
  v = self.postMessage({
    "type":"call",
    "function":"blocking_func",
    "args":[10, 20] 
  });
  var __blocking = function(v) {
    self.postMessage({
       "type":"call",
        "function":"async_func",
        "args":[v] 
    });
    self.postMessage({ "type":"terminate" });
    threading._blocking_callback = null;
  }

  threading._blocking_callback = __blocking;
}

3 comments:

  1. Interesting !
    Great to see, that running a webworker task is as simple as this using PythonJS.
    I'm also glad to see that you're working on a way to transpile PythonJS sync-like code to asynchronous javascript.

    That said, I'm not sure that using webworkers is a good idea to tackle the async problem. If I'm referring to client side development the most used async concept are xmlhttprequests for rpc to backend and settimeout for animations. Starting a web worker each time for an rpc call or an animation seems a bit heavy, and I can imagine the overhead of serialisation/deserialisation that would occur each time the webworker has to send the rpc response back as a message to the main thread.

    ReplyDelete
  2. Your correct, Webworkers are not the best general solution, it was just my starting point for this research. I am adapting the translator now to work with xmlhttprequests and rpc logic so that a webworker is not required.

    ReplyDelete
  3. I just read other articles from you and just saw that you're working on a webgl game using PythonJS. This is cool ! in fact I have a bigger interest in that subject than web development (even tough I still have to learn webgl). In this context I fully understand the necessity of webworkers !

    ReplyDelete