So... the server would maintain a set of grammar-of-graphics-like objects, which would be replicated as backbone models? In theory we could use whichever graphics libraries we want on top the backbone layer, right?

Bokeh is starting to sound like everything I've ever wanted.