Hi @spy16, I hope you're well and that things are returning to normal on your end.
As mentioned over in https://github.com/spy16/sabre/pull/26 and https://github.com/spy16/sabre/pull/27, I've been making heavy use of Sabre over the past few weeks in the context of Wetware, so I thought I'd share my thoughts on what works and what can be improved.
I'm well aware that the current runtime design has shown some limitations, and comfortable with the fact that parts of Wetware will have to be rewritten once we iron out the creases. My goal in publishing this is to:
- fill you in on the design requirements for Wetware (as per your comment over at https://github.com/spy16/sabre/pull/26#issuecomment-675853479).
- follow up on https://github.com/spy16/sabre/pull/26 and hopefully start working on the Clojure-style runtime. I've made room in my schedule to lend a hand, but am looking to you for general direction. 😃
This report is structured as follows:
- Context: A detailed description of Wetware, and how Sabre fits into the picture. It's a bit lenghty because I wanted to err on the side of completeness. Please forgive me as I slip into pitch-mode from time to time :sweat_smile:!
- The Good Parts of working with Sabre. This section highlights where Sabre provides a net gain in productivity, and was generally a delight to use.
- Pain Points, Papercuts and Suggestions. This section highlights where Sabre could be improved. I've tried to be as specific as possible, and to link to Wetware source code where appropriate. Possible solutions to these issues are also discussed.
- Miscellanea that are generally on my mind. These are basically issues that I expect to encounter in the mid-to-near term. They are important, but not urgent.
Context
Wetware is a distributed programming language for the cloud. Think Mesos + Lisp. Or Kubernetes + Lisp, if you prefer.
Wetware abstracts your datacenters and cloud environments into a single virtual cloud, and provides you with a simple yet powerful API for building fault-tolerant and elastic systems at scale.
It achieves its goals by layering three core technologies:
1. A Cloud Management Protocol
At its core, Wetware is powered by a simple peer-to-peer protocol that allows hosts to discover each other over the network, and assembles them into a fully-featured virtual cloud.
This virtual cloud is self-healing (i.e. antifragile), truly distributed (with no single point of failure), and comes with out-of-the box support for essential cloud services, including:
- Elastic process management using Virtual Machines (like Amazon EC2), Containers (like Amazon ECS), bare-metal UNIX processes, and even ultra-lightweight threads (goroutines).
- Persistent storage with support for binary blobs (like Amazon S3) and even structured data (maps, lists, strings, sets, etc.).
- Interprocess communication and orchestration via distributed PubSub (like Amazon SQS) and Channels (i.e. network-aware queues).
Wetware's Cloud Management Protocol works out-of-the-box, requires zero configuration, and features first-class support for hybrid and multicloud architectures.
2. A Distributed Data Plane
Unifying data across applications is a major challenge for current cloud architectures. Developers have to deal with dozens (sometimes even hundreds) of independent applications, each producing, encoding and serializing data in its own way. In traditional clouds, ETL and other data operations are time-consuming, error-prone and often require specialized stacks.
Wetware solves this problem by providing
- High-performance, immutable datastructures for representing data across applications,
- High-throughput protocols for efficiently sharing large datastructures across the network, and
- An intuitive API for working concurrently with data in a cluster-wide, shared-memory space.
With Wetware's dataplane, you can coordinate millions of concurrent processes to work on terabyte-sized maps, sets, lists, etc. These immutable and wire-native datastructures protect you from concurrency bugs while avoiding overhead due to (de)serialization.
Lastly, Wetware's location-aware caching means you're always fetching data from the nearest source, avoiding egress costs in hybrid and multicloud environments.
3. A Dynamic Programming Language
The Wetware REPL is the primary means through which users interact with their virtual cloud, and the applications running on top of it. Unsurprisingly, this REPL is a Lisp dialect built with Sabre.
Let's walk through a few examples.
We can simulate a datacenter from the comfort of our laptop by starting any number of Wetware host processes:
# Start a host daemon.
#
# In production, you would run this command once on each
# datacenter host or cloud instance.
#
# In development, you can simulate a datacenter/cloud of
# n hosts by running this command n times.
$ ww start
Next, we start the Wetware REPL and instruct it to dial into the cluster we created above.
# The -dial flag auto-discovers and connects to an
# arbitrary host process. By default, the shell runs
# locally, without connecting to a virtual cloud.
$ ww shell -dial
We're greeted with an interactive shell that looks like this:
Wetware v0.0.0
Copyright 2020 The Wetware Project
Compiled with go1.15 for darwin
ww »
From here, we can list the hosts in the cloud. If new hosts appear, or if existing hosts fail, these changes to the cluster will be reflected in subsequent calls to ls.
ww » (ls /) ;; list all hosts in the cluster
[
/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj
/cie5uM1dAuQcbTEpHi4GKsghNVBk6H4orjVs6fmd16vV
]
The ls command returns a core.Vector, which contains a special, Wetware-specific data type: core.Path. These paths point to special locations called ww.Anchor. Anchors are cluster-wide, shared-memory locations. Any Wetware process can read or write to an Anchor, and the Wetware language provides synchronization primitives to deal with the hazards of concurrency and shared memory.
Anchors are organized hierarchically. The root anchor / represents the whole cluster, and its children represent physical hosts. Children of hosts are created dynamically upon access, and can contain any Wetware datatype.
;; Anchors are created transparently on access.
;; You can retrieve the value stored in an Anchor
;; by invoking its Path without arguments.
;; Anchors are empty by default.
ww » (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo)
nil
;; Invoking a Path with a single argument stores
;; the value in the corresponding Anchor.
ww » (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo
› {:foo "foo value"
› :bar 42
› :baz ["hello" "world"] })
nil
;; The stored value is accessible by _any_ Wetware
;; process in the cluster.
;;
;; Let's fetch it from a goroutine running in the
;; remote host `cie5uM1...`
ww » (go /cie5uM1dAuQcbTEpHi4GKsghNVBk6H4orjVs6fmd16vV)
› (print (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo)))
nil
Why did this print nil? Because the form (print (/SV4e8.../foo)) was executed on the remote host cie5uM...! That is, the following things happened:
- A network connection to
cie5uM... was opened.
- The list corresponding to the
print function call was sent over the wire.
- On the other side,
cie5uM... received the list and evaluated it.
- During evaluation,
cie5uM... fetched the value from the Sv4e8.../foo Anchor and printed it.
If we were to check cie5uM...'s logs, we would see the corresponding output.
Important Technical Note: Wetware's datastructures are implemented using the Cap'n Proto schema language, meaning their in-memory representations do not need to be serialized in order to be sent across the network.
Our heavy reliance on capnp has implications for the design of varous Sabre interfaces, as discussed in part 2.
This concludes general introduction to Wetware.
While Wetware is very much in a pre-alpha stage, the foundational code for features 1 - 3 are in place, and the overall design has been validated. Now that we are leaving the proof-of-concept stage, developing the language (and its standard library) will be the focus of the next few months. For this reason, Sabre will continue play a central role in near-term development and I expect to split my development time roughly equally between Wetware and Sabre. As such, I'm hoping the following feedback can serve as a synchronization point between us, and motivate the next few PRs.
The Good Parts
(N.B.: I am exclusively developing on the reader branch, which is itself a branch of runtime.)
Overall, Sabre succeeds in its mission to be an "80% Lisp". The pieces fit together quite well, and most things are easily configurable. This last bit is particularly true of the runtime branch where I was able to write custom implementations for each atom/collection, as well as create some new, specialized datatypes. I have not encountered any fundamental design flaws, which is great!!
The REPL is a breeze to use, requring little effort to set up and configure. This is in large part thanks to your decision to make REPL (and Reader for that matter) concrete structs that hold interfaces internally, as opposed to declaring them as interface types. Doing so allows us to inject dependencies via functional arguments rather than re-writing a whole new implementation just to make minor changes to behavior. The result is a REPL that took me less time to set up than to write this paragraph, so this is a pattern we should definitely continue to exploit.
Relatedly, I think these few lines of code really showcase the ergonomics of functional options. They compose well, are discoverable & extensible, and visually cue the reader to the fact that the repl.New constructor is holding everything in the package together. I'm disproportionately pleased with the outcome.
Lastly, the built-in datatypes are very useful when developing one's own language because they serve as simple stubs until custom datastructures have been developed. In practice, this means I was able to develop other parts of the language in spite of the fact that e.g. Vectors had not yet been implemented in Wetware. It's hard to overstate not only how incredibly useful this is, and how much of that usefulness stems from the fact that Sabre is using native Go datastructures under the hood. Designing one's own language is quite hard, so every ounce of simplicity and familiarity is a godsend. I am strongly in favor of maintaining the existing implementations and not adding persistent datatypes for this reason. An exception might be made for LinkedList since the current implementation is dead-simple and shoe-horning a linked-list into a []runtime.Value is a bit ... backwards. In any case, Sabre really came through for me, here.
Pain Points, Papercuts & Suggestions
I want to stress that this section is longer than its predecessor not because there are more downsides than upsides in Sabre, but because there's always more to say about problems than non-problems! With that said, I've sorted the pain-points I've encountered into a few broad buckets:
- Error handling
- Design of container types
- Reader design
Error Handling
By far the biggest issue I encountered was the handling of errors inside datastructure methods. Throughout our design discussion in #25, our thinking was (understandably) anchored to the existing implementations for Map, Vector, etc. Specifically, we assumed that certain operations (e.g. Count() int) could not result in errors. This turns out to have been an incorrect assumption.
As mentioned in the Context section above, Wetware's core datastructures are generated from a Cap'n Proto schema. As such, simple things such as calling an accessor function often return errors, including for methods like core.Vector.Count(). The result is that my code is quite panicky: Count, Conj, First and Next all panic.
While there are (quite convoluted) ways of avoiding these panics, I think there's a strong argument for changing the method signatures to return errors. Sabre is intended as a general-purpose build-your-own-lisp toolkit, and predicting what users will do with it is nigh impossible. For example, they may write datastructures implemented by SQL tables, which make RPC calls, or which interact with all manner of exotic code. As such, I think we should take the most general approach, which means returning errors almost everywhere.
Design of Container Types
This issue is pretty straightforward. I'd like to implement an analog to Clojure's conj that works on arbitrary containers. Currently, runtime.Vector.Conj returns a Vector, so I'm wondering how this might work. Do you think it's best to resort to reflection in such cases? Might it not be better to return runtime.Value from all Conj methods?
Reader Design
Despite being generally well-designed, there is room for improvement in reader.Reader.
Firstly, https://github.com/spy16/sabre/pull/27 adds the ability to modify the table of predefined symbols, which was essential in my case as I have custom implementations for Nil and Bool.
Secondly, relying on Reader.Container to build containers is not appropriate for all situations. The Container method reads a stream of values into a []runtime.Value, and returns it for further processing. In the case of Wetware's core.Vector, this is quite inefficient since:
- I need to allocate a
[]runtime.Value.
- I might need to grow the
[]runtime.Value, causing additional allocs, but I can't predict the size of the container ahead of time.
- Once the
[]runtime.Value is instantiated, I have to loop through it and call core.VectorBuilder.Conj, which also allocates.
In order to avoid the penalty of double-allocation, I wrote readContainerStream, which applies a function to each value as it is decoded by the reader. The performance improvement is significant for large vectors, so I think we should add it as a public method to reader.Reader.
Thirdly, Wetware's reliance on Cap'n Proto means that I must implement custom numeric types. To make matters more complicated, I would like to add additional numeric types analogous to Go's big.Int, big.Float, and big.Rat. As such, I will need the ability to configure the reader's parsing logic for numerical values.
Currently, numerical parsing is hard-coded into the Reader. I suggest adding a reader option called WithNumReader (or perhaps WithNumMacro?) that allows users to configure this bit of logic. I expect this will also have repercussions on sabre.ValueOf, but it should be noted that this function is already outdated with respect to the new runtime datastructure interfaces.
Miscellanea
Lastly, a few notes/questions that are on my mind, but not particularly urgent:
- The
Position type seems very useful, but I'm not sure how it's meant to be used. Who is responsible for keeping it up-to-date, exactly? Any "use it this way" notes you might have would be helpful.
- I don't quite understand the distinction between
GoFunc, Fn and MultiFn. Best I can figure, GoFunc is used to call a native Go function from Sabre, while (Multi)Fn is meant to be dynamically instantiated by defn? From there, I assume MultiFn is used for multi-arity defn forms? (I think I might have answered my own question :smile:)
- I expect to start thinking about Macros in 6-8 weeks or so. Are there any major changes planned for the macro system, or can I rely on what already exists?
- I'm going to tackle goroutine invokation within the next 2-3 weeks and will keep you appraised of my progress in #15. If you have any thoughts on the subject, I'm very much interested.
Conclusion
I hope you find it as useful to read this experience report as I have found it useful to write. I'm eager to discuss all of this at your earliest convenience, and standing by to help with implementation! :slightly_smiling_face:
