067: ember-concurrency with Alex Matchneer

Alex Matchneer: @machty | FutureProof Retail

Show Notes:

• 01:07 - The Introduction of ember-concurrency
• 02:15 - What is ember-concurrency? What are the problems it solves?
• 05:37 - Why not use observables or other alternatives?
• 09:49 - Could observables be used in conjunction with ember-concurrency?
• 12:16 - Simple Made Easy
• 14:23 - Coming Soon to ember-concurrency
• 16:04 - Communicating Changes in State; Glimmer Reference Primitives
• 23:09 - Using References
• 29:31 - Submitting RFCs; Adding Pipelines
• 32:10 - Pipeline Use Cases

Resources:

• ember-concurrency
• The Frontside Podcast Episode 007: The Ember Router with Alex Matchneer
• The Frontside Podcast Episode 019: Origin Stories with Tom Dale and Alex Matchneer
• Introduction to ember-concurrency by Alex Matchneer from Global Ember Meetup
• RxJS
• Rich Hickey: Simple Made Easy
• Glimmer.js
• redux-saga
• Lauren Tan's RFC: Cancellable task pipelines
• Railway Oriented Programming
• Apache Kafka

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 67. My name is Charles Lowell, a developer here at The Frontside and podcast host-in-training. With me today also is Elrick Ryan, a developer here at The Frontside. Hello, Elrick.

ELRICK: Hey, what's going on, Charles.

CHARLES: Now, we have with us today someone who we love to have on the show. Everybody probably already know him. I know the first time I actually heard about him was when we had him on the podcast the first time, I was like, "Who the hell is this guy?" But since then, he's become one of my favorite developers, just with all of the things that he's done, from Router.js to more recently ember-concurrency. We have Alex Matchneer on the program.

ALEX: Hey, everybody. Thanks for having me.

CHARLES: Hey, Alex and you know what? I pronounced your name right this time. First time out of the gate. Boom!

ALEX: Nice. Which one did you go with? Matchnear? Matchner?

[Laughter]

ALEX: I really actually don't even know which ones correct anymore.

CHARLES: Was it about a year ago that you first introduced ember-concurrency?

ALEX: Yeah, I had a really embarrassing introduction of it at an Ember Meetup in January before it was really done and I just kind of botched it and didn't really introduce why it was even solving problems. Then a month later, I had some time to refine it, driven by the feel of that embarrassment. I guess around February of last year, it's been pretty much in its present state.

CHARLES: I remember when it came out. I must've seen the non-botched version because I remember hitting the ground running with it and being able to refactor all of this code. I definitely know that I got the honed version because you provided in that initial blog post a whole host of examples like what are the symptoms, what are the cases where it solves and then before presenting the solution. I think that was great because I didn't even realize that I had a lot of pain. I didn't realize that at all. I didn't realize I had a problem but then you were very, very elegantly packaged the problem with the solution which is always great because otherwise, it's just complaining.

Maybe we should talk a little bit about -- I don't think we've officially talked about -- ember-concurrency. Even though it's been out for quite a while, the way that you model these concurrent processes using the stack is just pretty incredible. Do you want to just very briefly touch on what the problem is and what have lead you to this solution?

ALEX: Sure. It's a little bit difficult to sort of succinctly say what ember-concurrency is because it kind of hits them like five different separate but kind of related but not really pain points. At its core, it's just like a task primitive and it's definitely not the first library to ever introduce that the JavaScript, I think particularly when the generator function syntax was introduced into the spec, I think a few years back.

Dave Herman who's also known as, I think a Little Calculist. I think he works on the TC39. I always get those groups a little confused in my head but he introduced a task.js library that let you use the generator function syntax and then lets you yield Promises to sort of pause where you were in that task and then continue when it resolved. It had some support for cancellation. It played well with Promises and I brought that to Ember in a way that fit really nicely with Ember more than it probably does or will with other frameworks like React or Vue.

By bringing it to Ember, basically if you're implementing any feature that involves async, if it's a button that needs to show that it's been clicked while you're waiting for some response to come back from the server, instead of using Promises, instead of using actions, here's an ember-concurrency task. It makes it easier to express that operation you're trying to do and it makes it really easy to drive your UI with state that comes from the state of that operation -- Is the test still running? Is your form still submitting? -- Rather than having to manage a bunch of mutable flags and properties on a component or state yourself and likely get it wrong.

CHARLES: Right. Essentially asynchronous processes is like a state machine and before, we were kind of managing that state machine by hand but I think what's so brilliant about this task-oriented programming, I guess is maybe a way to put it because I really think that some of these ideas are universal and not specific to ember-concurrency. But it almost like it uses the stack, just your normal programming stack to track where you are inside of a process, rather than what it felt like what we were doing before, which was managing this state machine by hand, if that makes any sense.

ALEX: It does make sense a lot of sense. A lot of people ask me, if you're going to go into sort of async territory, why aren't you using something like RxJS? Rx is observables and kind of popularized by the Netflix crowd who did a bunch of presentations on them. It's super popular these days. But one of the things I really like about RxJS or at least one of the realizations I had is that I think you're still building a state machine. You're just expressing it using different primitives. In Rx, you're still building a state machine but in Rx, they make you think about it in terms of streams and events firing over time. In ember-concurrency, also you're still building a state machine but you're using the generator function syntax and the call stack like you mentioned as another way of expressing that state machine but with a lot less code.

CHARLES: Right. I was actually talking to someone about ember-concurrency just a few days ago and they were saying the same thing, "Why not use observables," and at least from my perspective, maybe I didn't quite understand the question because I feel like observables are kind of only one of the concerns that ember-concurrency addresses. I'm curious when people talk about alternatives to ember-concurrency and put observables forward, maybe I don't understand it because I usually think you might be able to use observables to register the currently executing task state and every time it changes, emit a new state and is then observed by your observable subscribers.

But modeling the actual process using observables does seem weird to me because with observables, they seem like very purely functional and not heavily stateful. I don't really have that much experience with it. What's meant by using observables as an alternative? Maybe we can get more into those like how you would construct a stream or something like that?

ALEX: I think the canonical Rx example of something that's elegantly expressed in Rx that would be really hard to do in just normal JavaScript, if you weren't able to use observables, is that typeahead search where as you type characters into a text field, it's already beginning to hit the server and see what you might be searching for so it can drive the state of some drop down menu. That's probably the most popular example out there because one of the things it demonstrates is that if you want to debounce, you want to allow for the user to stop typing for like 200 milliseconds before it actually hits the servers so you don't overwhelm your server, then just add a debounce operator. You've basically transformed a stream of keyboard events into that text field into something that only kicks off after it hasn't gotten an event for 200 milliseconds.

If you already had a working prototype in vanilla JS and you had to debounce it, you've got to move a bunch of stuff around, you've got extract something into a function, you've got to deal with cancellation. But all those things are kind of pretty elegantly built into Rx and if you can train yourself to think in terms of streams of events, that inspires you to think about where else you could apply that in your app. I think a lot of people have felt that it's like winning, most powerful abstraction that you could think about. That's why things like cycle.js are a thing or redux-observable or just anybody working with observables in the Netflix territory.

I personally find [inaudible]. If you're going to express certain processes, Rx is the way to go but it has drawbacks which is it is really hard to learn. It took me a very long time and I'm pretty good at it but if you're going to adopt Rx in your code base, then a new developer comes on, it's going to be a pretty long time. In my experience, sharing some of the Rx code I've written with fellow very talented developer, it takes a really long time to explain how to invert your thinking and think of things in terms of events. If you can get to that point, more power to you but what I found with ember-concurrency stuff is you don't have to completely invert your thinking and think of everything in terms of events and streams of events. You can use this task primitive which feels really pretty close to the code you're already writing but gives you a lot of the safety guarantees and just makes it really easy to use this derived state to drive templates.

Rx is a powerful paradigm and sometimes you need that sort of event-driven push based model but I think when people wonder why aren't you just using observables, they haven't really grasped how easy and familiar it is to use task and get it right on the first try and with a lot less code.

CHARLES: Right. You're able to leverage the fact that I understand what a JavaScript function looks like and the sequencing is implicit by just the order in which you were numerate the steps, right?

ALEX: Right. Because I think that Rich Hickey of Clojure popularized the divide between simple versus easy and Simple Made Easy is one of his popular talks that everyone should probably --

CHARLES: It's a great talk. Yeah.

ELRICK: Do you see an area where observables could be used in conjunction with ember-concurrency?

ALEX: It's kind of. It's been hard for me to find that use case. Probably, there's a handful of use cases where maybe it's a little awkward to think about to have something that would be elegantly handled in Rx would be model using tasks but it really hasn't struck me enough in some of the apps that I'm building, to really try and flesh that out.

CHARLES: I would be curious to see a side by side comparisons. I build a lot of auto completes using ember-concurrency. I built a lot of asynchronous processes using ember-concurrency. What would that look like using nothing but Rx and just be able to have it on the left-hand side of the paper, then Rx on the right hand side of the paper are easy.

ALEX: I'd be very surprised if you could find an Rx example that is less code than the task equivalent because as much as I think the autocomplete example is the best canonical example of Rx, once you actually start making something that's production-ready, you want to start driving the button state while it's running or to show a loading indicator. When you start deriving other observables off of the source observable which is the user typing into the text field, you start having to worry about, "I'm dealing with a cold observable. If I create another stream based on it, it might double subscribe and I might kick off two things. I actually want to use a published.ref version of the stuff."

To actually get away from a toy example into something that's actually production-ready, requires a lot of code. From my own conversations with the people working on Rx, there's a lot of people that are working on it and they're pragmatic about it and don't think that you have to be just purist functional all the way. But when they actually ship production code, they usually resort to using like the do operator. With Rx observables, which is basically an escape hatch to let you do mutations and side effects in what is supposed to be this monadic functional thing.

If the paradigm is breaking that quickly to do production code, I'd wonder if maybe there is something better out there. I just kind of keep that in mind but I'd definitely think there should be a bake-off or comparison of how you do things in both the task paradigm and observable paradigm but I think you'd find that in most cases, just do a lot more with a task, with a lot less code.

CHARLES: I want to go back to the point you were about to make about Simple Made Easy.

ALEX: On the divide, ember-concurrency is very easy. I still choose easy. In the case of reservable, I'm constantly choosing easy over simple and then it always helps me out because I've made that decision. I think most people inspired by Rich Hickey from the Clojure community, would look at ember-concurrency and be like, "At a task that combines derive state and does five different things seems kind of gross. Why don't you just use observables," and the result of that if you follow it through is that you end up writing a bunch of observable code that is a mess in streams and going in different directions and you've written something that's really hard to understand, even if it's seasons Rx developers looking at the code. It's just very easy to write things that are tangled.

CHARLES: It's always good to have simplicity but also a system that simple without ease, I think is far less useful because like I said, it's always going to be a tradeoff between simple and easy but the problem is if your system is too simple, then it means that you're shouldering your day-to-day programming task or shouldering the complexity and you have this emergent complexity that you just can't shake because your primitives are just too simple. You could be programming in assembly language or something like that. That's really simple. You need to be able to construct simple primitives on top of simple primitives so for your immediate need, you have something that is both simple and easy, if that's ever possible.

Certainly, ember-concurrency is easy and I think it just means there's maybe work to do in trying to tease apart the different concerns because like you said, there are five. But in real complex systems, there are five bajillions, maybe teasing apart those individual concerns that is composed out of simple primitives. I'm sure you've thought about that a little bit of how do I separate this and make these tasks compose a little bit better and things like that.

ALEX: This is a nice segue because it might tie into some of the work that's going to be going into ember-concurrency in the next few months. A big theme of EmberConf is actually, a lot of people are joking that it should just be called GlimmerConf because a lot of it was talking about how Glimmer is going to be this composable subset of Ember, like Glimmer is going to be the rendering layer and then there might be a Glimmer router and a bunch of these Glimmer components that once you npm install them, you get Ember.

Glimmers is a chance to think about Ember as a bunch of components working together under a really nice rendering layer. There's definitely some interest in bringing ember-concurrency in thinking what is so-called Glimmer-concurrency going to look like. Part of thinking about that is going to mean teasing apart some of these details as you were just saying.

I don't have a lot of specifics to give right now, just that there is a lot of interest in making sure at the very early on, there is some sort of Glimmer-concurrency equivalent. Generally speaking, as part of that process is the question of how do we bring these magical ember-concurrency parameters to just Node or just vanilla JavaScript in general. Perhaps you could use these kinds of tasks on a server and in other environments. I think there's some questions of the way the ember-concurrency bundles together derived state with the actual tasks runner, are you actually going to use that derived state in the server setting?

I think some of these pieces are going to have to come apart a little bit. I don't have very concrete ideas for how that's all going to look in the end. Just that I have faith that it will happen pretty easily and the result of it is going to be something that fits pretty nicely into Glimmer as well.

CHARLES: Yeah, I hope so. It certainly seems like one of the core issues right because Glimmer-concurrency really should be universal. It should be some -- I don't want to prescribe your work for you --

ALEX: I don't mind.

CHARLES: That wouldn't be cool. I mean, Glimmer is very stripped down. You have a very little bit on top of a raw JavaScript environment so if you're going to go there, it'll makes sense. What is this concurrent process builder look like using nothing but JavaScript? It seems like one of the hardest problems is to disentangle it from Ember object because the way that it currently computes that derive state is very intertwined with Ember object. You know the details of this more than I do but it seems like that's one of the biggest challenges is how do you communicate those changes in state without using that? That what I was thinking, it would be a good case for using observable for ember-concurrency, although not for probably the reason that people are thinking, which is for task composition and stuff but I'm very curious.

ALEX: Likely the first stab at that direction would probably be using something similar, if not exactly these Glimmer reference primitives. Maybe it is worth talking about that. References are one of the core primitives that's used by Glimmer and it represents a value that might change over time and it's a value that can be lazily gotten, whereas observables, you have something that's firing events every time something changes and it makes the whole pipeline process it right then and there.

With references, when something changes, you just tell the world like something's dirty. Then at a later time, maybe when in a request animation frame or some point where it actually makes sense to get the latest values, then it goes through and finds out everything that changes, does a single rerender. What it means is that you don't have the observe recode that's firing every time some value has changed. It's one of the guiding abstractions in Glimmer that makes it possible for it to be so fast and performant.

It is very likely that a vanilla version of what ember-concurrency does uses references because those are already separate from the Ember object model and actually are used today in conjunction with Ember object model with the Glimmer that works with Ember today. I think that's probably, to me a first step. Clearly the reference attraction has worked wonders for Glimmer. I prefer to probably use that than observables and the push-based.

CHARLES: Observables or something else. That is really, really interesting because there's nothing like vanilla JavaScript programming these days, like the equivalent of a Haskell thunk where you're just passing these things around but you're not actually using them until you actually want to pull a value. At that point, you kick off the whole chain of computation required to get that one value that you need. But it immediately brings to mind and I don't know if this is of concern to you but I was very, very enamored of Ember objects back in the day, in 2012. I was like, "Wow, this is amazing. This solves every problem that I've had."

It has been a great companion and I've built some really great stuff on top of it. But now it's definitely turned into baggage. I think it's baggage for libraries that I've written and we're talking about it in the context of it being baggage here and being making it more available to the JavaScript community so I worry a little bit about Glimmer references. Would they possibly turn into something like that and could you counteract that by maybe trying to evangelize them to the wider JavaScript community like, "Here's a new reactive primitive," so that we don't end up in an eddy of the JavaScript community, do you think there's value in trying to say, "There should be some standard in the same way of observable, which is an emerging standard is for eager reactivity, have some lazy reactivity standard," or maybe it's too early for that. That might be a way of future-proofing or getting insurance for the future so you can say, "We can confidently build systems on top of this primitive."

ALEX: If the worry is something based on Glimmer reference as it's going to turn into the same baggage or [inaudible] or whatever, that maybe Ember object has turned into some apps, some applications, some libraries. I'm not sure. I guess I don't really see that happening and I know that it's already gotten some validation from some of the people that have worked on Rx. In fact, a very useful primitives for certain kinds of workloads. As much as evangelism certainly helps. It's already off to a much better start than this all-powerful, god object that you can only interact with if you're using .get and .set functions. It's very lightweight.

What I'm trying to say here is that there's UI workloads and then there's server-driven workloads and using Rx for both cases means that Rx suffers as a library because in the UI workload, you want something like references where you want to let a bunch of things change and then update stuff in one pass just a tick later or later in the micro task queue. But in Rx, they make you think about things in event-driven way, which might make sense for servers and stream processing but it's ugly when you want to actually build UIs with it.

I think if we pay due respect to the fact these workloads are pretty different, I think the reference is way better of an abstraction for things that are UI-centric. They're simple and their performant and I think it's often much better foot than Ember object which is kind of bloated and huge and very hard to optimize.

CHARLES: Right. I like that because you have to be precise with the server side things but ironically, with the references, you only care about the state at the point at which you observe it -- when the user observes it, not when the code observes it. The user observes stuff with every animation frame and there can be any number of intermediate states that you can just throw away and you don't care about. You don't need to compute them. I think what you're saying is Rx forces you to compute them.

ALEX: Right and you wouldn't use a Glimmer reference for something if you're trying to batch. But in the end, keep all of the events that were fired on all the change events. You wouldn't use references because you're losing all that information until you do that poll and you get the latest value. But 99% of the time, when you're building UIs, that's what you want to use.

CHARLES: Are Glimmer reference is their own standalone library or do they currently bundled with Glimmer?

ALEX: I'm not sure. If they are not now, I believe the intention is for them to be at their separate repo. I was talking to Kris Selden at EmberConf and I got the impression that the intent, it might not be there now and if I want to start extracting ember-concurrency stuff into vanilla JS, I'd probably want to use a reference-ish thing, if not the official one from Glimmer.

CHARLES: I know we talked about this so then, how will you able to use these lazy references to compute tasks state? How that might work or play out?

ALEX: The fundamental problem right now is that everything in ember-concurrency is so glued to the Ember object model. What that means is that all ember-concurrency has to do is broadcast so the changes has happened to the state of a certain task so that you can, maybe put a loading spinner up on your template. All it has to do is use object.set and then the built in computed property observer change detection that is in Ember object model. It's going to sort of propagate these changes but that's a bunch of heavy Ember stuff that is going to exist and a lighter weight Glimmer or vanilla JS context.

Instead of using .sets and expecting that the thing you're setting it on is a big, heavy Ember object model, you could just use references. Then whoever wants to get a reference to whether a task is running or not, it is running reference. Then just using the standard Glimmer abstractions. At the Glimmer-concurrency task runner, it would just basically kick those references and anyone who has one of those references can flush and get the latest value at some later point in time and then update the UI based on that.

Already, as a maintainer at ember-concurrency, I see all the pieces work with that and I could probably just start working on that today. But there's just a handful of other things that I want to align with the vision of Glimmer and Glimmer-concurrency before I start working on that.

ELRICK: What would be the referency equivalent in just plain JavaScript outside of Glimmer that you would use to build this on top of --?

CHARLES: Like what would the API look like? If you're like, "I don't have a Glimmer. I don't have anything. I'm just --?"

ELRICK: Yeah, you just have plain JavaScript. What would be the primitive that you will build this on top of?

ALEX: Whether we use a standalone Glimmer references library or this separate reference thing, then I would use the term based on something Kris Selden said. In the end, the APIs is going to be pretty similar between those two but if one thing is requires, as far as I understand it, you've got to set up where in an event loop, your response is something that's changed and then you schedule at a later request animation frame, to actually do the rendering based on that. In order to use something like references, it implies that you've got to flush at a later tick or flush at a later call back. If you've got that in whatever app you're working on, it should be pretty easy to figure out where references fit in.

CHARLES: I see, so you would basically say like new task, give it your task class or whatever -- I'm just making stuff up -- then you would just schedule, do a request animation frame and then just pull the task state or something like that? Or a new task reference or something like that?

ALEX: You might have some function that's schedule render pass, if not yet scheduled. Then if it hasn't been scheduled, then use requests animation frame. If you call that function again, it's going to no op until that requests animation frame happened.

CHARLES: Could you explain that again?

ALEX: Sure. If you're actually thinking of a really low level vanilla JavaScript to your app, in the browser or something and you were just using references, then you probably have something where the thing that kicks off the reference or dirties the reference in some way, also run some function called 'schedule rerender', if one hasn't already been scheduled or something less verbose. That would just make sure that some request animation frame has been scheduled. When it flushes, then it will get all the changes but if more references are dirty at that mean time, it won't schedule additional request animation frames. I don't know, that's kind of blue-skying but that's when --

CHARLES: Right. Here's the other things, you see like being able to integrated with a third-party state management solution like Redux or something. Basically, I've got my ember-concurrency tasks and their state is then reflected inside a Redux store. How might that work, if at all? Or was that a crazy idea?

ALEX: I don't know. I played around with Redux toy examples and Redux community and Ember is only stronger by the day. I'm not entirely sure how all those pieces fit together because in Redux, they really want you to propagate all of your state changes using the reducer in that single global atom. A lot of people asked me about redux-sagas, which is another generator function-driven way of firing these state mutating actions over some async operation and this is hugely popular but I don't think they have any concept of the derived state that I've been trying to do with ember-concurrency of just being able to ask a task if it's running.

You can't just do that. You've got to reflect that into the global atom -- the global store --somehow and to be honest, I don't really know if that's fundamentally at odds with the Redux model, to take something like Ember or Glimmer-concurrency and make it work that way. But ideally, you wouldn't have to forward all that state into the global atom. You just be able to reference that task object.

CHARLES: If the task object itself is immutable, it would have seemed like fairly trivial, like you could generate programmatically the reducer required to do that. If you had the state encapsulated, you could come and say, "Now, there's a new state here." It seems like you would be able to adapt that but you would need to be able to react to any time if that state changed to fire and action in the Redux store or fire the Redux action.

ALEX: Actually, this will be an easier question to answer because in the Ember community Slack, there's a Redux channel and I know everyone there is already starting to talk about how are references, how is Glimmer is going to, how can we kind of tie these things to Redux and I think when they have some solutions lined up, a lot of the stuff that will be in so-called glimmer-concurrency will just fit in nicely. If they've got nice models for tying references to the state atom, if you will, then it's going to work with the new way.

CHARLES: Okay, cool. One of the things that I wanted to talk about was a proposal that Lauren Tan, who put on to the ember-concurrency issues list, although it's an RFC. Are you accepting RFCs now for ember-concurrency?

ALEX: I'm not pompous enough to have a separate RFCs repo. Issue approaches perfectly humble for me.

CHARLES: But is this the first RFC or have there been a bunch of ember-concurrency RFCs?

ALEX: There's been a few. It's definitely great that Ember have standardize on this boilerplate RFC model that everyone can fit their proposal into because it means that all the add-ons that people really like and really want to invest in, they get these high-quality RFCs versus like, "Hey man. It kind of nice if you can just like, have a pipeline."

[Laughter]

ALEX: Just because Ember invested in that process, the whole add-on community benefits from it and it's great. There's been a few RFCs that are like that. I'm not sure how many of them have made it but I've seen a few that are in that format but this one's definitely one of the nicer ones and a lot of effort was put into it and it looks really nice.

ELRICK: I'm not familiar with the RFC. What was the brief overview of what was proposed?

ALEX: Lauren was basically proposing that we add concept of pipelines, which is if you have a series of tasks that are stepping the pipeline of operations, then we should standardize that and then define all the steps in the pipeline so rather than having each step in the pipeline, call the next step in the pipeline. They just return some their portion of that work and then the pipeline infrastructure will automatically run the next step in that pipeline.

CHARLES: It seems like also then you can derive state about the entire pipeline, rather than just the individual task. You have to manage that a little bit by hand. But the other thing, I guess I would add is it seems like if you're going to go with pipelines, rather than being a simple list, you might want to think about it as being a tree because can you have pipelines that are composed of sub-pipelines, so to speak?

ALEX: Yeah. I believe the answer is yes. I'm not sure if it's spelled out in this RFC but really a pipeline just fits the task interface so if there's a task-ish thing or taskable object that you declare as a step of a pipeline or sub-pipeline, it should just work. I'm not sure if there's more work that needs to be done in spelling that out but that seems baked into it. There's a lot of due consideration for making sure these things compose really well and it's already in a really good state.

CHARLES: Yeah. What are some of the use cases where you might want to use a pipeline? I'm sure, everyone who's been writing concurrent tasks has probably been maintaining their own pipeline so what is it that you're doing and how is this going to save your time and money?

ALEX: Let's use the example that I've actually used in EmberConf, which is something based on my own app, which is in my app, you have to geolocate and find nearby stores that you could walk into and that process is four async steps in a row. One is getting your geolocation coordinates and then the next step is passing those the store, getting values and the third step is maybe some additional validation or just setting a timer so that your animations or any of these little async things that you have to do. But it's really a sequential operation where each time, fetch your geolocation or get it out of a cache and then step to the next thing, then the next thing.

It looks okay as I have it in my production app code but it still feels a little gross that you can just look at this thing and be like, "What is the sequence of steps here?" You have to actually get the implementation of each task to see what happens next and where will it go after that. Basically, with ember-concurrency in general, there's a lot of opportunities for finding more conventions for building apps. I don't even know if we really talk about this so far but derived state is part of it. But generally speaking before ember-concurrency, there wasn't as much opportunity, I guess for some of these conventions for building these pretty standard UI flows without feeling that you're just building your own thing every single time, with chains of Promises and your own improvise cancellation scheme and all these things.

I see pipelines as a next step. Well, we're pre-building lots of pipelines in our apps. We have these processes that go through these multiple steps and right now, the best we can do is set a bunch of Boolean variables and the derived state that comes with ember-concurrency helps but with pipelines, there's even more and it also structures your thinking so that if something like pipelines catches on, hopefully as an Ember developer, you'll see them in a few different places and already have that tool in how to visualize a problem, visualize a component, visualize the async flow.

CHARLES: If I spent my entire morning reading the talk that Lauren referenced in RFC, which was the Railway Oriented Programming, which I think, maybe not quite but basically a visual explanation of a Maybe monad or the Either monad or whatever it's called. One of the greatest explanations of why monads are helpful and through explanation using like the Maybe, where you can have a computation that could have more than one result, either success or failure and how do I take these functions and compose them with functions that might always succeed or might not have a return value or whatever and show the tracks that move through a computation and be able to normalize every function to have the same number of tracks. I realized, I'm getting into the description of it without actually having the visuals in front of it so I'm just going to stop myself and say everybody go read it. It'll take you 35 minutes but it will eliminate so much like the chatter that you've been hearing in the background for a couple years.

ALEX: I used to tell people that they should learn Rx because regardless of me liking the task primitive a little bit better, it's great. It just scrambles your brain and reorganize your thought processes and it's such an interesting library to learn.

CHARLES: All right. I like it. I'm going to go learn Rx.

ALEX: I've been getting, on the server side, the sort of Kafka-based architecture, Apache Kafka. Particularly, they've released some libraries in maybe the last year or so. It's a very Rx-familiar feeling library for composing new data and new aggregates and joins between different topics and streams of events. It just seems like they're at the forefront of solving these really hard problems in a very conventional way. You get into some of that stuff and you’ll find that you're doing a lot of server side processing with step that just feels a lot like Rx and I find it very interesting. I haven't actually build anything with it yet but it is likely in my future and anybody that's into the event-driven model should definitely know what people are working on in this Kafka-streams world.

CHARLES: That is cool. It's so interesting to see how all the problems that you encounter on working on the server side, you will encounter on the client and vice versa. You can build up a huge corpus of knowledge on one side of the API divide and you'd be surprised that if you were to go work on the other side for a time, you'll be able to leverage 99% of that knowledge. That's fantastic.

I would love to get into Kafka but unfortunately, I think we're going to have to save that for another time. That's another one of those words like... I don't know. Is Kafka descended from Storm or something like that? Is it a similar concept? I remember everybody was big on Storm.

ALEX: I think Storm process the events and decides what to do with them. Kafka is really just a giant storage that plugs into something, I think like Storm or [inaudible] or any of these things that actually process the events.

CHARLES: Yeah, it's all Kubernetes to me.

ALEX: Yeah.

CHARLES: All righty. Well, with that, I think we'll wrap it up. Thank you so much, Alex for coming to talk to us. It's always enlightening. I love your approach to programming. I love how deeply you think about problems and how humble you are in approaching them because they are big.

ALEX: Well, thank you. It's great to be on here. It's fun.

CHARLES: All right, everybody. Take care. Bye Elrick, bye Alex.