From Emacs to Atom

I want to start this post by stating I have nothing but respect, admiration, and love for the Emacs community. Emacs’ extensibility, community packages, and it’s choice to effectively be an editor built on a lisp VM is amazing, and anyone choosing Emacs as an editor is investing in something that can grow with your needs.

Nevertheless, there are compelling reasons to switch to Atom. I am saying goodbye to Emacs, and have started using Atom as my main text editor.

My History with Emacs

I learned about Emacs during my college years (roughly 2008), when I happened to attend a house party for a family friend. The friend was a software developer who retired many years ago, but upon learning of my interest in software, he began to regale me with story after story of how much he does in Emacs.

“I check my e-mail with Emacs.”

“I built a program that opens my garage door from Emacs.”

“I share the same editor across multiple computers using the remote Emacs client.”

In all honesty I wasn’t really impressed by the idea of a program that you use to literally do everything, but it seemed like a great kernel for a text editor. I was using vim at the time and what I always lamented at (which I know others would say it’s vim’s greatest strength) was the fact that it could only be used to modify text. When I write code, I do so much more than write the code itself. I wanted an environment that made executing additional tasks seamless:

  • Interact with version control (git push / pull, commit, add)
  • Code search
  • Running command line scripts
  • Running a REPL and unit tests

Thus I dove into Emacs. The built-in terminal emulator, the ability to build whole programs in a single .el file and load them up, adding significant functionality to the editor, meant a lot of my needs were met quickly, and without significant effort. This became even easier with the release of Emacs 24, which included a built-in library to retrieve and install third-party packages, reducing the need to copy and paste.

I continued pretty happily for several years. I made a couple videos showcasing my Emacs setup, published my dotfiles, and wrote some tutorials as well.

Enter Atom

In 2015, Atom was released. An editor that was inspired by the flexibility of Emacs, but was built on HTML5 technologies. Using web technologies to build out massive native applications is not an advantage in every respect, but there are strong wins in some important areas.

A Powerful UI Framework

HTML, CSS, and Javascript have all grown to support the needs of the massive range of uses that websites are now used for, and the result from tackling such a large breadth is a powerful, general system for laying out various windows, styling them appropriately. Combine that with highly optimized runtimes to render said windows (web browsers) and you have a system that is not just developer-friendly, but also user-friendly.

Large Pool of Experienced Developers

A large portion of software engineers are web developers, and thus work in web technologies. The ability to transfer even some of this expertise as one is extending their editor removes a large chunk of the learning curve.

Impressed, But Not Sold

Atom was conceptually the editor I always wanted: the power of Emacs, a flexible UI framework, and a core built on technologies that I knew well and could contribute to. If I was starting from scratch, I may have chosen Atom, even when it just came out of beta.

However, I wasn’t starting from scratch. There was years of expertise in elisp, finding the right packages, learning to use them, and familiarizing myself with keybindings and the Emacs way of doing things. It didn’t make sense to throw those out the window for a nascent editor.

The Catalyst: Atom IDE and Language Server Integration

Since Atom came out, there was another text editor that entered the scene: VSCode. Similar in design to Atom, VSCode took a more opinionated approach to how an editor should be organized, and what tools to use (in the vein of visual studio). The more open world of Atom wasn’t a first priority (for example, VSCode did not provide support for more than three text windows at a time until recently).

However, VSCode did directly lead to the creation of the language server protocol, which enables any text editor to take advantage of IDE-like features, as long as they build an interface to a JSON-RPC based API.

When Atom implemented it’s language client, it was impressive, and it made me want to try Atom. But making the switch would require me to port all of my existing tools to find equivalents, and most likely learn a new set of keybindings. I already had a lot of that in Emacs. However, there was a final factor that really made me switch.

Community Critical Mass

For almost any tool or program, you’ll find one that is better in almost every significant way, but yet has not taken off. As much as we’d like to believe software engineering is a purely merit-based field, the reality is it depends on socio-economic factors as much as every other discipline. Market and mind-share matters.

The most impressive part of the language server protocol is not that it was built, it’s who built it. Facebook was a major contributor, teaming up with Github to build a real IDE experience for Atom.

Facebook’s business practices aside, they have a giant and talented engineer base. With Facebook engineers supporting a plugin like the Atom IDE, there’s a strong chance that you will see that integration improved and supported for years to come. And Atom is also a blessed project from Github.

I love Emacs, but it’s primarily supported by a volunteer base, who have other fulltime jobs. It’s very difficult trying to get a group of developers to implement something like language server support, and maintain and contribute back for years to come.

And the active community is larger around Atom. As of October 2018, here’s the counts of packages on the major package repositories per editor:

Unfortunately, Emacs does not have the development community in the same way Atom and VSCode can. That’s a conversation worth diving into, but it doesn’t change the state of the world today.

Migrating to Atom

So, I migrated my Emacs setup to Atom. Since I was a relatively late adopter a majority of my desired features were already a part of the editor, or were available as an extension.

I don’t think it’s valuable to dive into exactly what my setup looks like, but if you’d like to learn more, you can check out an Atom plugin I’m working on:

https://atom.io/packages/chimera

I am not using Atom 100% of the time, and I haven’t opened Emacs in about a year. The migration process took a couple of weeks.

The Future

Today, I have a lot invested in Atom, and I like my experience. Language server integration was a missing pain point, and that ecosystem (along with Atom’s integrations with it) is getting better every day.

The biggest lost I faced with Atom was performance: due to its reliance on a browser-based renderer, performance suffers vs draw calls in a native GUI. There are also improvements that can be made to Atom to ensure more non-blocking UI actions.

The Atom team has been working on xray, a text editor designed for performance whose improvements will be incorporated into the editor.

VSCode has also been a lot better on the performance front than Atom (still orders of magnitude slower than native editors). I tried it out recently and found the performance gain for me has been imperceptible, so it’s probably not worth the effort to lose the extension and keybinding knowledge.

 

 

 

The Why of Disp Pt. 1: The Syntax

Over the past few weeks, I’ve spent some intensive time on Disp, a programming language that looks syntactically like lisp, with the goal of making managing large codebases easier.

There’s a lot of ideas that went into it’s design, so I wanted to lay them out in a series here. I’m looking for feedback, so don’t hesitate to reply back if you disagree or have ideas. Also please check out the RFCs and leave some thoughts.

This first series is around the choice of lisp + indentation. Specifically, disp syntax looks something like:

(no highlighting unfortunately, Disp is it’s own fun syntax).

It’s very lisp-like: you’ll see the standard parens, which represent a function invocation. But you also see that some parens are missing, and indentation exists instead.

There are two extra rules to manage these syntactical changes:

  • every newline is considered an implicit expression
  • indentation means that you are providing a list to the previous, less indented statement.

It means the following two forms are identical to the parser:

to help reduce the parentheses, every statement on a new line is considered to be an expression, and a list as an argument can be represented with an indented block.

There were a couple reasons for this choice:

Readability

A major complaint with lisp is the number of parentheses required, making it hard to see where parentheses begin and where they end. Lisp experts say that you get used to it and it’s not a major issue in the long run. There are also plugins for many editors that match the parentheses using colors, that helps as well.

However, considering the language will be responsible for the parsing anyway, it seemed intuitive to remove unneeded symbols if the purpose could still be clear to a reader. The easiest removal was the surrounding of parenthesis on a newline statement: at that point, the syntax looks very similar to a non-lisp based programming language:

The indentation rules enable an almost python-like syntax: in many cases a block is represented by a list of statements or expressions, so allowing one to enumerate results in a similar level of readability, as far as expressions go:

Semantic Indentation for Consistency

Many languages are moving to the singleidiomatic formatting paradigm, which does a great job of quelling style discussions which ultimately have minor benefits for the reader, but consume a large amount of time. I think this is a must for a language for large organizations, and Disp continues in that vein.

Adding semantic meaning to indentation is almost a self-fulfilling prophecy: by adding semantic meaning style becomes more consistent, and it’s possible to add semantic meaning to indentation because style is consistent. It would be a waste to not use it to semantically.

Indentation is also used to improve readability and denote blocks of code in most styleguides, so it’s not a far stretch to use it as such.

Tabs instead of Spaces for Indentation

I’m sure this choice is a bit more controversial, but Disp uses tabs for indentation, instead of spaces (unfortunately many code snippets here use spaces because it’s hard to type tabs in browser).

There are many different flavors of indentation to choose from. Python, for example, is extremely lenient and allows one to use a mix of both. In the name of consistency and simplified parsing, it made sense to choose a single one.

Tabs was chosen to allows developers to modify the tab width settings in their IDE, choosing the spacing that is more legible to them.

Conclusion

Thanks for reading! I’m looking for any help to improve the readability or remove unneeded syntax. There’s more in this series coming up, so stay tuned.

 

 

 

 

Book Report: The Undoing Project

The Undoing Project: A Friendship That Changed Our Minds is written by Michael Lewis, the same author of other books such as Moneyball. This book is a sort of spiritual successor to Moneyball, focusing on the psychologists Daniel Kahneman and Amos Tversky and their work in the area of human decision making.

Although a significant part of the book focuses on the duo behind the papers, there’s quite a bit of content around the irrational behaviors that are captured in the papers published, as well as their impacts in multiple fields well outside of psychology.

The Impact of Phrasing

Many of Kahneman and Tversky’s papers are built of of data from asking their undergraduate students questions via surveys. This has illuminated the ways in which the human mind does not operate rationally (such as guessing a sequence of numbers 1 * 2 * …. * 10 differently based on whether the series starts with the low number of the higher number). The duo often played around with wording, and found that some gentle hinting of the question itself can yield results where the answers are more rational.

This implies that phrasing has a significant impact on the results that are yielded. Unfortunately that means many surveys can be designed in such a way that the results are skewed in some particular favor.

Human Fallibility Affects All Fields

A major theme in the research is around the human mind’s inability to evaluate situations rationally. One really interesting aspect was around the probability of various events: estimates were easily influenced by the recency of an event, or it’s recollection in the estimator’s mind.

The book dove a little bit on the idea of evidence-based medicine, and how in the past it was in many areas a handed-down, subjective practice. The ones who broke through this subjective practice was those that were willing to challenge the approach of the established groups in a particular field.

My interpretation is to have others double-check work and major decisions: personal biases can easily creep in. I find this is especially difficult in the field of software development, where many choices are stylistic approaches, and the pros and cons are not apparent. Many of the factors of good software development is behavioral, from making fewer coding errors to ensuring best practices are kept.

Personal Thoughts

I really enjoy Michael Lewis’s writing because of it’s ability to highlight specific real-life situations that further cement the specific idea. In Moneyball, Lewis did a great job of really showing how Billy Beane is able to apply the statistical approach to baseball in his recruiting and play. In The Undoing Project, Lewis explains the logical fallacies using the real experiments that Kahnaman and Tversky ran on their students. Humans love stories, and It’s a great way to leave an imprint and recall these lessons well after reading the book.

My New Dev Laptop: The HP Spectre X360

The Macbook Pro has been a staple of my time as a software developer: it was my desktop replacement and I used it at work and at home. I’ve used Macbook Pros constantly since 2008.

Last year I went on the hunt for a replacement, and landed on the HP Spectre x360. Here’s my setup, and why I chose that over alternatives.

Conclusion: My Setup

Here’s the setup I run now:

Here’s a picture:

dav

The Why

The Portability of 13 Inches and 1 Pound

I’ve owned many 15″ laptops, and a 17″ back in 2008. With the 17″ it was just a question of how soon my back would be hunched forever, but even with the 15″ the portability became a minor, but significant issue. I would often find myself in situations where space is limited, such as the bus or on a plane, and get into bizarre positions just to have enough space to type and see the screen. For my next laptop, I knew I wanted something truly portable, that would ensure a comfortable experience even in space-constrained situations.

The 13″ size and weight has been great: I can easily hold my laptop with one hand, and it’s my made backpack noticeably lighter. Despite the 4k display, it’s really hard to take full advantage of that with a 13″ screen, but I can get 3 columns of text windows side by side and read them without strain.

Driving 4k Displays

Despite being a super thin, lightweight laptop, the HP Spectre can handle high resolution displays. I’ve been able to drive 3 4k displays (using the dock) at 60 frames per second. It cannot run 3D games with any reasonable performance, but it can render 2 Atom text editors and a web browser window without issue.

USB-C Charging

Being one of the first real charging technologies that’s portable across laptop vendors, USB-C charging was a must. It has enabled me to bring a single charger for both my laptop and my phone, and even allowed for battery packs that can completely charge my HP Spectre on the road.

Linux

OSX has been decent, but as someone who has a highly specialized UI setup in Linux, the experience didn’t compare. I want to never leave the keyboard, using tiling window managers to easily navigate editors and windows, and remove rarely used UI elements. It’s really difficult to get that setup in OSX. OSX is tightly coupled with it’s default window manager, and the best you can do is something that manipulates said windows for you like Amethyst (given the limitations, Amethyst does an amazing job. I don’t believe anyone could do much better than the experience Amethyst provided).

The most frustrating issues stem from not being able to fix problems yourself. Closed source and limited pluggability mean twiddling thumbs and asking desperately for updates. For example, High Sierra had a bug where closing a Macbook pro connected to a thunderbolt dock, then removing the thunderbolt port, caused the screen to not come back. With Linux, I could find a fix and apply it.

Price

Apple is a premium brand, and it shows in their price. High-spec Macbooks run in the 2000+ price range. Meanwhile, a refurbished HP Spectre with a 4k display, 16GB ram, and I7 Quad Core can be had for $1000 USD.

The Cons

I think it’s fair to note some of the places where I’ve had issues, or a less than ideal experience.

CPU in high-load situations

When I’m attempting to run Atom in an IDE-like environment (e.g. running the Rust language server), there’s a noticeable delay compared to my desktop. It’s rarely an issue, but it does make me lean toward working on my desktop at home.

This probably will not differ from other laptops this generation, as the I7-8550u is a very popular model, and has been used in almost every high-end laptop that focuses on size.

Bonus: Thunderbolt Dock Talk

Thunderbolt promises to be a spec that allows for a lot more interoperability between docks and laptops, since it’s effectively a glorified PCIE port. However, the reality is a lot of the functionality of the dock is powered by the laptop itself: if your graphics card cannot run the resolution, your dock will not always help you.

Some docks use a technology known as DisplayLink, which is effectively embedding the graphics card in the dock and communicating through USB. I personally have not tried this, but reviews have noted that the latency of the DisplayLink setup is noticeable.

The dock I linked above exposes two ports for attaching displays: one DisplayPort and one USB-C. I can hook up one display to each, each at a 4k resolution.

Thunderbolt docks also require approving and authenticating the dock, due to the Thunderbolt protocol allowing direct access to hardware. Any operating system you run will need to support authenticating and approving such devices. Linux distros can use the thunderbolt userspace tooling. Arch and Ubuntu provide packages to download and use them.

Using Rust functions in LLVM’s JIT

LLVM is an amazing framework for building high-performance programming languages,
and Rust has some great bindings with llvm-sys. One challenge
was getting functions authored in Rust exposed to LLVM. To make this happen, there’s a few steps to walk through.

1. Exposing the Rust functions as C externs

When LLVM interfaces with shared libraries, it uses the C ABI protocol to do so. Rust provides a way to build do this, out of the box, using the ‘extern “C”‘ declaration:

extern "C" pub fn foo() {
  println!("foo");
}

This instructs the Rust compiler that this should be exposed in a way where it can be found and used as a library. In the case of an executable binary, this is still the case.

The big gotcha here is ensuring that you are declaring the function as public, AND you are declaring it as public in the main module too. If the function was located in a child module, you will need to re-export in the main file:

// src/my_mod.rs

extern "C" pub fn foo() {
  println!("I'm a shared library call");
}

// main.rs
mod my_mod
// note the pub here.
pub use self::my_mod::foo;

Book Report: The Whole Brain Child

The Whole Brain Child discusses strategies to teach children how to deal with difficult situations in an empathetic and rational way.

Despite the focus on teaching children, the book included a lot of great insights for adults as well. In general, it is a great guidebook on how to deal with emotional situations, and how to ensure that the approach is one that accounts for the impact on others and their well-being.

The general structure is emphasizing a few key strategies when encountering an emotional and upset child. Here is my interpretation of the general ideas:

Introspect when Encountering Conflict

The book posits that we have two different “brains” that work in tandem when encountering a difficult situation: the “lizard brain” which reacts with strong emotional responses, and the “upstairs brain” which can approach the situation rationally.

Along the theme of educating children on how to cope in situations of conflict, the book explains that, to best help someone understand the situation, it is first better to empathize with how the chid is feeling. Once the lizard brain is no longer in control, deconstruct the situation rationally. This quiets the immediate reaction of the lizard brain, and enables a discussion when your child is using their upstairs brain.

As an example: if your child is throwing a tantrum because they are not getting the ice cream they wanted, it is first best to acknowledge the feelings of the child first (“you seem angry”), then rationally explain that eating too much ice cream is not a good choice.

This lessons works great for children, but I see it as a great lesson for conflicts with adults: if you want to reach an agreement with someone and you have a strong emotional reaction, first acknowledge the emotions, then reconcile on a logical level.

Physical Activities to Clear Your Mind

The book references a study that explains that physical activities can help calm emotional reactions, and bring someone into the state to discuss the situation rationally. Thus, a good tool may be to help pace the room, or do some jumping jacks, before diving into the conflict itself.

Talking Through the Situation Repeatedly

When one encounters a traumatising situation, and one that is difficult to understand (like a loved one being taken away by an ambulance, or a car accident), an insecurity can linger: one may become more upset when a loved one leaves for work.

The insecurity stems from a lack of understanding, and the reassurance that, despite how traumatising the situation was, everyone still turned out ok. Talking it through multiple times, ensuring that the child has a good understand of what actually occurred during the scary part of the experience, and a reminder that the child is still ok at the end, will reduce that insecurity.

Thinking About the Larger Picture

When a conflict occurs, one can get invested and extremely emotional. This can occur with even small conflicts that have a minimal impact on our day to day lives, such as an argument at work. In those situations, the lizard brain takes over, and one does not weigh the argument appropriately. One often becomes invested and very upset if the outcome does not go their way (e.g. a minor technical disagreement at work).

In a situation like this, taking a step back, and considering the larger impact works well. Will this choice cause me to lose my job? Will it cause my company to lose a significant amount of cash? Will I be the one responsible if things go wrong? If the answer to the above is no, then it is a sign that it may not be worth the investment, or at least being emotional about it.

I find myself in this situation often: I am opinionated about many aspects of my job, and the company I work for. It is valuable to have a logical argument for what you are advocating, and to spend time on that. However, it does not mean that, if the outcome is to move forward with a different approach, I should be upset for hours or days afterward. The impact this decision will have on the part of life I care about is minimal. Keeping perspective on what is important helps focus me on the discussions I should be having, and spending more time on those.

Use Introspection to Understand Emotions

A major theme of the book is examing the situation in a rational light, allowing some time to consider whether the response is appropriate. Being able to explain why you feel a specific way is powerful: you can better understand why you react this way, and modify your behavior if it is appropriate.

Explaining How Others are Feeling

Children often act without regard to how others feel, such as grabbing a toy from another’s hand, or erasing another’s work. When a child does so, it is often without malice, but rather the lack of understanding of how it feels to have that done to you.

By explaining how one feels in that situation, the child learns how to empathize. Getting the child into the habit of considering other’s feeling before taking an action helps reduce interpersonal conflicts, and can often avoid them. Deliberately educating on why one should feel empathy on a situation when the opportunity arises ensures the lesson is learned, and allows the child to achieve competency in a valuable skill early.

Final Thoughts

For me, “The Whole Brain Child” did a great job of putting more abstract ideas in my head in writing. I use many of the techniques outlined in the book when encountering conflict, but I had never put deep consideration into why those techniques worked. I have also never thought deeply about how to share these skills.

The book does a great job at all of the above: it provides a step by step guide for moving from an emotional state of mind to a logical one, adding understanding of the situation to analyze what could be improved, and explains how to further build an empathic foundation in children.

Definitely recommmend a read.

Book Report: The Millionaire Next Door

The Millionaire Next Door is a explanation of the behavior and attributes of those who have achieved a significant amount of wealth. Contrary to the title, it does not just examine millionaires: instead, it looks at those who have a significant amount of net worth, multiple times their income.

How Much Net Worth Should You Have?

The main gauge for determining your success is your net worth relative to income. The equation looks like:

2 * income * age_in_years / 10

So, if you are 40 years old and you earn 80,000 dollars, then you should have 2 * 80k * 40 / 10 = 640k of net worth.

The 2 at the beginning of the equation is a multiplier factor that helps puts you in one of a few categories:

  • 0.5: UAW (under accumulator of wealth). Anything at or below this number is the last quartile of net worth of those with similar income.
  • 1: AAW (average accumulator of net worth).
  • 2: PAW (prodigiuos accumulator of net worth): Anyone at or above this number is within the top quartile.

The book was written a few years ago (1996), so it could be that these numbers have changed. But the general philosophy is to set a goal of net worth that is a multiplier of your income, in contrast to a set number for all.

The Behavior of the Wealthy

The book contends that the wealthy have two primary behaviors in common:

  • living considerably below their means
  • making sound, non-taxable investments

Choosing a lifestyle that’s below one’s means results in a significant amount of excess income. That excess income can then be used to fund investments, which compount in interest and have returns that increase exponentially as time allow. The investment being untaxable (or taxed minimally) increases the return of the gains (as there is less money pulled out of the investment pool) in contrast to an investment taxed at normal income levels.

By building a foundation of net worth that compounds, and reducing the cost of living in one’s day-to-day life, it tackles the goal of financial independence on both sides: reducing the target goal, and investing heavily to get to that target quickly.

The Spending Habits of the Wealthy

A majority of the wealthy interviewed had very similar behaviors around money. Oftentimes, the wealthy went with financially conservative options. This includes buying and keeping a car for decades, forgoing the finer things like exotic vacations and boats, and simple, cost effective hobbies.

The wealthy also strictly budget: they track spending closely to ensure that the wealth accumulates over time.

Thoughts

I think this book formalizes a lot of the ideas we understand in principle. Many of the ideas the book puts forth are logical, but it is nice to have them all in one place. The book has already influenced the way I am organizing my assets. I am looking toward investing more, and setting a clear budget goal on the amount I would like to invest.

Having the PAW target also helps, as it sets expectations on how much one should be saving, to continue toward the path of achieving a significant amount of wealth.

Conversely, I do not agree with the idea that we should live a frugal lifestyle and ignore the opportunity for unique experiences: vacations to new places and trying new things is a pillar of a satisfying life. Ultimately, wealth should be accumulated to enrich the lives of those you care about. Wealth does not accomplish that by sitting in a bank.

I aim to be more conservative around luxury items like products I do not need, or eating out when I can bring food from home. But I will continue to look for ways to use my wealth to get the most out of life, and I won’t hesitate to take a step back from my long-term wealth goal to have a once-in-a-lifetime experience.

Aiohttp vs Multithreaded Flask for High I/O Applications

Over the past year, my team has been making the transition from Flask to
aiohttp. We’re making this
transition because of a lot of the situations where non-blocking I/O
theoretically scales better:

  • large numbers of simultaneous connections
  • remote http requests with long response times

There is agreement that asyncio scales better memory-wise: a green thread
in Python consumes less memory than a system thread.

However, performance for latency and load is a bit more contentious. The best way to find
out is to run a practical experiment.

To find out, I forked py-frameworks-benchmark, and designed an experiment.

The Experiment

The conditions of the web application, and the work performed, are identical:

  • a route on a web server that: 1. returns the response as json 2. queries a
  • http request to an nginx server returning back html.
  • a wrk benchmark run, with 400 concurrent requests for 20 seconds
  • running under gunicorn, with two worker processes.
  • python3.6

The Variants

The variants are:

  • aiohttp
  • flask + meinheld
  • flask + gevent
  • flask + multithreading, varying from 10 to 1000.

Results

variant min p50 p99 p99.9 max mean duration requests
aiohttp 163.27 247.72 352.75 404.59 1414.08 257.59 20.10 30702
flask:gevent 85.02 945.17 6587.19 8177.32 8192.75 1207.66 20.08 7491
flask:meinheld 124.99 2526.55 6753.13 6857.55 6857.55 3036.93 20.10 190
flask:10 163.05 4419.11 4505.59 4659.46 4667.55 3880.05 20.05 1797
flask:20 110.23 2368.20 3140.01 3434.39 3476.06 2163.02 20.09 3364
flask:50 122.17 472.98 3978.68 8599.01 9845.94 541.13 20.10 4606
flask:100 118.26 499.16 4428.77 8714.60 9987.37 556.77 20.10 4555
flask:200 112.06 459.85 4493.61 8548.99 9683.27 527.02 20.10 4378
flask:400 121.63 526.72 3195.23 8069.06 9686.35 580.54 20.06 4336
flask:800 127.94 430.07 4503.95 8653.69 9722.19 514.47 20.09 4381
flask:1000 184.76 732.21 1919.72 5323.73 7364.60 786.26 20.04 4121

You can probably get a sense that aiohttp can server more requests than any
other. To get a real sense of how threads scale we can put the request count on
a chart:

 

The interesting note is that the meinheld worker didn’t scale very well at all.
Gevent handled requests faster than any threading implementation.

But nothing handled nearly as many requests as aiohttp.

These are the results on my machine. I’d strongly suggest you try the experiment
for yourself: the code is available in my fork.

If anyone has any improvements on the multithreading side, or can explain the discrepency in performance, I’d love to understand more.

MongoDB Streaming Pattern, Allowing for Batching

An interesting problem arose at work today, regarding how to build an
aggregate of changes to a MongoDB collection.

A more general version of the problem is:

  1. you have a document which has multiple buckets it could
    belong to. Say, an animal which an arbitrary set of tags,
    such as [“mammal”, “wings”], and a discrete type location [“backyard”, “frontyard”, “house”].

    an example document could look like:

    { "name": "Cat",
      "location": "house",
      "tags": ["mammal", "ears"]
    }
    
  2. Make it easy to retrieve the sum of each type, by tag. So:

    {
       "tag": "mammal",
       "location": {
         "house": 10,
         "backyard": 4,
         "frontyard": 2,
       }
    }
    

The animal location is updated regularly, so the aggregates
can change over time.

A First Attempt

The simplest way to perform this is to rely on Mongo to retrieve all
animals that match the tag by indexing the tag field, then handling
the query and count in the application.

This works well for small scales. However, performing the action in
this way requires a scanning query per aggregate, and that must scan
every document returned to perform the aggregate. So, O(matched_documents):

def return_count_by_tag(tag_name):
    result = {
        "tag": tag_name,
        "location": defaultdict(int)
    }
    for result in db.animals.find({"tag": tag_name}, {"location": 1}):
        result["type_count"][result["location"]] += 1

    return result

In our case, we needed to return an answer for every tag, within a
minute. We were able to scale the approach with this constraint in
mind to 35,000 tags and 120,000 documents. At that point, the
application was unable to build the aggregates fast enough.

The New Strategy

The main disadvantage of the previous design is the calculation of the
aggregate counts does not need to be on read: if we can ensure
consistent count updates as the location actually changes per
document, we can perform O(tag_count) updates per document instead.

The comparative complexity over a minute is:

  • old: len(distinct_tags) * len(average_animals_per_tag)
  • new: len(updates_per_minute) * len(average_tag_count_per_animal)

So, if we have:

  • 30,000 tags
  • 120,000 animals
  • 40 animals average per tag
  • (40 * 30,000) / (120,000) = 10 tags per animal
  • 10000 updates a minute

The number of documents touched is:

old: 30k * 40 = 1.2 million reads
new: 10k * 10 = 100,000 writes

So, we can scale a bit better by handling writes over reads. This
becomes an even better ratio if the updates occur at a less frequent
cadence.

So, the stream processing works by:

  1. every desired changes is enqueued into a queue (in Mongo, this can
    be implemented as a capped collection)
  2. a worker process pulls from the queue, and processes the results.

The worker process:

  1. reads a watermark value of where it had processed
    previously (Mongo ObjectIds increase relative to time and insertion
    order, so it can be used as the watermark)
  2. performs the work required
  3. saves works to the collection
  4. writes the watermark value of where it had finished processing.

You could also delete records as you process them, but it can cause
issues if you need to read a record again, or if multiple workers need them.
need them.

Starting from Scratch

So how do we allow starting from scratch? Or, rebuilding the
aggregates if an issue occurs?

There could be a function that performs the whole collection
calculation, dumps it to the collection, and sets the watermark to
whatever the most recent object is in the queue.

Unfortunately, this process and the worker process cannot run at the
same time. If that happens, then the aggregate collection will be
corrupted, as one could query an older version of the collection, have
updates that are applied to the original aggregate copy, and are overwritten
with a stale copy from the rebuild.

Thus, we must ensure that the update worker does not run at the same
time as the batch worker.

A locking strategy

In Mongo, the locking is decided by the database, and a user has no
control over that. Thus, we must implement our own locking functionality by
using Mongo primitives.

The same record that holds the watermark could also hold the lock. To
ensure that we can survive a worker dying halfway and not releasing,
the lock, we can provide a lock owner, ensuring the same process type
can begin an operation again:

{ "name": "pet-aggregates",
  "watermark: ObjectId("DEADBEEF"),
  "lock": {
      "type": "update" // could also be type: bulk
  }
}

Using this type of lock, the possible failure scenarios are:

  1. update process lock, failure, and update doesn’t run again:
    This requires manually looking at the issue, resolving, and restarting the queue.
  2. bulk process lock, failure, and bulk doesn’t run again:
    This requires manually looking at the issue, resolving, and restarting the queue.

deepmerge: deep merge dictionaries, lists and more in Python

Introducing deepmerge. It’s a library designed to provide simple
controls around a merging system for basic Python data structures like dicts and lists.

It provides a few common cases for merging (like always merge + override, or raise an exception):

from deepmerge import always_merger, merge_or_raise

base = {
    "a": ["b"],
    "c": 1,
    "nested": {
        "nested_dict": "value",
        "nested_list": ["a"]
    }
}

nxt = {
    "new_key": "new_value",
    "nested": {
        "nested_dict": "new_value",
        "nested_list": ["b"],
        "new_nested_key": "value"
    }
}

always_merge(base, nxt)
assert base == {
      "a": ["b"],
      "c": 1,
      "new_key": "new_value"
      "nested": {
          "nested_dict": "new_value",
          "nested_list": ["a", "b"],
          "new_nested_key": "value"
      }
}

deepmerge allows customization as well, for when you want to specify
the merging strategy:

from deepmerge import Merger

my_merger = Merger(
    # pass in a list of tuples,with the
    # strategies you are looking to apply
    # to each type.
    [
        (list, ["prepend"]),
        (dict, ["merge"])
    ],
    # next, choose the fallback strategies,
    # applied to all other types:
    ["override"],
    # finally, choose the strategies in
    # the case where the types conflict:
    ["override"]
)
base = {"foo": ["bar"]}
next = {"bar": "baz"}
my_merger.merge(base, next)
assert base == {"foo": ["bar"], "bar": "baz"}

For each strategy choice, pass in a list of strings specifying built in strategies,
or a function defining your own:

def merge_sets(merger, path, base, nxt):
    base |= nxt
    return base

def merge_list(merger, path, base, nxt):
    if len(nxt) > 0:
        base.append(nxt[-1])
        return base

return Merger(
    [
        (list, merge_list),
        (dict, "merge"),
        (set, merge_sets)
    ],
    [],
    [],
)

That’s it! Give and try, and Pull Requests are always encouraged.