The highs and lows of Rust (2017)

More than a year ago, I wrote about my experience programming in Rust and what I felt were its high and low points. Recently, I was asked if the things I wrote then are still relevant, and if the highs and lows are the same now. I realized there is enough to talk about since my last review that it was worth writing a new post.

The highs

The high points of using Rust for me are essentially the same as before, so I'd suggest reading that part if you didn't read it then.

To summarize, Rust has been a paradigm shift for me as a programmer. It greatly raised the bar for me in terms of what I require out of a programming language, and like all the best changes of its kind, left me with the sentiment, "How did I ever go without this?" I remember a time when I felt that about Ruby (my previous programming paradigm shift), but using Ruby drives me crazy now.

Rust gets attention largely because of its memory safety guarantees, making it a potential replacement for C and C++ for systems that require the highest levels of performance and low-level control. However, I'm not a low-level programmer, so for me, the thing that Rust brings to the table is correctness. Others have described this quality as "fearless" (e.g. fearless concurrency), and while a bit jargony, I agree with the sentiment. What Rust has done for me is allowed me to let go of the immense worry I didn't even realize I had in other languages. In dynamically typed languages, or even statically typed languages with weak guarantees (e.g. Go), I had to program defensively, always worrying about things like a value being unexpectedly nil. In Rust, the guarantees I get from reaching the simple goal of "it compiles" gives me more confidence in the correctness of my program than a full-blown test suite ever did in Ruby.

Rust's value is not just that it's statically typed. There are many languages that are that. Rust's value is that it brings all the benefits of static typing without sacrificing the expressive, natural feeling of writing code in a language like Ruby. I get the same feeling of freedom to design and explore that I felt with Ruby, but with essentially none of the dangers. This is largely because of Rust's fantastic type system. Algebraic data types—specifically Rust's enums—are something I simply cannot imagine programming without at this point. Many of the other great parts of Rust are not new ideas, but Rust melds so many great ideas from different languages together cohesively that it feels like you're getting the best of every world. The only other mainstream language I know of with a comparable type system to Rust is Haskell, but Rust doesn't force me into a purely functional world, and provides a lot of other great benefits Haskell doesn't.

The lows

I'm happy to report that the two specific lows I mentioned last year are now mostly resolved.

The big pain point of doing serialization on stable Rust was resolved in Rust 1.15 when "Macros 1.1" was stabilized. This allows any crate to do automatic implementations of traits using custom derive annotations. We now have a 1.0 version of Serde, which has matured into an absolutely fantastic serialization library. The only minor downside to the custom derive feature is that it is limited to generating implementations based on type declarations and does not offer a full-featured procedural macro system. The procedural macro system is being revamped, however, and is already partially available in the nightly compiler as "Macros 2.0."

The other specific pain point I mentioned last year, about not having a stable, robust crypto library has improved as ring has matured and become the de facto crate for crypto in Rust. It's still not 1.0 and still not audited, so we're not quite there yet. In my work I also frequently need to create and manage X.509 certificates, and there is still nothing in Rust that does that yet. We don't have a 1.0, audited, pure-Rust TLS implementation yet, but rustls is on its way, and unlike last year, we also have native-tls which greatly improves the TLS story on macOS and Windows.

Interestingly, there are more lows this year than there were last year. Not only are there more, but despite my perhaps-overzealous love of Rust, I actually feel more negative about the language this year. My negative feelings are not because anything in Rust is bad. It's because of the things that are on the horizon that we don't have yet.

Rust the language has been 1.0 since May 2015. The problem is that 1.0 only means that what's there is stable. It doesn't mean that it is featureful enough to write anything you might want, realistically. It doesn't mean that the ecosystem of libraries and supporting tools is featureful and easy enough to use that you will convince non-early adopters to try it out or use it for real work. This, by itself, is largely the same sentiment as last year: Rust the ecosystem is still just too young and immature for a lot of use cases.

The thing that's worse this year is that I've come much further in the development of my Rust programs, to the point where finishing things is blocked by certain features of Rust not being implemented yet. While I don't have to worry about things I write today breaking in some future version of Rust, the critical problem is this:

Knowing what features are coming, the APIs I would write when they're stable are significantly different than how I'd write them today.

This knowledge of what is coming, but isn't here yet, completely paralyzes me. I'm not going to declare any of my libraries 1.0 when I know for sure that I will make breaking changes to them once a feature I wanted but didn't have previously becomes available.

Here is a list of specific features that are either implemented but unstable, or still in the RFC process and not even implemented. The stabilization of each of these would change the design of at least one piece of code in one of my applications or libraries:

All of these are language features, and say nothing of the huge amount of unstable crates that would need to be 1.0, many of which themselves are blocked on unstable or nonexistent features.

Right now, The most discussed of these features within the Rust community is asynchronous I/O. For my use cases, this is the current state of things:

  • Anything that uses HTTP needs to be asynchronous.
  • This requires a 1.0 version of Hyper.
  • Hyper, in turn, requires a 1.0 version of the Tokio stack.
  • Tokio, in turn, requires a 1.0 version of futures.
  • Futures, in turn, require impl Trait for realistic adoption.
  • Futures are likely to be migrated to the standard library.
  • Even with all of the above 1.0 and stabilized, async/await is also needed to make async APIs ergonomic enough to consider stabilizing.

Even this particular chain of dependencies is going to take a while, and this is the stuff on my wishlist that the Rust team has prioritized most highly. It's likely going to be multiple years before all this stuff is done.

I'd like to make it very clear that the Rust teams and the Rust community are not doing anything wrong. All of the things I've mentioned here are long since known as desired by the Rust team, and there's a plan to get there. All of these things are making progress, and a lot of very smart and hard working people are making it happen.

My negative feelings are quite simply because of the paralysis I feel knowing how different Rust will be once we have all these things, and having no recourse but to simply continue waiting, contributing to discussions and generally staying involved. It wouldn't be so bad if I could continue using other languages in the meantime and consider Rust something I'd consider picking up again in a few years. The good parts of Rust, even right now, are so good that I have trouble bringing myself to go back to any other language I know. So I'm trapped in this limbo between a crippled version of the language I want, and this fantastic version of the language I know is coming. I feel like a pouty little child writing this, but this is honestly how I feel right now.

Should you use Rust?

It's always hard to generalize when the answer is nuanced, but if I had to pick an absolute yes or no for everyone, I'd have to say no. Rust is most certainly not ready for massive, widespread adoption. I can't confidently claim that yes, it will work well for whatever you want to do with it, as I could say for Go, Java, C++, etc.

What I can say confidently is that there will be a time when I will unequivocably say, "Yes, you should use Rust." The only reason I'm not saying yes today is because Rust is still young and its foundational pieces are still being built. Everything that is in Rust today is awesome, and for many use cases it is already enough. If you value the correctness of your programs over delivering quickly with minimal investment (i.e. the "always be shipping" mentality), you will already benefit from using Rust today.

That said, if you try to build anything significantly complex in Rust right now, I think you're likely to come across at least one place where you're unable to do something because the language doesn't support it yet. This may be acceptable for building an application, where you're the only consumer of your API, but for a library it can be a major blocker. Of course, even this may be less of an issue if you are more willing to stabilize code you know for sure will have breaking changes in the near-to-medium future than I am.

My current, totally unscientific estimate is that roughly two years from now, I will be able to recommend that people choose Rust for their next project, full stop.

The relationship between async libraries in Rust

After all the recent announcements and hype about these async libraries, I was still a little confused about what each of these crates does and how they relate to each other. The crates I'm talking about are Futures, MIO, Tokio, and to a lesser extent Hyper and even Iron. Futures and MIO were especially confusing considering that there are also (or were, at least) several futures-foo crates and tokio-foo crates. After reading a bit more, I think I understand how they all relate now, so I wanted to share the knowledge (and please correct me if I'm wrong!)

Futures contains primitives for general purpose non-blocking computation, not necessarily specific to IO. The most important types here are the Future trait, which represents a single non-blocking computation, and Stream, which is like an iterator that yields a sequence of non-blocking computations. All the related futures-foo crates that were in the repo when it was first announced seem to have been renamed to tokio-foo and moved into the tokio-rs organization on GitHub. Most of them were just examples of how Futures could be used as the underlying mechanism for a few different purposes.

MIO contains primitives for building cross-platform asynchronous IO systems, generally focused around network IO.

Tokio (as the overarching project) marries Futures and MIO to provide asynchronous IO using the Futures APIs. Tokio is split up into several crates, which are, roughly in order from lowest level to highest level abstractions: tokio-core, tokio-service, tokio-proto, and the currently vaporware tokio. tokio-core has the low level guts of asynchronous IO. tokio-service contains the the Service trait, which similar to the futures crate's Future and Stream traits, is the central abstraction that the project provides for writing composable network programs. tokio-proto provides additional types that are helpful for implementing a network protocol such as HTTP. Finally, the crate actually called tokio will provide a higher level API that combines the features of the lower level crates. This is the crate that most of us will use when we want to write an asynchronous network service. The other crates exist separately just as a nice separation of concerns and to allow programs with more specific requirements to cherry-pick only the functionality they need. The tokio crate itself does not exist at the time I'm writing this because the lower level building blocks are still under heavy development and the APIs are not finalized. The other tokio-foo projects in the tokio-rs GitHub organization are either helpers types for specific use cases or examples of how you would build a network service using Tokio.

For those of us writing HTTP clients and servers, Hyper is the HTTP library we've come to know and love. Hyper was originally synchronous, but since MIO's initial release has been undergoing some major architectural changes to switch to an asynchronous model. According to Carl Lerche's Tokio announcement post, Hyper is in the process of moving its async implementation to build on top of Tokio instead of MIO directly.

And last but not least, Iron is a higher level web development framework built on Hyper. It's one of the more popular frameworks of its kind currently, though development activity has been very quiet for several months now. It's not clear to me whether or not the primary authors are still working on the project, whether they have run out of time and need help maintaining it, whether it's intentionally abandoned, or whether they're simply waiting for all these lower level components to stabilize before revising Iron's own APIs to use the Futures/MIO/Tokio/Hyper stack. Whether or not Iron becomes a framework that uses this stack, surely a web development framework using this stack will materialize sooner rather than later!

The highs and lows of Rust

I really like programming. I find programming languages very interesting. I've learned a lot of them over the years. Some I like and some I don't. Once in a while, I learn a new language that makes a lightbulb go on for me. One that changes the way I think about programming, helps me mature as a programmer, and becomes my default for most if not all new projects. The first time that happened to me was when I learned Ruby around 2009. Over the last year and a half or so, it has happened again with Rust. I love Rust so much that using anything else now drives me nuts.

The highs

What's so great about Rust? In short, it greatly increases my confidence that my programs are going to do what I want them to do without limiting expressiveness. My programming background is with almost entirely dynamic languages, as I've mostly worked on things for the web and other high level applications. People who only know dynamic languages, or people who escaped to them from the likes of Java, are afraid of static typing. They find it unnecessarily restrictive and ultimately not helpful. But in my experience, programs in dynamic languages are riddled with subtle bugs. Lack of static typing results in huge numbers of type errors all the time. In Ruby, you see "NoMethodError: undefined method 'foo' for nil:NilClass" so often that you become numb to it. You do this all in the name of "speed" and "productivity." In some cases, you do end up writing more terse code. But you end up with code that you can't trust. You write libraries that consumers will break in ways that you can't prevent. You can't write a library that adequately protects users from themselves.

With Rust, I have learned that a rich type system is a beautiful thing. There's a strong culture of automated testing in the Ruby community, and I was once religious about it just as many people in that community are. But in Rust, I have realized that so much of what you agonize to verify in a dynamic language is handled for you by static type checking and a good compiler. Testing is still a requirement, but you really only have to cover real logic, not the types of things that so many tests do in programs for dynamic languages. Rust's type system is not restrictive. It's very expressive, and typing out a few extra words provides guarantees that data is what you think it is and provides greatly improved clarity when reading and reviewing code. Every time I get a compiler error in Rust, I feel satisfied, not frustrated, because very often it's caught something that would have been a run time error that I might never have noticed in a Ruby program.

I'm not alone in feeling the pain of dynamic languages like Ruby for building large and complicated programs. In the last few years, many people have embraced Go as their new language of choice. It's been a very popular language for people coming from Ruby, Python, and JavaScript, because for many people, it provides the same feeling of "speed" and "productivity" that made dynamic languages attractive in the first place, but also provides additional benefits that static typing and ahead-of-time compilation bring like more reliable code and static binaries that you can just drop on a server and run without needing a language runtime installed.

Personally, I am not a fan of Go. For whatever reason, I actually find it much harder to read than Rust, whereas people often say that Rust is hard to read. General opinion seems to be that Go is easier to learn and that you can largely hit the ground running with it. Rust has a reputation of having a much higher learning curve, which of course affects its popularity. I really haven't been able to understand why, but somehow Rust just clicks with my mental model of programming better than Go does, and though it was indeed difficult to learn, once I have, I feel that it was worth all the time it took and then some. Go also doesn't appeal to me because I don't think it does enough to advance the state of the art. That is, it doesn't provide that much more than what dynamic languages already do. It has a mediocre type system which can be completely subverted with things like empty interfaces. And it has a nil value, which is a completely insane thing to have in a modern language. Escaping the pain of nil is one of the best parts about leaving Ruby. (See my previous post, Option types and Ruby for more on that.) Go is also not a safe language. Although many are fond of its use of channels for managing concurrent programs, Go knows only shared mutable state. It even comes with a data race detector to help debug your concurrent programs. To me, that is a sign of a fundamental error in the design of the language. In Rust, data races are not possible (assuming you use only safe code) because of its brilliant concept of memory ownership and borrowing.

I won't go over all the bullet points of the features and strengths of Rust, since they are well documented on the Rust website and various other articles on the topic. While the language is described as a "systems language" and is generally intended for lower level applications, my domain is in higher level applications, and I've found Rust to be just as suitable and strong for my purposes. It really is a fantastic general purpose language.

The lows

Being a Rust programmer is not all enjoyable, however. There are some major pain points in Rust so far. They largely have to do with the language and its ecosystem being very young, having reached 1.0 status less than a year ago, in May 2015. The most obvious issue with the ecosystem is simply the lack of libraries and the immaturity of existing ones. In older and more established languages, there is a library for just about anything you want, and for the most common needs, there is a library that is battle-tested and trusted. In Rust, very few libraries are trustworthy. Most Rust software lives on the bleeding edge. While this is frustrating when you just want to get something done, it is also exciting because building some of these needed libraries yourself means you're making a huge contribution to the ecosystem. I'm trying to help pave the way myself by never saying, "I'm not going to build X in Rust, because it requires libraries that do Y and Z, which don't exist yet." Instead, I start on X anyway, and build Y and Z myself too, giving everything back to the community.

The excitement of the new and the excitement of contributing aside, it really is painful to just get things done in Rust a lot of the time. The worst pain points for me so far are 1) lack of a feature-complete, trustworthy cryptography library and 2) lack of a feature-complete, stable serialization library. I'll address these two points in a bit more detail.

With all of the highly publicized security vulnerabilities in the last few years (roughly beginning with OpenSSL's Heartbleed, and up to and including the recent glibc DNS resolver vulnerability) the topic of the danger of programs written in C has come up many times. The general consensus is that it's just not possible to write a large, complicated program in C safely. Even without the history of bad code quality and neglect in some of these high profile C libraries, C is a language driven by the fear of making a critical mistake that is all too easy to make. Naturally, people discuss Rust as being the probable candidate for the next generation of these types of libraries. While it's not really reasonable for now to rewrite things like OpenSSL in Rust simply because of how much work it is and how many programs specifically target this older C software, not having crypto libraries in Rust makes it very difficult to write Rust programs without binding to the same C libraries we're trying to get rid of. This is one area where I envy Go, which has pure-Go crypto libraries that are suitable for production use. There are a few crypto libraries in Rust, most notably ring and Sodium Oxide, but there is currently nothing production-ready or even usable in Rust for TLS and X.509 certificate generation and management. The best you can do for the latter right now is shell out to openssl. For more discussion on this topic, see my post on the Rust Internals forum.

The second major pain point, and the most massive one in my experience, is serialization and deserialization. That is, converting between Rust types and formats like JSON and YAML. Originally, serialization was built into the Rust compiler, but as Rust prepared for 1.0, many pieces that were once part of the compiler or the language itself were removed or extracted to external libraries in order to keep the language and the standard library as small as possible. What was once the built-in "serialize" crate (a crate is Rust's term for an individual unit of compilation; essentially a package or library) now exists as the external crate rustc-serialize. It was decided that the approach taken by that crate was not the best way to do things long term, and that it would be punted until a better crate could be created, eventually becoming the de facto serialization library for the Rust community. As a compromise, the Rust compiler has special knowledge of rustc-serialize, and allows you to automatically make your Rust types serializable by putting a special attribute above them in the source code. However, with rustc-serialize being sort of deprecated, it lacks some important features, like being able to customize certain details of how a type is serialized. For example, there is no way with rustc-serialize's automatic serialization to have a Rust type with snake cased fields serialize to JSON with lower camel cased fields.

Today we have Serde, a great serialization library by Erick Tryzelaar, which is likely to be the de facto serialization library everyone is hoping for. Unfortunately, Serde does not enjoy the same special treatment that the compiler gives rustc-serialize, and so the story for using Serde is much more complicated. In order to do the same type of automatic serialization that rustc-serialize does, Serde has a sub-crate that works as compiler plugin to do code generation, turning a special attribute into a full implementation just like rustc-serialize. The rub is that compiler plugins are an unstable feature of Rust, meaning they are only available in Rust's nightly releases, and not on stable Rust 1.x or even the beta version. It is possible to use Serde's automatic serialization on stable Rust, but it requires some clever yet very hacky indirection. One of the crates under the serde organization on GitHub is Syntex, which is literally a fork of the Rust compiler's private "libsyntax" crate that compiles on stable Rust. When using Serde on stable Rust, the Serde code generation crate uses Syntex instead of the real Rust compiler to generate the code with serialization implemented in a separate build step before your program is actually compiled. There are some ugly edges to this, including needing to have two copies of every source file in your program (one with the code to be processed by Serde + Syntex and one to include the file it generates). You also need to have a build script which manually specifies which source files need to go through this process. Because compiler plugins are in flux within the Rust compiler, there are frequent changes to libsyntax, each of which requires a new version of Syntex to be released, and inevitably causes a waterfall of broken programs down the dependency chain as the new versions of Rust and Syntex are rolled out. This happens every few weeks and it's a nightmare to keep up with. As a result, any program of non-trivial size that needs serialization is really better off just sticking to nightly Rust. But because not every library has nightly Rust in mind, sometimes you end up with dependencies that don't work well together, and you're stuck in a spiral of making trivial fixes to other people's libraries so you can get your own to compile. This whole issue is not something I would expect from a language that has reached 1.0 for something as necessary and ubiquitous as data serialization. Nick Cameron has been working on a new macro system for Rust which will eventually do what compiler plugins are being used for right now, but this new system doesn't even exist yet, so it will be quite some time before it makes it to stable Rust. The near future feels very bleak on this issue.

Should you use Rust?

Rust is a fantastic language that gets almost everything right, but its immaturity can make it hard to get your work done. So should you use it? I think it depends on what you might use it for, and what you value the most. If you use programs just to get your work done, and are most concerned with productivity, Rust is not ready for prime time yet. However, if you check back in a year or so, it will likely be a very good candidate for almost anyone. That said, there is a lot you can learn about good programming from learning Rust. There are fundamentals about the language that will not change that you can benefit from right now. If you have time to look into a new language simply because you're interested in getting better as a programmer, and you're forward thinking about how you might write programs in the future, Rust is a great choice. If you're like me, and the idea of a programming language you can really trust outweighs your desire to get things done quickly, or you are excited to help build the libraries that will be the foundation of Rust for years to come, you should stop what you're doing and learn Rust right now.

This article has been translated to Russian: Максимумы и минимумы Rust

Page 1