Back in the olden days, when software literally came from BigCo R&D departments, we managed to invent Unix, and the mouse, and GUIs, and Ethernet, and TCP/IP, and a ton of other stuff we all use constantly today. Those research divisions didn’t ship viable consumer products, though. Doug Englebart demoed a mouse-driven GUI in 1968, but you couldn’t buy a home computer with a mouse-driven GUI until 1979, and they didn’t become commercially popular until the Macintosh in 1984.

Even years or decades of research wasn’t enough, and years (or decades!) of development work also needed to be done before the results was ready for people to use. Early literature about creating software, written by Fred Brooks and his peers, seems to contain the internalized view that both R and D are required. That’s not surprising, since R&D departments created most software back then, but we seem to have lost track of that connection.

Even though our jobs are descended from those R&D labs of yore, we somehow lost the industry job of “software researcher”, and only “software developer” remains. Instead, research happens in academia, where an argument and some pseudocode is all you need to publish a paper. In that world, development is effectively non-existent.

(I admit the division isn’t perfectly clear-cut. Sometimes academics will start companies around their research that create a product, or more likely get acquired to add a feature to a product. And sometimes Linus Torvalds will just build a new operating system, without doing any academic research on it, and it will get so popular everyone uses it. The point is that industry and academia have each publicly claimed one half of R&D while disowning the other.)

The broader separation of research and development into academia and industry is really unfortunate, because good software needs both research and development as inputs. If you don’t do any research, you can’t identify which parts will be hard (or impossible) until after it’s too late. You also won’t have a good idea of what parts are important until after you’ve put in most or all of the work to create the parts that don’t matter. If you don’t do development, you won’t ever have something robust enough that other people can use it successfully.

Meanwhile, on the other side, it feels like developers work hard to convince themselves there are no research aspects involved in their jobs. We call anything research-ish by another name, like “design”, “user experience”, “prototyping”, “de-risking”, “a spike”, and a lot of other funny euphemisms that avoid referring to the work as research. It seems like we’re trying to convince ourselves that we don’t do Research any more, because we are just Developers.

This cultural lack of clarity around research in software development spaces really hit hard for me this week, as I read yet another treatise on working with LLM-driven agents for development. The two most popular takes that I have seen are “these tools are a fundamental shift in the nature of software development” and “these tools change nothing about building software at all”. Then the two sides start screaming at each other about how the other side is delusional and time will prove them completely wrong, and I lose interest.

If we instead start from the premise that all software work requires research (where the problem space must be explored) and development (where solutions must be implemented and refined), there’s something hiding in the sometimes messy overlap between those two ideas that I’m not seeing come up in any discussions.

No one can take the output of software research and treat it like it’s the output of software development. Not Bell Labs, not Xerox PARC, not Microsoft middle managers, and not “solo founders managing a team of AI agents” today.

Unfortunately, seeing a prototype and becoming convinced it’s complete is not a new problem. It’s been the bane of software development possibly since the very beginning, when (apocryphally) a manager would review a mockup and conclude the project was now complete and could be shipped to customers immediately. Today, instead of telling that story as a joke, software developers have have somehow turned themselves into the boss from the joke, shouting that it’s time to ship the research prototype because it “looks finished”. How did we do this to ourselves?

It seems like, back when we always had to do all the work ourselves, it was harder for software developers to be confused this way. If a developer knows they skipped every validation and edge case, it’s much easier to realize it’s not finished. If an LLM agent says “here’s a comprehensive implementation”, without mentioning all the validations and edge cases it skipped, many (and possibly most) developers will not notice the parts that are missing.

This phenomenon is bad for a lot of reasons, including one reason you have probably already thought of: we’re going to get a lot more software claiming to be “comprehensive” and “fully implemented” when it’s really a partially finished prototype that’s full of holes.

In a world full of research prototypes being pitched as completed development work, life is about to get worse for everyone who uses software. The docs are even more wildly wrong than they were before, customer support is telling you that your problem is solved by a feature that doesn’t exist, and company leadership is so excited they are planning to fire as many humans as possible so they can have more of it.

I don’t want worse software! The software we already have is mostly terrible. Not only much worse software, but also much more of it, is pretty much my worst case scenario. What I actually want is better software, even if that means less of it.

Unfortunately, instead of making better software, software developers have decided to become the butt of their own joke, shipping software that doesn’t work, with a footnote that says they know it doesn’t work but they are still shipping it. I don’t see any way to stop it, but I hate it anyway.

In all three cases, Ruby Central found no evidence of compromised end user data, accounts, gems, or infrastructure availability. I hope this can conclusively put to rest the idea that I have any remaining access to the RubyGems.org production systems, or that I caused any harm to the RubyGems.org service at any time.

I also appreciate that Ruby Central is taking its share of responsibility, recognizing that its lack of communication with the former maintainers (including me) created confusion and frustration that contributed, in part, to how we ended up where we are today.

Ruby Central board members Freedom, Brandon, and Ran state that their intent is now to work towards an amicable resolution. I salute their new commitment, and would like to do my part to help the RubyGems community move past these unfortunate events, with a resolution that puts the dispute fully behind us, and allows all of us to move forward.

For my part, despite my claims against Ruby Central, and the threats they have directed against me, I am willing to completely settle all of my disputes with them, and pledge to take no legal action against Ruby Central regarding any of their actions prior to today.

In exchange, I am requesting two things.

First, I am asking Ruby Central to drop their legal threats, including releasing their claims against me and reimbursing my legal costs. Those costs arise from Ruby Central’s actions, including litigation threats, other escalations, and most recently contacting law enforcement. In addition to forcing me to retain counsel, these actions caused considerable stress and disruption. I am willing to provide invoices to ensure the reimbursement precisely matches only my actual costs.

Second, I am asking Ruby Central lay our disagreement to rest with a public statement acknowledging that I did no harm to the RubyGems.org service.

If Ruby Central fully drops their legal claims, and states I did not harm the RubyGems.org service, I would consider our disagreement amicably settled.

It’s more complicated than it sounds

Hello, and welcome to How To Install A Gem. My name is André Arko, and I go by @indirect on all the internet services. You might know me from being 1/3 of the team that shipped Bundler 1.0, or perhaps the 10+ years I spent trying to keep RubyGems.org up and running for everyone to use.

More recently, I’ve been working on new projects: rv, a CLI to install Ruby versions and gems at unprecedented speeds, and gem.coop, a community gem server designed from the ground up so Bundler and rv can install gems faster and more securely than ever before.

So, with that introduction out of the way, let’s get started: do you know how to install a gem? Okay, that’s great! You can come up and give this talk instead of me. I’ll just sit over here while you write the rest of this post.

Slightly more seriously, do you know how gem install converts the name that you give it into a URL to download a .gem file? It’s called the “compact index”, and we’ll see how it works very soon.

Next, who in the audience knows how to unpack a .gem file? Do you know what format .gem files use, and what’s inside them? We’ll look at gem structure and gemspec files as well.

Then, do you know where to put the files from inside the gem? Where do all of these files and directories get put on disk so we can use them later? Does anyone know off the top of their head?

Once those files have been unpacked into the correct places, the last thing we need to know is how to require them. How do these unpacked files on disk get found by Ruby, so you can require "rails" and have that actually work?

This exercise was mostly to show that using gems every day actually skips over most of the way they work underneath. So let’s look at what a gem is, and examine how they work. By the end of this talk, you’ll know what’s inside a gem, how RubyGems figures out what to download, and where and how that download gets installed so you can use it.

And if you already everything we just talked about, please feel free to go straight to rv.dev and start sending us pull requests!

How does a name become a .gem URL?

First, we’re going to look at how the name of a gem becomes a URL for a .gem file. Let’s use rails as our example. Historically, there have been at least five or six different ways to look up information about a gem based on its name, but today there is one canonical way: the compact index. It’s so simple that you can do it yourself using curl. Just run curl https://gem.coop/info/rails, and you’ll be able to read the exact output that every tool uses to look up the versions of a gem that exist. Each line in the file describes one version of the gem, so let’s look at one line.

❯ curl -s https://gem.coop/info/rails | tail -n 1
8.1.3 actioncable:= 8.1.3,actionmailbox:= 8.1.3,actionmailer:= 8.1.3,actionpack:= 8.1.3,actiontext:= 8.1.3,actionview:= 8.1.3,activejob:= 8.1.3,activemodel:= 8.1.3,activerecord:= 8.1.3,activestorage:= 8.1.3,activesupport:= 8.1.3,bundler:>= 1.15.0,railties:= 8.1.3|checksum:6d017ba5348c98fc909753a8169b21d44de14d2a0b92d140d1a966834c3c9cd3,ruby:>= 3.2.0,rubygems:>= 1.8.11

We can break down that line with split(" "), and tackle each part one at a time.

First, 8.1.3. That’s the version of rails that this line is about. So we now know for sure that rails 8.1.3 exists.

Next, a list of dependencies. The rails gem (version 8.1.3) declares dependencies on a bunch of other gems: actioncable, actionmailbox, actionmailer, actionpack, actiontext, actionview, activejob, activemodel, activerecord, activestorage, activesupport, bundler, and railties. Each dependency has a version requirement attached, and for almost every gem it is exactly version 8.1.3, and only version 8.1.3. For bundler, Rails is a little bit more flexible, and allows any version 1.15.0 and up.

The final section contains a checksum, a ruby requirement, and a rubygems requirement. The checksum is a sha256 hash of the .gem file that contains the gem, so after we download the gem we can check to make sure we have the right file by comparing that checksum.

For this version of Rails, the required Ruby version is 3.2.0 or greater, and the required RubyGems version is 1.8.11 or greater. It’s up to the client to do something with that information, but hopefully you’ll see an error if you are using Ruby or RubyGems that’s too old.

Great! So now we know the important information: Rails version 8.1.3 is real, and strong, and is our friend. We can download it, and check the checksum against the checksum we were given in the info file line. Let’s do that now:

❯ curl -sO https://gem.coop/gems/rails-8.1.3.gem
❯ sha256sum rails-8.1.3.gem
6d017ba5348c98fc909753a8169b21d44de14d2a0b92d140d1a966834c3c9cd3  rails-8.1.3.gem

Notice that the checksum produced by sha256sum exactly matches the checksum we previously saw in our line from the info file: 6d017ba5348c98fc909753a8169b21d44de14d2a0b92d140d1a966834c3c9cd3. That lets us know that we got the right file, and there were no network or disk errors.

What exactly is in a gem?

Now that we have the gem, we can investigate: what exactly is inside a gem? At this point, we’re going to pivot from the rails gem to the railties gem. There’s a good reason for that, and the reason is… the rails gem doesn’t actually have any files in it. So it’s a bad example. In order to show off what a gem looks like when it has files in it, we’ll use railties-8.1.3.gem instead.

So, we have our .gem file downloaded with curl. What do we do now? The first piece of secret knowledge that we need: gems are tarballs. That means we can open them up with regular old tar. Let’s try it.

❯ tar xfvz railties-8.1.3.gem
x metadata.gz
x data.tar.gz
x checksums.yaml.gz

So what’s inside the .gem tarball is… another tarball. And also two gzipped files. Let’s look at the files first.

❯ gzcat checksums.yaml.gz
---
SHA256:
  metadata.gz: 8326ab6cc8e325055394ebd19c41d895e9ebd48e4752ec90d5c4675935516e6e
  data.tar.gz: 4152d8f55ae639d899f1cb6c54e1e93bb158bb76026b253482c5ae0343ac5aec
SHA512:
  metadata.gz: f6aa3390b6b1699255f1dbc6c6f24c6d9c18d3bfa48f10b6d720595384b4d1bb26f92232adb7011d3b1e7e977ca775cd253a12a135fe83eaa21e10dd0f14f779
  data.tar.gz: a001cebc5b97f627336a3e8d394c4ecac4a5d2b9e62c82de3b484470a58deac7c8ff0a8e8b497843386b1639c9cbfdfabee2cc7b2d483469e00a0b01da6bd41d

As you might expect from its name, the checksums.yaml.gz file is a gzipped YAML file, containing checksums for the other two files. It’s maybe a bit silly to have multiple layers of checksumming here, but it does confirm that the outer layer of tarball and zip was removed without any errors.

Okay, so what’s inside metadata.gz? The answer is… Ruby, sort of. It’s a YAML-serialized instance of the Gem::Specification class. We can see exactly what was put into this object at the time the gem was built.

After snipping out the YAML that lists the dependencies (which we already looked at, because they are included in the info file), what’s left is some relatively simple information about the gem. Author, author’s email, description, homepage, license, various URLs.

❯ gzcat metadata.gz
--- !ruby/object:Gem::Specification
name: railties
version: !ruby/object:Gem::Version
  version: 8.1.3
platform: ruby
bindir: exe
executables:
- rails
require_paths:
- lib
authors:
- David Heinemeier Hansson
summary: Tools for creating, working with, and running Rails applications.
description: 'Rails internals: application bootup, plugins, generators, and rake tasks.'
email: david@loudthinking.com
homepage: https://rubyonrails.org
licenses:
- MIT
metadata:
  bug_tracker_uri: https://github.com/rails/rails/issues
  changelog_uri: https://github.com/rails/rails/blob/v8.1.3/railties/CHANGELOG.md
  documentation_uri: https://api.rubyonrails.org/v8.1.3/
  mailing_list_uri: https://discuss.rubyonrails.org/c/rubyonrails-talk
  source_code_uri: https://github.com/rails/rails/tree/v8.1.3/railties
  rubygems_mfa_required: 'true'
rubygems_version: 4.0.6
specification_version: 4
files:
- CHANGELOG.md
- MIT-LICENSE
- RDOC_MAIN.md
- README.rdoc
- exe/rails
- lib/minitest/rails_plugin.rb
- lib/rails.rb
[...]

For the purposes of installing and using the gem, we care about exactly six pieces of information: name, version, platform, bindir, executables, and require_paths.

We’re going to combine those items with the files in the remaining data.tar.gz file to get our unpacked and installed gem.

What gets unpacked, and where?

Now that we know what’s in the gem specification, let’s look at what’s inside the data tarball. It matches up very closely with the long list of entries in the files array in the gemspec.

❯ mkdir railties-8.1.3
❯ tar xfvz data.tar.gz -C railties-8.1.3
x CHANGELOG.md
x MIT-LICENSE
x RDOC_MAIN.md
x README.rdoc
x exe/rails
x lib/minitest/rails_plugin.rb
x lib/rails.rb
[...]

So now we have a bunch of files. Where are we going to put these files? Enter: the magic of RubyGems. The scheme that RubyGems has come up with is largely shaped by the constraints of how Ruby finds files to require, which we’re going to look at soon. For now, it is enough for us to know that RubyGems keeps track of a list of directories, a lot like the way $PATH works for your shell to find commands to run. To find the current directory, you can run ruby -e 'puts Gem.dir. Here’s what that looks like:

❯ ruby -e 'puts Gem.dir'
/Users/andre/.gem/ruby/4.0.0
❯ ls ~/.gem/ruby/4.0.0 | xargs -L1 echo
bin
build_info
cache
doc
extensions
gems
plugins
specifications

From this list, we can see that RubyGems organizes its own files into a few directories. To install a gem, we’re going to need to put the files we have into each of those directories, with specific paths and filenames.

Just to recap, the files we need to place somewhere are:

• railties-8.1.3.gem (the .gem file itself)
• metadata.gz (the YAML Gem::Specification object from inside the gem)
• the unpacked data.tar.gz files (the contents of the gem)

So let’s move the files into the directories we see RubyGems offers.

First, cache the .gem file so RubyGems doesn’t need to download it again later:

mv railties-8.1.3.gem ~/.gem/ruby/4.0.0/cache/

Then, add the gem specification so that RubyGems will be able to find it. There’s a small twist here, which is that the specifications directory doesn’t contain YAML files, it contains Ruby files. So we also need to convert the YAML file back into a Ruby object, and then write out the Ruby code to create that object into a file that RubyGems can load later.

❯ gunzip metadata.gz
❯ ruby -ryaml -e 'puts YAML.unsafe_load_file("metadata").to_ruby' > ~/.gem/ruby/4.0.0/specifications/railties-8.1.3.specification

Next, we need to put the files that make up the contents of the gem into the gems/ directory.

❯ mv railties-8.1.3 ~/.gem/ruby/4.0.0/gems/
❯ ls ~/.gem/ruby/4.0.0/gems/railties-8.1.3
CHANGELOG.md
exe
lib
MIT-LICENSE
RDOC_MAIN.md
README.rdoc

One more thing we need to do: set up the executables provided by the gem.

You can check out the files that RubyGems generates by looking in ~/.gem/ruby/4.0.0/bin, but for our purposes we just need to tell RubyGems what gem and executable it needs to run, so we can do that:

❯ cat <<EOF > ~/.gem/ruby/4.0.0/bin/rails
#!/usr/bin/env ruby
require "rubygems"
Gem.activate_and_load_bin_path("railties", "rails")
EOF
❯ chmod +x ~/.gem/ruby/4.0.0/bin/rails

And with that, we’ve installed the gem! You can run the rails file that we just created to prove it:

❯ ~/.gem/ruby/4.0.0/bin/rails
Usage:
  rails COMMAND [options]

You must specify a command:

  new          Create a new Rails application. "rails new my_app" creates a
               new application called MyApp in "./my_app"

As we wrap up here, there are three aspects of gems that we haven’t touched on at all: docs, extensions, and plugins. We don’t have time to talk about them today in this meetup talk slot. Hopefully a future (longer) version of this talk will have space to include all of those things, because they are all super interesting, I promise.

In the meantime, I will have to direct you to the docs for RDoc to learn more about docs, to the source code of rv or RubyGems itself if you want to learn more about gem extensions and plugins.

How does require find a gem?

There’s one last thing to figure out before we wrap up: how does require find a gem for us to be able to use it? To explain that, we’ll have to drop down to some basic Ruby, and then look at the ways that RubyGems monkeypatches Ruby’s basic require to make it possible to have gems with versions.

The first thing to know about require is that it works exactly like $PATH does in your shell. There’s a global Ruby variable named $LOAD_PATH, and it’s an array of paths on disk. When you try to require something, Ruby goes and looks inside each of those paths to see if the thing you asked for is there.

You can test this out for yourself in just a few seconds! Let’s try it.

❯ mkdir lib
❯ echo 'puts "this is some ruby"' > lib/my-file.rb
❯ ruby -Ilib -e 'puts $LOAD_PATH.first; require "my-file"'
/Users/andre/lib
this is some ruby

The Ruby CLI flag -I lets you add directories to the $LOAD_PATH variable, and then the require function looks inside that directory to find a file with the name that you gave to require. No magic, just a list to check against for files on disk.

Now that you understand how the $LOAD_PATH variable makes require work, how does RubyGems work? You can’t just put ten different versions of rake into the $LOAD_PATH and expect require to still work. RubyGems handles multiple versions of the same rake.rb file by monkeypatching require.

Let’s look at what happens when we require "rails", which is a file located inside the railties gem that we just installed. RubyGems starts by looking at all of the gem specifications, including the one we saved earlier. In each specification, it combines the name and version with the values in require_paths to come up with a path on disk.

So for our just-installed railties gem, that would mean a path of: ~/.gem/ruby/4.0.0/gems/railties-8.1.3/lib. RubyGems knows that directory contains a file named rails.rb, so it is a candidate to be “activated”, which is what RubyGems calls it when a gem is added to your $LOAD_PATH.

As long as internal bookkeeping shows that no other versions of railties have already been added to the $LOAD_PATH, we’re good! RubyGems adds this specific directory to the $LOAD_PATH, and delegates to the original implementation of require. Require finds the file at ~/.gem/ruby/4.0.0/gems/railties-8.1.3/lib/rails.rb, reads it, and evaluates it.

Gem installed, congratulations

With that, we’ve done it! We have found, downloaded, unpacked, and installed a gem so that Ruby is able to run a command and load ruby files, without ever touching the gem install command.

If you’re interested in contributing to an open source project that works a lot with gems, we would love to work with you on rv, where we are working to create the fastest Ruby and gem manager in the world.

And of course, if your company could use faster, easier, or more secure gems for developers, for CI, and for production deployments, we can help. We’d love to talk to you and you can find our contact information at spinel.coop.

Last year, Ruby Central flipped that successful formula on its head. They now claim ownership of both Bundler and RubyGems, but refuse to provide governance. Ruby Central now claims sole control over all code and decisions, despite paying for only a few percent of the work required to create and sustain the projects across 22 years. Instead of providing stable and predictable processes, Ruby Central suddenly hijacked the Bundler and RubyGems codebases away from the existing maintainers, shut out the community, and started issuing the threats to sue.

When confronted by the former maintainers after the hijacking, Marty Haught of Ruby Central stated (in a recorded video call) on September 17 that “yeah, we shouldn’t have changed that”. On September 18, Marty went on to write:

In the past, we’ve made the mistake of conflating ownership of the code with ownership of the infra, and vice versa, and we’d like to straighten this out so that we aren’t put in a legal bind that requires us to take control of the entire codebase when, we all agree, that is not proper or correct given the existing model.

In the words of Ruby Central itself, “we all agree, [taking control of the entire codebase] is not proper or correct.” Since the beginning of this conflict, Ruby Central has privately admitted it was wrong to hijack the GitHub organization and steal the repos, but has refused to acknowledge this in public. Unfortunately, despite privately admitting their actions were wrong, Ruby Central has publicly continued to dig their hole deeper. Instead of owning up to their mistake, they secretly negotiated a deal with Matz for ruby-core to take over the stolen RubyGems and Bundler repository, further violating the project governance policies.

If this situation were just about me personally, I could believe it sprang from from individual disagreements. Ruby Central claims they had good reasons to unilaterally kick me out of the project, even though I don’t think their claims hold water. With that said, regardless of what you think about me personally, the other five long-term maintainers have never gotten any explanation of why they were suddenly kicked out or bypassed entirely, all in violation of existing project governance.

In her only public interview about the situation, Ruby Central Executive Director Shan Cureton defended stealing Bundler from its team of fifteen years by saying the removed team “didn’t need to have the story, and it wasn’t their story to have”. Ruby Central has made their position clear: if they steal your project, you are not entitled to know their reasons, and neither is anyone else. There is nothing “community-oriented” about stealing the most-used gem in Ruby and refusing to share your reasons with the community.

Despite Ruby Central’s unacceptable treatment of both projects and maintainers, the former RubyGems and Bundler team said we want to move Ruby forward. We offered Ruby Central a path to move past their illegitimate GitHub takeover, past their vicious personal attacks, and past their threats to sue us.

It has been four months since we made that offer, and Ruby Central has not accepted.

While declining to accept our offer, Ruby Central has nonetheless found the time to propose new governance documents for RubyGems. In those documents, they explicitly require existing maintainers approve adding or removing team members. That rule was already present in the previous governance, and is the exact rule that Ruby Central violated to execute their takeover. When asked why they violated the previous governance, and why the new governance would be any more trustworthy, Ruby Central refused to respond substantively, and then the question itself was hidden by marking it “off topic”.

Instead of working to resolve the situation, Ruby Central has spent 4 months rejecting requests for an explanation, while repeatedly threatening to sue me personally. After Ruby Central suddenly took over the Bundler repo, I sent them a standard trademark notice. They replied with a threat to sue me. When I later informed Ruby Central I had learned they violated state employment law, they simply replied with the same threat to sue me again. They are threatening to sue me for “hacking” them, despite their own analysis publicly concluding “no evidence that user data or production operations were harmed”.

Without seeking common ground, or even looking for some sort of resolution we can just live with and move on from, Ruby Central has offered all of us — nothing. Ruby Central has made no offer in reply to outreach from the other five maintainers. To me, after four grueling months of private “negotiation”, their entire offer is nothing more than to refrain from suing. But only if I agree to everything that they want.

They say I must agree that I have no claim on the name Bundler, despite helping create it and leading the Bundler team for the last 15 years. They say I must agree I was paid legally and fairly, when California law clearly states I was not. They say I must agree that Ruby Central can take over open source projects they host, any time they feel like it, with no explanation, and no consequences.

I don’t agree.

Letting this situation stay unaddressed sets a dangerous precedent for all open source projects written in Ruby. Ruby Central has resolved nothing. Don’t let their delaying tactics convince you otherwise. The Ruby community cannot trust Ruby Central with control over our gems until there is accountability for destroying the very governance they were supposed to be providing.

Until accountability arrives, take action. Tell Ruby Central they owe everyone an explanation for violating the project governance around six long-term maintainers, not just me. Don’t sponsor, attend, or speak at RubyConf. Contribute to projects that aren’t controlled by Ruby Central.

The exiled maintainers are working on new projects, with a focus on clear governance, long-term financial sustainability, and community input: Join the gem.coop beta, and stop using RubyGems.org. Use jwl instead of RubyGems. Use rv or Ruby Butler instead of Bundler.

A better world is possible! Ruby Central might want to keep Ruby in the past, but we can work together to build Ruby a future.

As part of our quest to build a fast Ruby project tool, we’ve been hard at work on the next step of project management: installing gems. As we’ve learned over the last 15 years of working on Bundler and RubyGems, package managers are really complicated! It’s too much to try to copy all of rbenv, and ruby-build, and RubyGems, and Bundler, all at the same time.

Since we can’t ship everything at once, we spent some time discussing the first project management feature we should add after Ruby versions. Inspired by npm and orogene, we decided to build clean-install. Today, we’re releasing the rv clean-install command as part of rv version 0.4.

So, what is a clean install? In this case, clean means “from a clean slate”. You can use rv ci to install the packages your project needs after a fresh checkout, or before running your tests in CI. It’s useful by itself, and it’s also concrete step towards managing a project and its dependencies.

Even better, it lays a lot of the groundwork for future gem management functionality, including downloading, caching, and unpacking gems, compiling native gem extensions, and providing libraries that can be loaded by Bundler at runtime.

While we don’t (yet!) handle adding, removing, or updating gem versions, we’re extremely proud of the progress that we’ve made, and we’re looking forward to improving rv based on your feedback.

Try running brew install rv; rv clean-install today, and see how it goes. Is it fast? Slow? Are there errors? What do you want to see next? Let us know what you think.

While working on rv, there’s a specific question that has come up over and over again, in many different forms. In the simplest possible form, it boils down to:

What is the difference between rv exec and rv run? Why have both?

We haven’t finished implementing either rv exec or rv run yet, but every time one or the other comes up in a conversation, everything instantly becomes more confusing.

This post will summarize the history of exec and run in Bundler, npm, Cargo, and uv. Once we have the history laid out, we can take a look at what we plan to do in rv, and you can give us your feedback.

Bundler exec

Bundler manages project-specific packages, but not generally available “global” commands. Project-specific packages installed with Bundler can include their own commands.

While working on Bundler 1.0, we needed a standard way to do something new: run commands completely scoped inside a project, rather than scoped to the entire Ruby installation on the current machine. We tried both a wrapper command (bundle exec COMMAND) and generating dedicated scripts in the project’s bin/ directory. With binstubs, you could run bin/rake to get the project rake, and rake to get the global rake.

I personally preferred the binstub approach, but it was bundle exec that ultimately became the popular way to use commands inside your project. My theory is that it won because you can use it to run any command, including ruby, or bash, or anything else you want.

RubyGems exec

Somewhat confusingly (inspired by the npm exec command explained below) there is a separate gem exec command that is not related to Bundler and instead installs and runs a command from a package. RubyGems only manages global packages and commands, so gem exec is more of a convenience to make it easier to globally install and run a package with just one command.

npm run and exec

npm manages both project-specific and global packages, and can install any package so its commands are available either only within a project or globally across every project that uses the same version of Node.

The project-focused design of npm expects commands from project packages to be run by first adding the command to package.json in the script section, and the run via npm run SCRIPT. This is even more inconvenient than Bundler’s binstubs, and so I think there was pent-up demand to be able to “just run a command directly”. That was eventually provided by npm exec and its alias npx.

The npx COMMAND setup makes it very easy to run any command, whether the package is in the local project or not, whether a script is set up or not, and even whether the package is installed at all or not. Simply running the command is enough to install the needed package and run its command.

It’s especially helpful to have npx available when you need to create a new project by running an npm package, since it’s a huge pain to create a project and install a package and set up a script just so you can run the package to overwrite your project. The most popular example of this I am aware of is npx create-react-app, but it’s a popular idiom for many packages that contain commands.

Cargo run and install

Cargo is simalarly a package manager, but unlike Ruby and Node, project-level packages do not include commands, and package commands are installed globally. Library packages are added to a project with cargo add, while command packages are installed globally with cargo install. Once a package is installed globally, it can simply be available on your $PATH, and Cargo no longer has to be involved in running it.

The cargo run command is extremely limited in scope, and only allows you to build and run a binary created by your own project – it does not work to run commands from packages, either project-local or global.

uv exec and run

In uv, the exec command seems to be most strongly inspired by npm exec, including having its own short alias of uvx. The uv exec command is exclusively for running commands directly from packages, automatically installing them if necessary. To give an example, that means you can use uv exec github-backup to install and run the github-backup command from the Python package named github-backup, whether or not that packge is included in your current project.

Conversely, the uv run command is closer to bundle exec: it installs and configures Python, installs project packages if inside a project, and then runs a command from $PATH or runs a file. That means uv run can be used for both uv run bash to get a shell with only your project’s Python and packages, and can also be used as uv run myscript.py to run a script file directly.

Summary

bundle exec runs:

• commands created by your package
• commands from project packages (like bundle exec rails)
• commands from $PATH (like bundle exec bash)
• scripts from files (like bundle exec ./myscript.rb)

npm run runs:

• project-defined script commands (like npm run my-project-script), which can call:
  • commands from project packages (like generate_pdf.js)
  • commands from $PATH (like bash)
  • scripts from files (like ./myscript.js)

npm exec installs and runs:

• non-project commands from any package (like npx create-react app)

cargo run builds and runs:

• commands created by your package

uv run installs python and any project packages, then runs:

• commands from project packages (like uv run datasette)
• commands from $PATH (like uv run python)
• scripts from files (like uv run ./myscript.py)
• project-defined script commands (like uv run my-project-script), which can call:
  • commands from project packages (like github-backup)
  • commands from $PATH (like bash)
  • scripts from files (like ./myscript.py)

uv exec installs python and the named package, then runs:

• non-project commands from any package (like uv exec github-backup)

The question for rv

With all of that background now set up, what should rv exec do? What should rv run do?

To be the most like Bundler, we should use rv exec to run commands for a project. To be the most like npm and uv, we should use rv exec to install and run a single package with a command.

Today, we’re leaning towards an option that includes all of the useful functionality from every command above, and aligns Ruby tooling with JS tooling and Python tooling:

rv run installs ruby and any project packages, then runs:

• commands from project packages (like rv run rspec)
• commands from $PATH (like rv run bash)
• scripts from files (like rv run ./myscript.rb)
• project-defined script commands (like rv run my-project-script), which can call:
  • commands from project packages (like rspec)
  • commands from $PATH (like bash)
  • scripts from files (like ./myscript.rb)

rv exec installs ruby and the named package, then runs:

• non-project commands from any package (like rv exec rails)

If we try to combine those two commands into one command, we quickly run into the ambiguity that has been so frustrating to handle in Bundler for all of these years: if you rv run rake, do you want the global rake package? Or do you want the project-local rake package? How can we know which you want?

In my opinion, uv solves this relatively elegantly by having uvx always run a package globally, and uvr always run a package locally inside the current project, if one exists.

What do you think? Let us know in the GitHub discussion about this post.

Welcome! This is meant to serve as an introduction to deployment and operations for newer developers, but it’s also a checklist that I refer back to even after 20 years of deploying Rails apps to production. What needs to be covered when going to production the first time? What needs to be covered when going to production for the 1000th time?

Before we get into those details, let me introduce myself. I’m André Arko, better known as @indirect on the internet. My biggest Ruby claim to fame is leading the Bundler and RubyGems OSS team for the last 15 years or so. Ruby Central recently kicked out the existing team, so the former RubyGems team has set up a new community-focused gem server at https://gem.coop. I hope you’ll check it out.

But I’m here to talk about deploying and operating Rails applications, so let’s get going. I built my first Rails application in 2004, and trying to get that application off my laptop so other people could use it is what started me down the path to madness and devops. It’s been a long journey, and I have learned many things from over 20 years of watching Rails apps break in production.

Let’s set the scene: an excited Ruby developer follows a Rails tutorial, creating a blog, a todo app, or some other cool idea. They walk through the code, the tests, the features. They finish the tutorial, and have a full Rails application, something cool that they want to show off to other people! That’s when they realize… this is only on their laptop, and no one else can see or use it.

The most useful tutorials will mention hosting options at this point, maybe Heroku, Fly.io, Render, Railway, or DigitalOcean. If they’re unhelpful tutorials, they might mention AWS, GCP, or Azure. But no matter what service they suggest, coding tutorials need to stay focused, and refer developers out to a hosting service when they want to deploy.

This is a problem, because the tutorials about how to deploy on those hosting sites all say things like “of course this introduction does not include how to secure your secrets in production”. In fact, hosting tutorials usually just wave their hands and skip over most aspects of deploying and operating a true production application. I’m not sure how they expect a new dev to learn these skills, exactly, but that’s what I’m going to try to cover in this talk today.

So our hypothetical Rails developer has followed a hosting tutorial, and has a shiny new application and database deployed to Heroku, or Fly.io, or Render, or whatever. Now is when they actually start to run up against the real problems of running in production: are the secrets secure? is the database backed up? what happens if there’s an exception? how do you debug something that only happens on the server?

Let’s break down each of the new kinds of preparation needed by category, and walk through what you need to do, and why you need to do it. Ultimately, everything that we are going to talk about today is about confidence. Everything happening in deployment and operations, in devops, in SRE, is all being done to increase confidence in the application—your own, your coworkers, or your customers.

The areas we’re going to look at today are Errors, Data, Speed, Security, and what usually gets called Lead Time. Lead Time is the amount of time it takes for a change to go from a commit to live in production. Investing in a shorter Lead Time can return absolutely outsized benefits in every other area, because it means you can try a change, learn from it, and make another change in reaction, faster and faster. We’ll talk a bit more about it at the end.

Errors

To start with, let’s focus on Errors. Conceptually, an error is any time that your user isn’t able to use your site. More specifically, an error could mean your Ruby code threw an exception, it could mean some configuration is preventing your app from working correctly, or it could mean that your servers have completely blown up and there is no app available for your users to talk to. With that in mind… how will you know? That’s the first and most important rule of running applications in production. You need to know in advance how you’re going to find out if something is broken.

It’s not popular deployment advice, but the most important way to build confidence and in your application is to test it. Test it manually, test it automatically, have other people try it and report back. Each of those things will save you more time debugging than you can even imagine right now.

Next, one of the most popular ways to increase confidence before a deploy is a second environment where you can deploy your code to test it before it is live on the main site. The test area is usually called “staging”, and the main area is usually called “production”. Rails has built-in support for both of those, as well as “development”, which runs by default on your local development machine.

Use the staging environment as a place where you can put code that you aren’t sure about, and try the new version there. After you have confidence (there it is again) in your new changes, you can “promote” them to production. Some platforms even let you bump staging to production without having to wait for a full new production deploy.

After you’re releasing code with manual and automatic tests to staging and then production, the next most common mitigation strategies are exception reporting and uptime monitoring. Exception reporting does exactly what it sounds like, and sends a report to you when there is an exception in your application.

Personally, I tend to use honeybadger.io to set up exception tracking and uptime monitoring. It’s free for a single user, which is great for my personal project, and I am apple to connect it to Slack so I get notifications if there are Ruby or Javascript errors, or if the site stops being reachable from the internet. Other popular options for tracking exceptions include Sentry, Airbrake, Bugsnag, and more that you can easily find with a search.

For anything that’s less significant than an exception, or to help you investigate when an exception does occur, you’ll also want to think about what usually gets called “instrumentation” or “observability”. The most basic level here is just seeing the logs from your application, and most hosting will provide a way for you to do that.

The more advanced levels of observability include setting up additional tracking in your application, both inside the Rails framework and inside your own code. This tracking can take the form of metrics, which are numbers counting when and how many times some particular thing has happened. Monitoring that focuses primarily on metrics includes using the Prometheus open source tool, or DataDog, AppSignal, NewRelic, and others like them.

Monitoring can also take the form of traces, where your application keeps track of how long each part of a response takes, letting you see each controller, parameter, database query, http request, and anything else that happened. If you’ve ever heard of OpenTelemetry, this is what that is about. Honeycomb.io is the pioneer of this style of monitoring, but others like Sentry and DataDog have also added the ability to collect and review traces.

I don’t always find Honeycomb’s “only events and traces” style easier to use than simply reading logs, but it regularly helps me solve debugging mysteries that I would not have been able to figure out from solely reading logs, so I’m very glad to have it as an option.

Lastly on the topic of errors in production, the open-ended question to ask yourself is “how will people tell me about problems?”. Do you have a contact form? A support email? A social media profile? Make sure there’s some way to hear from your users, because there will always be something that doesn’t set off your monitoring but still needs to be fixed… if you can hear about it.

Data

Next up is Data. Data includes not just the stuff that users have uploaded and put directly into your database, it also includes uploaded files, and anything else that you might want while recovering from a catastrophic server failure. Server failures range from the very mundane server part died to the very surprising truck rammed into the data center’s power transformer, but they all have the same end state: your server is gone.

Now that your server is gone, you need to set up something again. This is the touchiest part of any production service. Do you have a copy of the latest code? A recent database backup? Uploaded files saved somewhere you can still get to them?

Start with the code. Make sure that you deploy to production by pushing to a code host and deploying from there. Most often that means GitHub and a deploy from GitHub Actions, but the important part is that any code that goes live needs to be easy to find even if your laptop vanishes.

Next, handle the database. Use a system that automatically takes daily snapshots of your database, or build something yourself that creates daily copies. For my side projects, I have set up a daily GitHub Action that connects to my database, pulls out a copy, and uploads it as an artifact. Actions artifacts have no size limit, and just time out after 90 days, which is perfect for daily backups.

Test your backups. More than one startup has completely failed and shut down when they discovered that their backups weren’t actually running, or the results were corrupted, only after it was too late. Have more than one backup, preferably one per day, and test your backups system regularly. That’s the second, and most important, rule of running applications in production.

Once you have your database(s) taken care of, it’s time to look at your user uploaded or created files. Using a file service like S3, B2, R2, Google Cloud Storage, or Tigris can be a huge advantage here. Those services all provide a basic promise that your files will be copied to at least 3 places at all times. While it’s possible to run your own copy of Minio and treat it like S3, if the filesystem backing your Minio server crashes, those files are gone. I don’t recommend doing that unless you’re sure those files don’t really matter.

Lastly for data, you now need to think about how you are going to deal with all of the existing production data, now constantly growing, that you can’t just erase. In local development, if your schema gets into a weird state, it’s easy to just create a new database and import the schema again. Now that you have a live, production database, you can’t do that.

Adding, removing, and renaming database columns are all now potentially actions that could break your application. To avoid full site downtime, you need to deploy code that can handle both the old schema and the new schema, then run the schema migration, then remove the handler code and deploy again. I recommend adding the strong_migrations gem as soon as you have a live production site, to add some guardrails that will make it harder to accidentally break either the site or your users’ data.

Speed

Now let’s talk about speed. Speed from a Rails app? It’s possible! In my experience, the biggest reasons that Rails apps get slow are external API calls, responses with no pagination, and accidental N+1 database queries. Rails may not be the most performant framework in the world, but it can be perfectly usable as long as you avoid those traps.

Why does the speed of your application matter? As mentioned at the start, it comes down to confidence. If users can see what is happening quickly, they trust your app to be working correctly. If your application matter takes so long that users lose interest or get distracted, that confidence is gone. Whatever did happen with that thing… hmm, I dunno.

Use the monitoring and observability tools we talked about before to keep an eye on how long your application is taking to do things. Don’t use the average, because averages don’t reflect any actual user experience. Instead, track a percentile that reflects how many users you can tolerate having a bad experience.

If you have 10 users per day, maybe the 90th percentile performance is what to watch because only 1 person will have a worse experience. If you have 10 million users per day, you might want to track the 99.99th percentile, because that’s still 1000 people every day having a worse experience than that number.

When you find something that’s slow, your main tools to fight against it are monitoring and profiling. Monitoring can tell you which users experienced the slowness, and what they were trying to do. It might even be able to tell you which SQL queries or HTTP calls made things so slow. Meanwhile, profiling can tell you what code is being run inside a given action, and help you optimize it. The rack-prof gem is the most well-known profiler, but there are multiple options available.

Security

Now that you know how and why to keep your app fast, let’s talk about security. There are two resources your production application has that you now need to defend: user data, and servers on the internet. One class of attacks will try to break in to your application to steal user data, like emails, phone numbers, whatever you have stored. The other type of attack is about hijacking your server to do something for the attacker, like send spam or show ads or mine cryptocurrency.

The best defense for user data is to not store it at all. If you don’t absolutely need it, don’t collect it, and then no one can steal it. If you need to collect user information, make sure you are only collecting what you need. If your users are supplying personal information, especially sensitive financial or identity information, make sure the data is encrypted inside the database so a simple database leak won’t reveal it. It’s not hard to encrypt a single model column anymore, so be sure to do it!

The best defense against attackers who want to hijack your servers is to update your libraries. Use a tool like Dependabot or Renovate to apply security fixes, so your application isn’t vulnerable to known hacks. Set up a monitoring alert on your CPU and disk usage. Set up a billing alert for if your usage goes above your usual amount, so you can investigate.

Ultimately, it’s always worth it to have a security policy, even if it’s very simple. Provide contact information so problems can be brought to your attention. You don’t need to have a security team and a HackerOne program, but you do need to have a basic awareness of what could go wrong and how you can approach handling it. Knowing you have backups that can’t be deleted by someone hacking into your server is a huge advantage here.

Lead Time

Finally, lead time. Lead time is one of the for metrics tracked by the DORA program’s research into software development teams. The book Accelerate, written by the founders of the DORA research program, goes into much more depth than I can fit here, so I’m going to briefly summarize just this one aspect.

Lead time is the amount of time it takes to go from code being committed to that same code being live in production. If you can make a change and get it deployed into production in just a few minutes, you will have a completely different experience developing your application than if it takes hours or days. The DORA research found that small lead times is one of the biggest predictors of software projects being successful.

The overall, main finding of the DORA research project is that conventional wisdom of going slowly and carefully is exactly backwards—the teams and projects with the least errors and rollbacks are the teams that deploy the fastest and the most frequently. If you’re already operating a larger or slower application in production, the biggest advice that I have is to reduce the size of each change, and make rolling out each change as fast as you possibly can.

The beginning of a project is by far the easiest time to set up the standard automations that you plan to use going forward. I strongly recommend setting up completely automatic testing and deployment. With GitHub Actions and modern hosting platforms, you can expect new commits to be tested and fully deployed to production within 5 minutes. That’s a fantastic starting point to use as a base for future expansion, as your application and team grows.

Strategy

Now that I’ve filled your heads with more ideas than you can probably remember, let’s quickly review and then briefly discuss an overall strategy that you can use whenever you are trying to operate a production application. The big concerns after you deploy your app are: Errors, Data, Speed, Security, and Lead Time. Use those categories to review proposed changes, and look for issues you might have missed. Will the proposed change impact your ability to monitor errors? Back up user data? Respond to requests quickly? Keep your application secure? Ship changes quickly?

By keeping those categories of concern in mind, you are already better prepared to operate a production deployment than the vast majority of Rails developers. If the thought of working with those concerns is interesting or exciting to you, you probably have a bright future in DevOps or SRE. If the thought of working with those concerns fills you with unspeakable dread, you probably want to stick to bigger teams where someone else is available to handle operations and deployment.

Self-promotion

If your team could use a boost while handling everything we’ve talked about today, you can get that support. I work for Spinel Cooperative, with developers from the core teams of Rails, Hotwire, RubyGems, and more. We would love to give your team all the advantages of 20 years spent building, deploying, and operating Ruby and Rails. Come say hi after the talk, or drop us an email at hello@spinel.coop.

Conclusion

In the end, will listening to all the advice in this talk mean your app never goes down? Unfortunately, probably not. Downtime is something that comes for us all, and usually includes some aspect that is completely unexpected or outside of our control. Instead, this advice will make you prepared. You’ll be prepared to recover from downtime, and prepared to add more protections in the future as you learn the particular issues your application needs to defend against.

Ultimately, that’s the best any of us can do, and I wish you all good luck with your future deployments.

• None of the “temporary” changes made by Ruby Central have been undone, more than six weeks later.
• Ruby Central still has not communicated with the removed maintainers about restoring any permissions.
• Ruby Central still has not offered “operator agreements” or “contributor agreements” to any of the removed maintainers.
• The Ruby Together merger agreement plainly states that it is the maintainers who will decide what is best for their projects, not Ruby Central.
• Last week, Matz stepped in to assume control of RubyGems and Bundler himself. His announcement states that the Ruby core team will assume control and responsibility for the primary RubyGems and Bundler GitHub repository.
• Ruby Central did not communicate with any removed maintainers before transferring control of the rubygems/rubygems GitHub repo to the Ruby core team.
• On October 24th, Shan publicly confirmed she does not believe the maintainers need to be told why they were removed.

While we know that Ruby Central had no right to act the way they did, it is nevertheless clear to us that the Ruby community will be better off if the codebase, maintenance, and legal rights to RubyGems and Bundler are all together in the same place.

To bring this about, we are prepared to transfer our interests in RubyGems and Bundler to Matz, end the dispute over the GitHub enterprise account, 2 GitHub organizations, and 70 repositories, and hand over all rights in the Bundler logo and Bundler name, including the trademark applications in the US, EU, and Japan.

Once we have entered into a legal agreement to settle any legal claims with Ruby Central and transfer all rights to Matz, the former maintainers will step back entirely from the RubyGems and Bundler projects, leaving them fully and completely to Matz, and by extension to the entire Ruby community.

Although Ruby Central’s actions were not legitimate, our commitment to the Ruby community remains strong. We’re choosing to focus our energy on projects to improve Ruby for everyone, including rv, Ruby Butler, jim, and gem.coop.

Signed,
The former maintainers: André, David, Ellen, Josef, Martin, and Samuel

Just like git, jj offers tiers of configuration that layer on top of one another. Every setting can be set for a single repo, for the current user, or globally for the entire system. Just like git, jj offers the ability to create aliases, either as shortcuts or by building up existing commands and options into new completely new commands.

Completely unlike git, jj also allows configuring revset aliases and default templates, extending or replacing built-in functionality. Let’s look at the ways it’s possible to customize jj via configurations. We’ll cover basic config, custom revsets, custom templates, and custom command aliases.

config basics

Let’s start with some configuration basics. You can globally configure jj to change your name and email based on a path prefix, so you don’t have to remember to set your work email separately in each work repo anymore.

[[--scope]]
--when.repositories = ["~/work"]
[--scope.user]
email = "me@work.domain"

Or perhaps you want jj to wait for your editor if you are writing a commit message, but you don’t want jj to wait for your editor to exist if you are editing your jj configuration file. You can ensure that using a scope.

ui.editor = "code -w"
[[--scope]]
--when.commands = ["config"]
[--scope.ui]
ui.editor = "code"

I also highly recommend trying out multiple options for formatting your diffs, so you can find the one that is most helpful to you. A very popular diff formatter is difftastic, which provides syntax aware diffs for many languages. I personally use delta, and the configuration to format diffs with delta looks like this:

[[--scope]]
--when.commands = ["diff", "show"]
[--scope.ui]
pager = "delta"
diff-formatter = ":git"

Another very impactful configuration is which tool jj uses to handle interactive diff editing, such as in the jj split or jj squash -i commands. While the default terminal UI is pretty good, make sure to also try out Meld, an open source GUI.

[ui]
diff-editor = "meld" # or vimdiff, vscode, etc

In addition to changing the diff editor, you can also change the merge editor, which is the program that is used to resolve conflicts. Meld can again be a good option, as well as any of several other merging tools.

[ui]
merge-editor = "meld" # or vimdiff, vscode, mergiraf etc

Tools like mergiraf provide a way to attempt syntax-aware automated conflict resolution before handing off any remaining conflicts to a human to resolve. That approach can dramatically reduce the amount of time you spend manually handling conflicts.

You might even want to try FileMerge, the macOS developer tools built-in merge tool. It supports both interactive diff editing and conflict resolution.

[merge-tools.filemerge]
program = "open"
edit-args = ["-a", "FileMerge", "-n", "-W", "--args",
             "-left", "$left", "-right", "$right",
             "-merge", "$output"]
merge-args = ["-a", "FileMerge", "-n", "-W", "--args",
              "-left", "$left", "-right", "$right",
              "-ancestor", "$base", "-merge", "$output",]

Just two more configurations before we move on to templates. First, the default subcommand, which controls what gets run if you just type jj and hit return. The default is to run jj log, but my own personal obsessive twitch is to run jj status constantly, and so I have changed my default subcommand to status, like so:

[ui]
default-command = ["status"]

The last significant configuration is the default revset used by jj log. Depending on your work patterns, the multi-page history of commits in your current repo might not be helpful to you. In that case, you can change the default revset shown by the log command to one that’s more helpful. My own default revset shows only one change from my origin. If I want to see more than the newest change from my origin I use jj ll to get the longer log, using the original default revset. I’ll show that off later.

[revsets]
log = "(trunk()..@):: | (trunk()..@)-"

templates

While the jj template docs are a great reference, they don’t do very much to show off what’s possible by using templates, so we’ll show some examples.

The first and most obvious template is the jj log template, which controls how each change is rendered in the log output. The default template is named builtin_log_compact, and jj comes with a few pre-built template options for the log view, like builtin_log_detailed and builtin_log_oneline. You can see them all by running jj log -T.

Use jj log -T NAME to try them out and see how they look. If you want to experiment with your own custom log formats, you can provide a template string instead of the name of an existing template. Here’s an example inline template that prints out just the short change ID, a newline, and then the change description:

jj log -T 'change_id.short() ++ "\n" ++ description'

Try out the various documented template properties yourself! Once you’re happy with a template that you’ve tested, you can add it to your config with a name, and then use it by name.

Here’s a more complicated example, adapted from @marchyman in the jj Discord, with several of the elements that we’ve discussed so far. This example changes the default command, adding extra options. It also uses a named revset alias, and a named template alias.

[ui]
default-command = ["log", "-T", "log_with_files", "--limit" "7"]

[revset-aliases]
'recent_work' = 'ancestors(visible_heads(), 3) & mutable()'

[template-aliases]
log_with_files = '''
if(root,
  format_root_commit(self),
  label(if(current_working_copy, "working_copy"),
    concat(
      format_short_commit_header(self) ++ "\n",
      separate(" ",
        if(empty, label("empty", "(empty)")),
        if(description,
          description.first_line(),
          label(if(empty, "empty"), description_placeholder),
        ),
      ) ++ "\n",
      if(self.contained_in("recent_work"), diff.summary()),
    ),
  )
)
'

The named template recreates the regular template from log, and uses a revset filter to include the list of changed files in changes that are both mutable (that is, not yet pushed) and also within 3 commits of the end of a branch. By showing up to 7 changes, but the file list for up to 3 mutable changes, the log output becomes more useful, reminding you what files have been changed in the most recent commits that you might want to push.

One more template you might want to adjust is the default description, shown when running jj commit or jj desc for a change that does not yet have a description. If you don’t use a VCS GUI, it can be helpful to see the diff of what is being committed at the same time as you write the commit message. In git, that meant running git commit --verbose, but in jj that means adjusting the default description. The jj config docs provide an example template that will replicate that effect, and show you the diff while you write the message.

revset aliases

Building on the earlier section where we talked about jj log, creating your own revset aliases is a powerful way to construct views tailored to your personal needs.

A built-in revset alias we can use to illustrate this is immutable(). In the same way that git requires --force to push over an existing remote commit, jj requires --ignore-immutable to edit a commit matched by immutable().

(Incidentally, I believe this arrangement is also an example of the way jj’s design is an improvement on git. Instead of deciding to overwrite published commits during a push, you are forced to decide much earlier, during the edit itself, if you are okay with changing a published commit. Anyway, back to revset aliases.)

The default immutable_heads() revset is present(trunk()) | tags() | untracked_remote_bookmarks(), which composes four other revsets together. Let’s look at each one. The trunk() revset is simply the primary branch, whether it is named main or master, wrapped in present() to remove it if none of those branches exist. The tags() revset is every change that has been given a tag. The untracked_remote_bookmarks() revset is exactly what it sounds like: any branch provided by the remote that you have not manually opted in to tracking locally (which is what you would do if you are working on the branch). All three revsets are combined into one overall list with the | operator. Those heads are then used to construct the full list of immutable commits, which is every ancestor of those heads.

Now that you know how that works, we can change it. For example, perhaps you want the same immutability rules that git provides, where commits are immutable once they have been pushed to any remote at all. In that case, you could add this to your config file:

[revset-aliases]
"immutable_heads()" = "present(trunk()) | tags() | remote_bookmarks()"

With that configuration, jj will extrapolate that it cannot change any commits on the primary branch, all the commits leading up to a tag, and all commits leading up to a named branch in the remote. If you use this revset, jj will stop you from changing commits once you have pushed them to a branch, since you told it to make those immutable.

With this power at your disposal, you can change the default revset shown when you run jj log, or you can create your own named revsets for your own purposes. You can see my revset aliases in my dotfiles, and read more about the default aliases in the jj docs.

command aliases

Okay. Now we can talk about command aliases. First up, the venerable tug. In the simplest possible form, it takes the closest bookmark, and moves that bookmark to @-, the parent of the current commit.

tug = ["bookmark", "move", "--from", "heads(::@- & bookmarks())", "--to", "@-"]

What if you want it to be smarter, though? It could find the closest bookmark, and then move it to the closest pushable commit, whether that commit was @, or @-, or @---. For that, you can create a revset for closest_pushable, and then tug from the closest bookmark to the closest pushable, like this:

[revset-aliases]
'closest_pushable(to)' = 'heads(::to & mutable() & ~description(exact:"") & (~empty() | merges()))'
[aliases]
tug = 'bookmark move --from "heads(::@ & bookmarks())" --to "closest_pushable(@)"'

Now your bookmark jumps up to the change that you can actually push, by excluding immutable, empty, or descriptionless commits.

What if you wanted to allow tug to take arguments, for those times when two bookmarks are on the same change, or when you actually want to tug a different bookmark than the closest one? That’s also pretty easy, by adding a second variant of the tug command that takes an argument:

tug = ["util", "exec", "--", "sh", "-c", """
if [ "x$1" = "x" ]; then
  jj bookmark move --from "closest_bookmark(@)" --to "closest_pushable(@)"
else
  jj bookmark move --to "closest_pushable(@)" "$@"
fi
""", ""]

This version of tug works just like the previous one if no argument is given. But if you do pass an argument, it will move the bookmark with the name that you passed instead of the closest one.

How about if you’ve just pushed to GitHub, and you want to create a pull request from that pushed bookmark? The gh pr create command isn’t smart enough to figure that out automatically, but you can tell it which bookmark to use:

pr = ["util", "exec", "--", "bash", "-c", """
gh pr create --head $(jj log -r 'closest_bookmark(@)' -T 'bookmarks' --no-graph | cut -d ' ' -f 1)
"""]

Just grab the list of bookmarks attached to the closest bookmark, take the first one, pass it to gh pr create, and you’re all set.

What if you just want single commands that let you work against a git remote, with defaults tuned for automatic tugging, pushing, and tracking? I’ve also got you covered.

init = ["util", "exec", "--", "bash", "-c", """
jj git init --colocate
# only track origin branches, not upstream or others
jj bookmark track 'glob:*@origin'
"""]

Use jj init to colocate jj into this git repo, and then track any branches from upstream, like you would get from a git clone.

pull = ["util", "exec", "--", "bash", "-c", """

# Find the closest bookmark
closest="$(jj log -r 'closest_bookmark(@)' \
  -n 1 -T 'bookmarks' --no-graph | cut -d ' ' -f 1)"

# Remove the trailing * from the name if there is one
closest="${closest%\\*}"

# Now fetch from the git remote
jj git fetch

# If the closest bookmark still exists, rebase on it (else trunk)
jj log -n 1 -r "${closest}" 2>&1 > /dev/null \
  && jj rebase -d "${closest}" || jj rebase -d 'trunk()'

# Show the new state of things after the pull
jj log -r 'stack()'
"""]

Then, you can jj pull to find the closest bookmark to @, do a git fetch, rebase your current local commits on top of whatever just got pulled, and then show your new stack. When you’re done, just jj push.

push = ["util", "exec", "--", "bash", "-c", """

# Check to see if we can tug a bookmark to @
tuggable="$(jj log -r 'closest_bookmark(@)..closest_pushable(@)' \
  -T '"n"' --no-graph)"

# If we can, tug that bookmark as close to @ as possible
[[ -n "$tuggable" ]] && jj tug

# Now find the closest thing that we can push to `origin`
pushable="$(jj log -r 'remote_bookmarks(remote=origin)..@' \
  -T 'bookmarks' --no-graph)"

# If we have something to push, run `jj git push`
[[ -n "$pushable" ]] && jj git push || echo "Nothing to push."

# Now that we have pushed, find the closest bookmark and remove *
closest="$(jj log -r 'closest_bookmark(@)' -n 1 -T 'bookmarks' \
  --no-graph | cut -d ' ' -f 1)"
closest="${closest%\\*}"

# Check to see if that bookmark is already tracking origin
tracked="$(jj bookmark list -r ${closest} -t \
  -T 'if(remote == "origin", name)')"

# If that bookmark isn't tracking origin, start to track origin
[[ "$tracked" == "$closest" ]] \
  || jj bookmark track "${closest}@origin"
"""]

This push handles looking for a tuggable bookmark, tugging it, doing a git push, and making sure that you’re tracking the origin copy of whatever you just pushed, in case you created a new branch.

further reading

For another perspective on jj configuration, partly overlapping with this post, check out my JJ Con talk, stupid jj tricks.

You can also try reading some jj config files directly, like my jj config, or thoughtpolice’s jj config, or pksunkara’s jj config.

next time

Hopefully this tour through jj configuration options has revealed some ways that jj can be used to do more than was possible with only git. Next time, we’ll focus on the ways that jj goes beyond git, offering things that were impractical or even impossible before.

Now that you hopefully have an idea of how to operate jj, let’s look at the commands you need to get work done in jj. One great aspect of jj layering on top of git repos is that the git repo is still there underneath, and you can use any git command exactly like you usually would if there’s anything missing from your jj workflows.

submit a pull request

The flow to create and send a PR will probably look pretty familiar: use jj git clone to get a copy of the repo, make your changes, use jj commit to create your new commits. When you’re ready, use jj bookmark set NAME to give your changes a name and jj git push to create a new branch on the remote. Use GitHub.com or gh pr create --head NAME to open the PR.

If you amend the commits in your PR, you can force-push the new commits with jj git push. If you add new changes on top, you’ll need to jj bookmark set NAME to update the bookmark to the latest change before you jj git push again.

That’s the whole flow! Congratulations on migrating from git to jj for your everyday work.

If using bookmark set all the time gets tedious, there’s a community alias named jj tug that finds the closest bookmark and moves it to the closest pushable change. I personally wrote an alias for myself named jj push that I use to handle pushing new changes to existing remote branches. We’ll talk about those aliases in the next major section, which is about configuring jj.

work on multiple PRs at once

One situation I often find myself in is working on two (or even more) pull requests at the same time. With the powerful commit-editing primitives provided by jj, there are at least two (and probably more) ways to structure this kind of parallel work.

The first option is what I think of as merge-based: create a merge commit that unifies the tips of your two or more branches using jj new -d A -d B, do your work, and create new commits with jj split or jj commit. Then, rebase those commits using jj rebase -r @- --insert-before A or the like, moving the new commits backwards into one of the PR branches. This is the same as the “megamerge” strategy described above, but it works just as well with two branches.

The second option is to liberally rebase every branch on top of each other, creating a completely linear history where PR #4 contains PR #3, which also contains PR #2, which also contains PR #1. Since jj uses change IDs to keep track of changes as their commits are amended or rebased, you can rebase the entire chain on top of new commits to main. Your bookmarks will stay in the same place, and you can jj git push to update each remote branch. Any time one of the PRs lands, you rebase your full chain on top of the new main and resume work where you left off.

If you want to work on multiple branches at once, you will probably also find the articles Jujutsu Merge Workflow and Jujutsu Megamerges and jj absorb interesting.

further workflow reading

There are many new workflows that jj users have already developed, and this brief overview is just the tip of the iceberg. The jj docs include a section on using jj with GitHub or GitLab, and there are some great reflections on different workflows in the blog posts Jujutsu VCS Introduction and Patterns, Git experts should try Jujutsu, and jj tips and tricks.

next time

Next up, we’re going to talk about configuring jj. Want to use a diff tool? A merge tool? Add your own commands? Optimize your day to day work? We’ve got options.

Continue with jj part 4: configuration.

This version dramatically expands support for Rubies, shells, and architectures.

Rubies: we have added Ruby 3.3, as well as re-compiled all Ruby 3.3 and 3.4 versions with YJIT. On Linux, YJIT increases our glibc minimum version to 2.35 or higher. That means most distro releases from 2022 or later should work, but please let us know if you run into any problems.

Shells: we have added support for bash, fish, and nushell in addition to zsh.

Architectures: we have added Ruby compiled for macOS on x86, in addition to Apple Silicon, and added Ruby compiled for Linux on ARM, in addition to x86.

Special thanks to newest member of the maintainers’ team @adamchalmers for improving code and tests, adding code coverage and fuzzing, heroic amounts of issue triage, and nushell support. Additional thanks are due to all the new contributors in version 0.2, including @Thomascountz, @lgarron, @coezbek, and @renatolond.

To upgrade, run brew install rv, or check the release notes for other options.

First, and most importantly: I was a primary operator of RubyGems.org, securely and successfully, for over ten years. Ruby Central does not accuse me of any harms or damages in their post, in fact stating “we have no evidence to indicate that any RubyGems.org data was copied or retained by unauthorized parties, including Mr. Arko.”

The actions I took during a time of great confusion and uncertainty (created by Ruby Central!) were careful, specific, and aimed to defend both Ruby Central the organization and RubyGems.org the service from potential threats.

The majority of the team, including developers in the middle of paid full-time work for Ruby Central, had just had all of their permissions on GitHub revoked. And then restored six days later. And then revoked again the next day. Even after the second mass-deletion of team permissions, Marty Haught sent an email to the team within minutes, at 12:47pm PDT, saying he was (direct quote) “terribly sorry” and “I messed up”. Update: Added email timestamp.

The erratic and contradictory communication supplied by Marty Haught, and the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call. My contractual, paid responsibility to Ruby Central was to defend the RubyGems.org service against potential threats. 

Marty’s final email clearly stated “I’ll follow up more on this and engage with the governance rfc in good faith.”. Just a few minutes after that email, at 1:01pm PDT, Marty also posted a public GitHub comment, where he agreed to participate in the proposed governance process and stated “I’m committed to find the right governance model that works for us all. More to come.” Update: screenshot of comment removed and replaced with link, since the comment appears to still be visible (at least to logged out users) on GitHub.

Given Marty’s claims, the sudden permission deletions made no sense. Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers. I did not change the email addresses on any accounts, leaving them all owned by a team-shared email at rubycentral.org, to ensure the organization retained overall control of the accounts, even if individuals were somehow taking unauthorized actions.

Within a couple of days, Ruby Central made an (unsigned) public statement, and various board members agreed to talk directly to maintainers. At that point, I realized that what I thought might have been a malicious takeover was both legitimate and deliberate, and Marty would never “fix the permissions structure”, or “follow up more” as he said.

Once I understood the situation, I backed off to let Ruby Central take care of their “security audit”. I left all accounts in a state where they could recover access. I did not alter, or try to alter, anything in the Ruby Central systems or GitHub repository after that. I was confident, at the time, that Ruby Central’s security experts would quickly remove all outside access.

My confidence was sorely misplaced.

Almost two weeks later, someone asked if I still had access and I discovered (to my great alarm), that Ruby Central’s “security audit” had failed. Ruby Central also had not removed me as an “owner” of the Ruby Central GitHub Organization. They also had not rotated any of the credentials shared across the operational team using the RubyGems 1Password account.

I believe Ruby Central confused themselves into thinking the “Ruby Central” 1Password account was used by operators, and they did revoke my access there. However, that 1Password account was not used by the open source team of RubyGems.org service operators. Instead, we used the “RubyGems” 1Password account, which was full of operational credentials. Ruby Central did not remove me from the “RubyGems” 1Password account, even as of today.

Aware that I needed to disclose this surprising access, but also aware that it was impossible for anyone except former operators to exploit this security failure, I immediately wrote an email to Ruby Central to disclose the problem.

Here is a copy of my disclosure email, in full.

From: André Arko <andre@arko.net>
Subject: Re: RubyGems.org access
Date: September 30, 2025 at 10:23:12 AM PDT
To: Marty Haught <marty@rubycentral.org>

Hi Marty,

It has come to my attention that despite the statements in [your] email, I have had uninterrupted access to RubyGems.org production environments from September 18 until today, September 30, via the root credentials of the Ruby Central AWS account, as well as continued and ongoing access to the full feed of production alerts and logs in DataDog.

It seems that the only permissions I have had removed are from the GitHub organization named "rubygems", which as you know is unrelated to the RubyGems.org production access you mention in your email.

I have also noticed I am still, as of September 30, the owner of the GitHub organizations named "rubycentral" and "rubytogether".

I am unable to transfer the HelpScout or PagerDuty accounts, as you have disabled my andre@rubygems.org Google account.

Please advise as to your desired resolution of this situation.

Thank you,
André Arko

Ruby Central did not reply to this email for over three days.

When they finally did reply, they seem to have developed some sort of theory that I was interested in “access to PII”, which is entirely false. I have no interest in any PII, commercially or otherwise. As my private email published by Ruby Central demonstrates, my entire proposal was based solely on company-level information, with no information about individuals included in any way. Here’s their response, over three days later.

From: Marty Haught <marty@rubycentral.org>
Subject: Re: RubyGems.org access
Date: October 3, 2025 at 6:54:01 PM MDT
To: André Arko <andre@arko.net>

Hi André,

Please confirm that you cannot access the Ruby Central AWS root account credentials, either through the console or by access keys.

In addition, please confirm whether you are in possession of any RubyGems.org production data,  including, but not limited to, server logs, access logs, PII, or other organizational data.

Thank you,
Marty

In addition to ignoring the (huge) question of how Ruby Central failed to secure their AWS Root credentials for almost two weeks, and appearing to only be aware of it because I reported it to them, their reply also failed to ask whether any other shared credentials might still be valid. There were more.

From: André Arko <andre@arko.net>
Subject: Re: RubyGems.org access
Date: October 5, 2025 at 11:59:35 AM PDT
To: Marty Haught <marty@rubycentral.org>

Hi Marty,

Thanks for letting me know you got my email disclosing my unintended access. I’m concerned that security must not be a very high priority for Ruby Central since no one acknowledged my disclosure for more than three days, but I appreciate the confirmation.

As far as I can tell, I can no longer access the Ruby Central AWS root account either through the console or via access keys.

I confirm I did not download or save any production data after your email of September 18, including server logs, access logs, PII, or other organizational data.

However, while checking AWS credentials in order to write this email, I discovered that several other service credentials have not been rotated, and are still valid for production AWS access. That means both myself and the other former operators all still have access to AWS via those previously-shared credentials.

I would appreciate it if you could answer the request from my first email, and reply with your desired resolution for this remaining unintended production access, as well as the GitHub organization ownership.

Thanks,
André

Unbeknownst to me, while I was answering Marty’s email in good faith, Ruby Central’s attorney was sending my lawyer a letter alleging I had committed a federal crime, on the theory that I had “hacked” Ruby Central’s AWS account. On the contrary, my actions were taken in defense of the service that Ruby Central was paying me to support and defend.

With my side of the story told, I’ll leave it to you to decide whether you think it’s true that “Ruby Central remains committed to transparent, responsible stewardship of the RubyGems infrastructure and to maintaining the security and trust that the Ruby ecosystem depends on.”

The new server’s governance policies are being prepared in coordination with Mike McQuaid of Homebrew, and will be released later this week.

The current versions of RubyGems and Bundler work with this new server already, and any Ruby developer is welcome to switch to using this new server immediately.

We have exciting plans to add new features and functionality in the coming days. Join us!

Now, let’s take a look at the most common jj commands, with a special focus on the way arguments are generally consistent and switches don’t hide totally different additional commands.

jj log

The log command is the biggest consumer of revsets, which are passed using -r or --revisions. With @, which is the jj version of HEAD, you can build a revset for exactly the commits you want to see. The git operator .. is supported, allowing you to log commits after A and up to B with -r A..B, but that’s just the start. Here’s a quick list of some useful revsets to give you the flavor:

• @- the parent of the current commit
• kv+ the first child of the change named kv
• ..A & ..B changes in the intersection of A and B’s ancestors
• ~description(glob:"wip:\*") changes whose message does not start with wip:, because tilde negates a revset
• heads(::@ & mutable() & ~description(exact:"") & (~empty() | merges())) the closest “pushable” change, meaning the nearest ancestor of @ that is mutable (by default mutable means “not in the main/trunk branch”), that has some description set, and that either has some changes or is a merge commit. (Some jj merge commits can be empty, if there were no conflicts.)

Using the jj config file, you can give any revset an alias, and then use that alias. I use closest_pushable(@) quite a bit, especially when naming branches and pushing.

For a full review of everything that’s possible with revsets, check out the revset documentation and the blog post Understanding Revsets for a Better JJ Log Output.

jj commit / desc / new / edit / split

The functionality of git commit is broken up into four separate jj commands. You use new to create a new empty child change, defaulting to @, and edit it. The desc command lets you set the description (or message) on a given change. The commit command works like git, but is effectively the same as jj desc && jj new. You use edit to re-open an existing change for amending, and split to interactively select a diff to break out into a second change. These are all common git workflows, done by using flags or multiple git commands, made direct and straightforward single commands in jj.

jj restore / abandon

What if using checkout and reset to roll back either files or full commits had clearer names?

If you want to get back a file from a previous change, you can use restore. Specify which change you want to bring back, and also provide a file name or glob to limit the restoration to specific files.

Where you might have previously used git reset or git checkout to manipulate which commits are included in the current branch, you can now use abandon to remove entire changes from your history. Without any arguments it will remove @, the working commit, which is similar to git reset --hard. With arguments, abandon will remove all changes in the given revset from the local history.

jj bookmark list / set / track

Bookmarks are jj’s alternative to named git branches, and can be set up to automatically track a branch in a git remote. While compatibility with git branches is nice, names aren’t required by jj’s model. You can push your current unnamed change instantly with jj git push --change @, and jj will use the change ID (which stays the same across amends and rebases) as the git branch name. Now you don’t have to think of a good name for your branch before you can work on it (or push it!).

For more detail comparing and contrasting bookmarks to branches, I recommend the post Understanding Jujutsu bookmarks.

jj git push / fetch

It does what you would expect based on git, but the defaults are different than you might expect. Unless you configure the git.fetch and git.push settings, jj will only push to or fetch from origin. To operate on another remote, pass --remote NAME. To operate on all remotes, use glob:* as the remote name.

jj rebase / squash

The rebase command works like you would expect, but better. You can rebase a single change to a different place with jj rebase -r id --insert-before A, or rebase a change and all it’s descendants with jj rebase -s id --insert-after B. You can even rebase an entire branch automatically with jj rebase -b @ --destination C, moving every ancestor of @ that is not an ancestor of C into a new chain of commits descending from C. I did all of these constantly in git, and it’s much more involved.

The squash command is just a clear, single command for the common git operation where you move a diff into a commit or move a diff out of a commit, by change ID and/or filename.

jj merge (doesn’t exist)

The git rebase and merge commands (also including apply-patch, cherry-pick, and others) are all a bit special because they can create conflicts that have to be resolved before git will allow the commit to be… committed. This is the other half of the magic of jj: your new commit just holds any conflicts inside it. It’s impossible to lose work in a merge disaster because everything is always committed. You can resolve conflicts immediately, after other merges, or never! The results are always immediately stored, no matter how complete or incomplete your resolution is at the time.

Thanks to this feature, you don’t need a dedicated merge command—any new change can have however many parents you want, regardless of conflicts. It’s just as valid to jj new A B C D E as it is to jj new A. One pattern that is common in jj but was miserable in git is to create a “megamerge” combining all your current work branches. All editing happens on top of the megamerge, and you move individual changes backwards into a specific branch as you decide where to put them. Compared to git, it feels like magic.

commands beyond git

There are many jj commands that have no analogous git command. Some real standouts include jj absorb, jj parallelize, and jj undo. We’ll talk more about those commands in a future post about jj beyond git.

further command reading

The previously mentioned jj cheat sheet PDF has a second page, containing a quick summary of each command, what it does, and the arguments it accepts.

next time

Now that we have talked about commands, next up is workflows! How can you use jj to work on a pull request? How can you work on multiple branches or PRs at the same time?

Continue with jj part 3: workflows.

The full series also includes: part 4: configuration

For the last ten years or so of working on Bundler, I’ve had a wish rattling around: I want a bigger, better dependency manager. It doesn’t just manage your gems, it manages your ruby versions, too. It doesn’t just manage your ruby versions, it installs pre-compiled rubies so you don’t have to wait for ruby to compile from source over and over. And more than all of that, it makes it completely trivial to run any script or tool written in ruby, even if that script or tool needs a different ruby and gems than your application does.

For the entire ten years of daydreaming, I’ve been hoping someone else would build it and I could just use it. Then I discovered that someone did build it… but for Python. It’s called uv, and almost exactly one year ago version 0.3 shipped with all the features I had wished for, and even more that I hadn’t thought to wish for.

At this point, I’ve been using uv for almost a year and every time I use a project written in Python, the experience is delightful. Not only can you run a command directly out of any package that isn’t even installed, you can run a command that requires a python you don’t have installed, and uv takes care of installing the right python, installing the right packages, and running your command, in just a second or two.

Whether you want to run a CLI tool, a webapp, or a random script, uv always ensures the environment is correct as part of running the command. No more installing a new package version only to realize later you broke something old, no more setting up dependencies manually only to have the script running inside cron silently break later.

Earlier this year, my long time consulting job disappeared and I found myself looking for something to replace it. One of my ideas was to start a company inspired by Geomys in the Go language, offering expert advice from open source maintainers, but the idea felt weak to me without a “spotlight” project to show off our expertise.

In July of this year, I finally realized that these two ideas could go together extremely well—the company can show our expertise by building this developer tool, and clients paying for our advice to solve their problems can ensure we are able to support and expand the tool.

I talked to some Ruby friends about the idea, and it resonated with them, so we started working on both the company and the open source project. Today, Spinel Cooperative has a website at spinel.coop, and rv has a website at rv.dev. The team has expanded, and includes notable RubyGems and Bundler contributors Samuel Giddins and David Rodriguez, notable Rails contributors Kasper Timm Hanson and Sam Stephenson, who is also the original creator of rbenv and ruby-build.

Our goal is a completely new kind of management tool, where you don’t need to install rvm and then some ruby and then update rubygems and bundler and then bundle install your gems—you just run your command, and everything is handled. Not a version manager, or a dependency manager, but both of those things and much more.

With that vision in place, we were now faced with a very practical question. What can we build that would be useful right away? We landed on precompiled rubies for development work as the most useful place to start, and got to work.

We’re using Rust to build rv, for two reasons. The obvious reason is that Rust produces very fast results, which is also why our biggest inspiration uv is written in Rust. The less obvious reason is based on years of trying to onboard new contributors to Bundler and RubyGems—it turns out if you are a Ruby developer, you unfortunately don’t (yet) know the subset of Ruby that we are forced to use for Bundler and RubyGems.

There are two major things that basically every Ruby program does that you can’t do if you are managing gems. First, you can’t use any gems. If you want to use code that’s inside a gem, you need to copy that code wholesale into Bundler or RubyGems, and then you need to constantly update it anytime that gem has any changes. Second, you can’t use anything with native extensions, ever. JSON gem? Psych gem for YAML? Completely impossible, because Bundler and RubyGems need to be installable even if there is no compiler present.

So with those constraints in mind, and with a clear goal in mind of a tool so fast you normally can’t even tell it’s running, we settled on Rust, and started building a CLI. I’ve used Rust for smaller personal projects in the past, but never created a full CLI tool. I am happy to report that the clap library for creating CLIs in Rust is great, and recommend it to anyone who might be interested.

The next piece that we needed was precompiled Rubies. There are a few big projects out there compiling Ruby in advance, mostly for use on servers. The setup-ruby GitHub action and the official Ruby docker images are both based on the ruby-build project originally started as part of rbenv.

Unfortunately, those existing precompiled Ruby versions aren’t usable for our needs because they aren’t statically compiled and because they aren’t relocatable. Statically compiled (as opposed to dynamically compiled) means that Ruby copies the code from a shared library into its own binary.

Show of hands… have any of you ever had trouble compiling Ruby because of OpenSSL? Okay, put your hands down. Now, how many of you have had an already-installed Ruby suddenly stop working because of OpenSSL, and you had to install it again? Good news, rv fixes both of those problems by putting the OpenSSL inside the Ruby, so they can never get separated.

There is a tradeoff here—if there is a critical security flaw in OpenSSL, we will need to compile Ruby again to include the critical security update. The first reason we are okay with this tradeoff is that OpenSSL doesn’t have huge security issues very often. The second reason we are okay with this is that your production servers are probably using the official Ruby docker images and not Ruby installed by rv, so it’s even less of a concern.

In the end, the closest existing system we were able to build on top of was Homebrew’s portable-ruby project. That’s the system Homebrew uses to build the Ruby install that Homebrew itself runs on. The Homebrew team built some excellent infrastructure for building a statically linked Ruby, and even added the changes needed to make sure that Ruby could be relocated.

Since Homebrew needs to be able to install into /usr/local on x86, but /opt/homebrew on Apple Silicon, and into any user’s home directory for Linuxbrew, they need to be able to take a single precompiled Ruby and put it in any location on disk. That’s another one of the requirements that isn’t met by the setup-ruby or Docker image Rubies—if you move them to another directory, they stop working.

Using Homebrew’s portable-ruby as a base, we were able to start with macOS ARM and Ubuntu x86, add Ubuntu on ARM, and then build every version in the Ruby 3.4.x series. After a few weeks of work, we had some initial functionality working. rv could switch between installed Ruby versions in zsh, but most importantly it could install precompiled Ruby 3.4.x on macOS and Ubuntu in one second flat. Yes, you heard that right. rv ruby install 3.4.5. Wait 1 second. Done. You can run Ruby commands now.

With that proof of concept complete, we announced version 0.1. People got very excited! It was fantastic to see how many people were excited by the vision for a new, fast tool for Ruby development.

It’s been a few weeks since that 0.1 release, and we have been hard at work. We’ve expanded the team to include community contributors, merged pull requests, and compiled more Rubies than ever.

When 0.2 is released, in the very near future, it will include support for not just zsh, but bash, fish, and nushell. We have also added support for macOS on x86, meaning we now fully support x86 and ARM on both macOS and Linux. Finally, the precompiled Ruby versions available will expand to include all of Ruby 3.3 as well as 3.4. Just to top all of that off, every version of Ruby will have YJIT built in.

Our short-term plans include finishing support for Ruby 3.2, adding rubies that work with musl libc on Linux, and testing on more Linux distributions. Our longer-term plans including improving the way gems are compiled, so that installing your entire application and all of its gems can happen in just a few seconds.

We want to live in a future where anyone can run a Ruby command, or tool, or application in seconds (or less!). We’re going to build that future, for ourselves and for everyone else.

WARNING: This content was written for (and presented at) the inagural JJ Con, a conference for jj enthusiasts and contributors. If you’re new to using jj, I strongly recommend you read my other posts about jj first: part 1, part 2, part 3, part 4.

Welcome to “stupid jj tricks”. Today, I’ll be taking you on a tour through many different jj configurations that I have collected while scouring the internet. Some of what I’ll show is original research or construction created by me personally, but a lot of these things are sourced from blog post, gists, GitHub issues, Reddit posts, Discord messages, and more.

To kick things off, let me introduce myself. My name is André Arko, and I’m probably best known for spending the last 15 years maintaining the Ruby language dependency manager, Bundler. In the jj world, though, my claim to fame is completely different: Steve Klabnik once lived in my apartment for about a year, so I’m definitely an authority on everything about jj. Thanks in advance for putting into the official tutorial that whatever I say here is now authoritative and how things should be done by everyone using jj, Steve.

general configuration

The first jj tricks that I’d like to quickly cover are some of the most basic, just to make sure that we’re all on the same page before we move on to more complicated stuff.

To start with, did you know that you can globally configure jj to change your name and email based on a path prefix? You don’t have to remember to set your work email separately in each work repo anymore.

[[--scope]]
--when.repositories = ["~/work"]
[--scope.user]
email = "me@work.domain"

I also highly recommend trying out multiple options for formatting your diffs, so you can find the one that is most helpful to you. A very popular diff formatter is difftastic, which provides syntax aware diffs for many languages. I personally use delta, and the configuration to format diffs with delta looks like this:

[[--scope]]
--when.commands = ["diff", "show"]
[--scope.ui]
pager = "delta"
diff-formatter = ":git"

Another very impactful configuration is which tool jj uses to handle interactive diff editing, such as in the jj split or jj squash -i commands. While the default terminal UI is pretty good, make sure to also try out Meld, an open source GUI.

[ui]
diff-editor = "meld" # or vimdiff, vscode, etc

In addition to changing the diff editor, you can also change the merge editor, which is the program that is used to resolve conflicts. Meld can again be a good option, as well as any of several other merging tools.

[ui]
merge-editor = "meld" # or vimdiff, vscode, mergiraf etc

Tools like mergiraf provide a way to attempt syntax-aware automated conflict resolution before handing off any remaining conflicts to a human to resolve. That approach can dramatically reduce the amount of time you spend manually handling conflicts.

You might even want to try FileMerge, the macOS developer tools built-in merge tool. It supports both interactive diff editing and conflict resolution.

[merge-tools.filemerge]
program = "open"
edit-args = ["-a", "FileMerge", "-n", "-W", "--args",
             "-left", "$left", "-right", "$right",
             "-merge", "$output"]
merge-args = ["-a", "FileMerge", "-n", "-W", "--args",
              "-left", "$left", "-right", "$right",
              "-ancestor", "$base", "-merge", "$output",]

Just two more configurations before we move on to templates. First, the default subcommand, which controls what gets run if you just type jj and hit return. The default is to run jj log, but my own personal obsessive twitch is to run jj status constantly, and so I have changed my default subcommand to status, like so:

[ui]
default-command = ["status"]

The last significant configuration is the default revset used by jj log. Depending on your work patterns, the multi-page history of commits in your current repo might not be helpful to you. In that case, you can change the default revset shown by the log command to one that’s more helpful. My own default revset shows only one change from my origin. If I want to see more than the newest change from my origin I use jj ll to get the longer log, using the original default revset. I’ll show that off later.

[revsets]
log = "(trunk()..@):: | (trunk()..@)-"

templates

Okay, enough of plain configuration. Now let’s talk about templates! Templates make it possible to do many, many things with jj that were not originally planned or built in, and I think that’s beautiful.

First, if you haven’t tried this yet, please do yourself a favor and go try every builtin jj template style for the log command. You can list them all with jj log -T, and you can try them each out with jj log -T NAME. If you find a builtin log style that you especially like, maybe you should set it as your default template style and skip the rest of this section. For the rest of you sickos, let’s see some more options.

The first thing that I want to show you all is the draft commit description. When you run jj commit, this is the template that gets generated and sent to your editor for you to complete. Since I am the kind of person who always sets git commit to verbose mode, I wanted to keep being able to see the diff of what I was committing in my editor when using jj. Here’s what that looks like:

[templates]
draft_commit_description = '''
  concat(
    coalesce(description, default_commit_description, "\n"),
    surround(
      "\nJJ: This commit contains the following changes:\n", "",
      indent("JJ:     ", diff.stat(72)),
    ),
    "\nJJ: ignore-rest\n",
    diff.git(),
  )
'''

If you’re not already familiar with the jj template functions, this uses concat to combine strings, coalesce to choose the first value that isn’t empty, surround to add before+after if the middle isn’t empty, and indent to make sure the diff status is fully aligned. With this template, you get a preview of the diff you are committing directly inside your editor, underneath the commit message you are writing.

Now let’s look at the overridable subtemplates. The default templates are made of many repeated pieces, including IDs, timestamps, ascii art symbols to show the commit graph visually, and more. Each of those pieces can be overrides, giving you custom formats without having to change the default template that you use.

For example, if you are a UTC sicko, you can change all timestamps to render in UTC like 2025-02-17 21:23:47.000 +00:00, with this configuration:

[template-aliases]
"format_timestamp(timestamp)" = "timestamp.utc()"

Or alternatively, you can force all timestamps to print out in full, like 2025-02-13 01:53:08.000 -08:00 (which is similar to the default, but includes the time zone) by returning just the timestamp itself:

[template-aliases]
"format_timestamp(timestamp)" = "timestamp"

And finally you can set all timestamps to show a “relative” distance, like 7 months ago, rather than a direct timestamp:

[template-aliases]
"format_timestamp(timestamp)" = "timestamp.ago()"

Another interesting example of a template fragment is supplied by @scott2000 on GitHub, who changes the node icon specifically to show which commits might be pushed on the next jj git push command.

[templates]
log_node = '''
if(self && !current_working_copy && !immutable && !conflict && in_branch(self),
  "◇",
  builtin_log_node
)
'

[template-aliases]
"in_branch(commit)" = 'commit.contained_in("immutable_heads()..bookmarks()")'

This override of the log_node template returns a hollow diamond if the change meets some pushable criteria, and otherwise returns the builtin_log_node, which is the regular icon.

It’s not a fragment, but I once spent a good two hours trying to figure out how to get a template to render just a commit message body, without the “title” line at the top. Searching through all of the built-in jj templates finally revealed the secret to me, which is a template function named remove_prefix(). With that knowledge, it becomes possible to write a template that returns only the body of a commit message:

description_body = 'description.remove_prefix(description.first_line()).trim_start()'

We first extract the title line, remove that from the front, and then trim any whitespace from the start of the string, leaving just the description body.

Finally, I’d like to briefly look at the possibility of machine-readable templates. Attempting to produce JSON from a jj template string can be somewhat fraught, since it’s hard to tell if there are quotes or newlines inside any particular value that would need to be escaped for a JSON object to be valid when it is printed. Fortunately, about 6 months ago, jj merged an escape_json() function, which makes it possible to generate valid JSON with a little bit of template trickery. For example, we could create a log output of a JSON stream document including one JSON object per commit, with a template like this one:

log_json_stream = '''
  "{" ++ 
    "change_id".escape_json() ++ ": " ++ stringify(change_id).escape_json() ++ ", " ++ 
    "author".escape_json() ++ ": " ++ stringify(author).escape_json() ++
  "}\n"
'''

This template produces valid JSON that can then be read and processed by other tools, looks like this.

Update: there is now a json() template function, which makes it much simpler to output valid JSON, like so: jj log --no-graph -T 'json(self) ++ "\n"'.

Templates have vast possibilities that have not yet been touched on, and I encourage you to investigate and experiment yourself.

revsets

Now let’s look at some revsets. The biggest source of revset aliases that I have seen online is from @thoughtpolice’s jjconfig gist, but I will consolidate across several different config files here to demonstrate some options.

The first group of revsets roughly corresponds to “who made it”, and composes well with other revsets in the future. For example, it’s common to see a user(x) type alias, and a mine() type alias to let the current user easily identify any commits that they were either author or committer on, even if they used multiple different email addresses.

'user(x)' = 'author(x) | committer(x)'
'mine()' = 'user("me@personal.domain") | user("me@domain")'

Another group uses description prefixes to identify commits that have some property, like WIP or “private”. It’s then possible to use these in other revsets to exclude these commits, or even to configure jj to refuse to push them.

'wip()' = 'description(glob:"wip:*")'
'private()' = 'description(glob:"private:*")'

Thoughtpolice seems to have invented the idea of a stack, which is a group of commits on top of some parent:

# stack(x, n) is the set of mutable commits reachable from 'x',
# with 'n' parents. 'n' is often useful to customize the display
# and return set for certain operations. 'x' can be used to target
# the set of 'roots' to traverse, e.g. @ is the current stack.
'stack()' = 'stack(@)'
'stack(x)' = 'stack(x, 2)'
'stack(x, n)' = 'ancestors(reachable(x, mutable()), n)'

Building on top of the stack, it’s possible to construct a set of commits that are “open”, meaning any stack reachable from the current commit or other commits authored by the user. By setting the stack value to 1, nothing from trunk or other remote commits is included, so every open commit is mutable, and could be changed or pushed.

'open()' = 'stack(mine() | @, 1)'

Finally, building on top of the open revset, it’s possible to define a “ready” revset that is every open change that isn’t a child of wip or private change:

'ready()' = 'open() ~ descendants(wip() | private())'

It’s also possible to create a revset of “interesting” commits by using the opposite kind of logic, as in this chain of revsets composed by @sunshowers.

'uninterested()' = '::remote_bookmarks() | tags()'
'interested()' = 'mine() ~ uninterested()'
'open()' = '''
    ancestors(interested(), 3)
      | tracked_remote_bookmarks()
      | ancestors(@, 3)
'''

You take remote commits and tags, then subtract those from our own commits, and then show anything that is either local-only, tracking the remote, or close to the current commit.

commands

Now let’s talk about jj commands. You probably think I mean creating jj commands by writing our own aliases, but I don’t! That’s the next section. This section is about the jj commands that it took me weeks or months to realize existed, and understand how powerful they are.

First up: jj absorb. When I first read about absorb, I thought it was the exact inverse of squash, allowing you to choose a diff that you would bring into the current commit rather than eject out of the current commit. That is wildly wrong, and so I want to make sure that no one else falls victim to this misconception. The absorb command iterates over every diff in the current commit, finds the previous commit that changed those lines, and squashes just that section of the diff back to that commit. So if you make changes in four places, impacting four previous commits, you can jj absorb to squash all four sections back into all four commits with no further input whatsoever.

Then, jj parallelize. If you’re taking advantage of jj’s amazing ability to not need branches, and just making commits and squashing bits around as needed until you have each diff combined into one change per thing you need to submit… you can break out the entire chain of separate changes into one commit on top of trunk for each one by just running jj parallelize 'trunk()..@' and letting jj do all the work for you.

Last command, and most recent one: jj fix. You can use fix to run a linter or formatter on every commit in your history before you push, making sure both that you won’t have any failures and that you won’t have any conflicts if you try to reorder any of the commits later.

To configure the fix command, add a tool and a glob in your config file, like this:

[fix.tools.black]
command = ["/usr/bin/black", "-", "--stdin-filename=$path"]
patterns = ["glob:'**/*.py'"]

Now you can just jj fix and know that all of your commits are possible to reorder without causing linter fix conflicts. It’s great.

aliases

Okay. Now we can talk about command aliases. First up, the venerable tug. In the simplest possible form, it takes the closest bookmark, and moves that bookmark to @-, the parent of the current commit.

tug = ["bookmark", "move", "--from", "heads(::@- & bookmarks())", "--to", "@-"]

What if you want it to be smarter, though? It could find the closest bookmark, and then move it to the closest pushable commit, whether that commit was @, or @-, or @---. For that, you can create a revset for closest_pushable, and then tug from the closest bookmark to the closest pushable, like this:

[revset-aliases]
'closest_pushable(to)' = 'heads(::to & mutable() & ~description(exact:"") & (~empty() | merges()))'
[aliases]
tug = 'bookmark move --from "heads(::@ & bookmarks())" --to "closest_pushable(@)"'

Now your bookmark jumps up to the change that you can actually push, by excluding immutable, empty, or descriptionless commits.

What if you wanted to allow tug to take arguments, for those times when two bookmarks are on the same change, or when you actually want to tug a different bookmark than the closest one? That’s also pretty easy, by adding a second variant of the tug command that takes an argument:

tug = ["util", "exec", "--", "sh", "-c", """
if [ "x$1" = "x" ]; then
  jj bookmark move --from "closest_bookmark(@)" --to "closest_pushable(@)"
else
  jj bookmark move --to "closest_pushable(@)" "$@"
fi
""", ""]

This version of tug works just like the previous one if no argument is given. But if you do pass an argument, it will move the bookmark with the name that you passed instead of the closest one.

How about if you’ve just pushed to GitHub, and you want to create a pull request from that pushed bookmark? The gh pr create command isn’t smart enough to figure that out automatically, but you can tell it which bookmark to use:

pr = ["util", "exec", "--", "bash", "-c", """
gh pr create --head $(jj log -r 'closest_bookmark(@)' -T 'bookmarks' --no-graph | cut -d ' ' -f 1)
"""]

Just grab the list of bookmarks attached to the closest bookmark, take the first one, pass it to gh pr create, and you’re all set.

What if you just want single commands that let you work against a git remote, with defaults tuned for automatic tugging, pushing, and tracking? I’ve also got you covered.

init = ["util", "exec", "--", "bash", "-c", """
jj git init --colocate
# only track origin branches, not upstream or others
jj bookmark track 'glob:*@origin'
"""]

Use jj init to colocate jj into this git repo, and then track any branches from upstream, like you would get from a git clone.

pull = ["util", "exec", "--", "bash", "-c", """
closest="$(jj log -r 'closest_bookmark(@)' -n 1 -T 'bookmarks' --no-graph | cut -d ' ' -f 1)"
closest="${closest%\\*}"
jj git fetch
jj log -n 1 -r "${closest}" 2>&1 > /dev/null && jj rebase -d "${closest}" || jj rebase -d 'trunk()'
jj log -r 'stack()'
"""]

Then, you can jj pull to find the closest bookmark to @, do a git fetch, rebase your current local commits on top of whatever just got pulled, and then show your new stack. When you’re done, just jj push.

push = ["util", "exec", "--", "bash", "-c", """
tuggable="$(jj log -r 'closest_bookmark(@)..closest_pushable(@)' -T '"n"' --no-graph)"
[[ -n "$tuggable" ]] && jj tug
pushable="$(jj log -r 'remote_bookmarks(remote=origin)..@' -T 'bookmarks' --no-graph)"
[[ -n "$pushable" ]] && jj git push || echo "Nothing to push."
closest="$(jj log -r 'closest_bookmark(@)' -n 1 -T 'bookmarks' --no-graph | cut -d ' ' -f 1)"
closest="${closest%\\*}"
tracked="$(jj bookmark list -r ${closest} -t -T 'if(remote == "origin", name)')"
[[ "$tracked" == "$closest" ]] || jj bookmark track "${closest}@origin"
"""]

This push handles looking for a tuggable bookmark, tugging it, doing a git push, and making sure that you’re tracking the origin copy of whatever you just pushed, in case you created a new branch.

combo tricks

Last, but definitely most stupid, I want to show off a few combo tricks that manage to deliver some things I think are genuinely useful, but in a sort of cursed way.

First, we have counting commits. In git, you can pass an option to log that simply returns a number rather than a log output. Since jj doesn’t have anything like that, I was forced to build my own when I wanted my shell prompt to show how many commits beyond trunk I had committed locally. In the end, I landed on a template consisting of a single character per commit, which I then counted with wc.

jj log --no-graph -r "main..@ & (~empty() | merges())" -T '"n"' 2> /dev/null | wc -c | tr -d ' '

That’s the best anyone on GitHub could come up with, too. See? I warned you it was stupid.

Next, via @marchyman on Discord, I present: jj log except for the closest three commits it also shows jj status at the same time.

[aliases]
ll = ["log", "-T", "log_with_files"]

[revset-aliases]
'recent_work' = 'ancestors(visible_heads(), 3) & mutable()'

[template-aliases]
log_with_files = '''
if(root,
  format_root_commit(self),
  label(if(current_working_copy, "working_copy"),
    concat(
      format_short_commit_header(self) ++ "\n",
      separate(" ",
        if(empty, label("empty", "(empty)")),
        if(description,
          description.first_line(),
          label(if(empty, "empty"), description_placeholder),
        ),
      ) ++ "\n",
      if(self.contained_in("recent_work"), diff.summary()),
    ),
  )
)
'

Simply create a new template that copies the regular log template, while inserting a single conditional line that adds diff.summary() if the current commit is inside your new revset that covers the newest 3 commits. Easy. And now you know how to create the jj ll alias I promised to explain earlier.

Last, but definitely most stupid, I have ported my previous melding of git branch and fzf over to jj, as the subcommand fuzzy_bookmark, which I alias to jj z because it’s inspired by zoxide, the shell cd fuzzy matcher with the command z.

z = ["fuzzy_bookmark"]
za = ["bookmark", "list", "-a"]

fuzzy_bookmark = ["util", "exec", "--", "sh", "-c", """
if [ "x$1" = "x" ]; then
  jj bookmark list
else
  jj bookmark list -a -T 'separate("@", name, remote) ++ "\n"' 2> /dev/null | sort | uniq | fzf -f "$1" | head -n1 | xargs jj new
fi
""", ""]

This means you can jj z to see a list of local bookmarks, or jj za to see a list of all bookmarks including remote branches. Then, you can jj z some to do a fuzzy match on something, and execute jj new something. Jump to work on top of any named commit trivially by typing a few characters from its name.

shell prompt tricks

I would love to also talk about all the stupid shell prompt tricks that I was forced to develop while setting up a zsh prompt that includes lots of useful jj information without slowing down prompt rendering, but I’m already out of time. Instead, I will refer you to my blog post about a jj prompt for powerlevel10k, and you can spend another 30 minutes going down that rabbit hole whenever you want.

acknowledgements

Finally, I want to thank some people. Most of all, I want to thank everyone who has worked on creating jj, because it is so good.

I also want to thank everyone who has posted their configurations online, inspiring this talk. All the people whose names I was able to find in my notes include @martinvonz, @thoughtpolice, @pksunkara, @scott2000, @avamsi, @simonmichael, and @sunshowers. If I missed you, I am very sorry, and I am still very grateful that you posted your configuration.

Last, I need to thank @steveklabnik and @endsofthreads for being jj-pilled enough that I finally tried it out and ended up here as a result.

Thank you so much, to all of you.

pls, I just want to use jj with GitHub

Sure, you can do that. Convert an existing git repo with jj git init --colocate or clone a repo with jj git clone. Work in the repo like usual, but with no add needed, changes are staged automatically.

Commit with jj commit, mark what you want to push with jj bookmark set NAME, and then push it with jj git push. If you make any additional changes to that branch, update the branch tip by running jj bookmark set NAME again before each push.

Get changes from the remote with with jj git fetch. Set up a local copy of a remote branch with jj bookmark track NAME@REMOTE. Check out a branch with jj new NAME, and then loop back up to the start of the previous paragraph for commit and push. If you just want a slightly more detailed version of that, try jj for my workflow. That’s probably all you need to get started, so good luck and have fun!

jj concepts

Still here? Cool, let’s talk about how jj is different from git. There’s a list of differences from git in the jj docs, but more than specific differences, I found it helpful to think of jj as like git, but every change in the repo creates a commit.

Edit a file? There’s a commit before the edit and after the edit. Run a jj command? There’s a commit before the command and after the command. Some really interesting effects fall out of storing every action as a commit, like no more staging, trivial undo, committed conflicts , and change IDs.

When edits are always immediately committed, you don’t need a staging area, or to manually move files into the staging area. It’s just a commit, and you can edit it by editing the files on disk directly.

Any jj command you run can be fully rewound, because any command creates a new operation commit in the op log. No matter how many commits you just revised in that rebase, you can perfectly restore their previous state by running jj undo.

Any merge conflict is stored in the commit itself. A rebase conflict doesn’t stop the rebase—your rebase is already done, and now has some commits with conflicts inside them. Conflicts are simply commits with conflict markers, and you can fix them whenever you want. You can even rebase a branch full of conflicts without resolving them! They’re just commits. (Albeit with conflict markers inside them.)

Ironically, every action being a commit also leads away from commits: how do you talk about a commit both before and after you amended it? You add change IDs. Changes give you a single identifier for your intention, even as you need many commits to track how you amended, rebased, and then merged those changes.

Once you’ve internalized a model where every state is a commit, and change IDs stick around through amending commits, you can do some wild shenanigans that used to be quite hard with git. Five separate PRs open but you want to work with all of them at once? Easy. Have one commit that needs to be split into five different new commits across five branches? Also easy.

One other genius concept jj offers is revsets. In essence, revsets are a query language for selecting changes, based on name, message, metadata, parents, children, or several other options. Being able to select lists of changes easily is a huge improvement, especially for commands like log or rebase.

further reading

For more about jj’s design, concepts, and why they are interesting, check out the blog posts jj strategy, What I’ve Learned From JJ, jj init, and jj is great for the wrong reason. For a quick reference you can refer to later, there’s a single page summary in the jj cheat sheet PDF.

next time

Keep an eye out for the next part of this series in the next few days. We’ll talk about commands in jj, and exactly how they are both different and better than git commands.

Continue with jj part 2: commands & revsets.

The full series also includes: part 3: workflows, part 4: configuration

I didn’t come up with the original idea for Bundler (that was Yehuda). I also didn’t work on the first six months worth of prototypes. That was all Carl and Yehuda together, back when “Carlhuda” was a super-prolific author of Ruby libraries, including most of the work to modularize Rails for version 3.

I joined the team at a pivotal moment, in February 2010, as the 0.9 prototype was starting to be re-written yet another time into the shape that would finally be released as 1.0. By the time Carl, Yehuda, and I released version 1.0 together in August 2010, we had fully established the structure and commands that Bundler 2.7.2 still uses today.

I gave my first conference talk about Bundler at Red Dirt Ruby in May 2010. Because they would be too busy with Rails 3 talks, Yehuda and Carl asked me to give the first RailsConf talk about Bundler, in June 2010.

As Carl and Yehuda drifted off to other projects, in 2011 and 2012 respectively, I took on a larger role, co-maintaining the project with Terence Lee, then on the Ruby platform team at Heroku. We shipped (and, embarrassingly, broke) many versions of Bundler on our way to the 1.1 release and its major speed improvements. We also gave several conference talks together, sharing what we had learned about Bundler, about gems, and about maintaining open source.

In 2013, I managed to convince the owner of bundler.io to sell me his domain, and rebuilt the website to host a separate copy of the documentation for every version of Bundler, ensuring even users on old versions could still access accurate documentation.

By the end of 2013, Terence had drifted away from the project as well, and I realized that everyone using Ruby was now one bus (or one lottery ticket) away from Bundler having no significant maintainers. During 2014, I made sure to settle any remaining ownership issues, including purchasing the rights to the Bundler logo, and began investigating various funding ideas. I tried specialized consulting, corporate sponsorships, and asking Ruby Central about sponsoring Bundler and RubyGems development work. Ruby Central declined, citing their desire to stay focused on conferences, but suggested that if I wanted to pursue something myself they would be happy to collaborate.

In 2015, I founded Ruby Together specifically to raise funds to pay the existing maintainers team of Bundler, RubyGems, and RubyGems.org. Over time, we were able to raise enough money to quietly but scrappily keep the entire RubyGems ecosystem maintained and functional. Ruby Together did not ever, at any point, demand any form of governance or control over the existing open source projects. Maintainers did their thing in the RubyGems and Bundler GitHub orgs, while Ruby Together staff and board members did their thing in the rubytogether GitHub org.

By 2021, when Ruby Central and Ruby Together were both interested in merging together, funds were harder to find. Ruby Together had a membership program. Ruby Central wanted a to have a membership program. The confusing split between “Ruby Central owns the AWS account, but Ruby Together pays all the devs” continued to be a problem.

We prepared a merger agreement (which you can read in full at the link), stating that Ruby Central’s new goal after the merger would be “paying maintainers to do the programming”. The agreement also states that Ruby Central will follow Ruby Together’s Vision, Mission, and Values, a document that is hosted in the rubycentral GitHub organization today. That document includes a very specific list of goals, including:

• Project users and maintainers are empowered to decide what’s best for their projects
• Ruby open source developers are paid for their work
• Give control to the community
• Be accountable and transparent to the community
• Establish a collaborative, positive space for projects
• Have a clear and transparent funding process

You can read much more in both the merger agreement and in the Mission, Vision, and Values document, but the fundamental goal for both the non-profit and the open source projects is clear: this is all for the Ruby community. Without the community, there is no point to this work, and there is no way it could ever have been done in the first place. Without the 354 individuals who contributed to Bundler and to RubyGems, I could never have become “the Bundler guy” in the first place.

In the last few weeks, Ruby Central has suddenly asserted that they alone own Bundler. That simply isn’t true. In order to defend the reputation of the team of maintainers who have given so much time and energy to the project, I have registered my existing trademark on the Bundler project.

Trademarks do not affect copyright, which stays with the original contributors unchanged. Trademarks do not affect license terms, which stay MIT and unchanged. Trademarks only impact one thing: who is allowed say that what they make is named “Bundler”. Ruby Central is welcome to the code, just like everyone else. They are not welcome to the project name that the Bundler maintainers have painstakingly created over the last 15 years.

While the trademark has been registered under my name as an individual, I will not keep it for myself, because the idea of Bundler belongs to the Ruby community. Once there is a Ruby organization that is accountable to the maintainers, and accountable to the community, with openly and democratically elected board members, I commit to transfer my trademark to that organization.

I will not license the trademark, and will instead transfer ownership entirely. Bundler should belong to the community, and I want to make sure that is true for as long as Bundler exists.

Last week, David Barsky of ERSC sent me an extremely compelling counter-example, and I spent several days running benchmarks to understand it better.

The top level summary is that the test suite for the jj VCS exhibits an absolutely huge difference between running on an SSD and running against a ramdisk. In my first reproduction attempt, I found the SSD took 239 seconds, while the ramdisk took just 37 seconds. That’s bananas! How was that even possible?

What I discovered will amaze, distress, and astound you. Probably.

First, the context. The jj project recently shipped a change to always use fdatasync() when persisting a temporary file. My understanding is that this change was made to prevent certain kinds of bad data being written.

After adding more calls to fdatasync(), which is a variation of fsync(), contributors to jj noticed that the tests ran about the same speed on linux, but dramatically slower on macOS. This eventually produced a pull request suggesting a ramdisk for tests on macOS, noting that it was much faster.

This situation intrigued me—how much faster was it? And why? At the highest level, there is an explanation that makes sense to me: testing fdatasync() causes a huge difference between physical disks and RAM, because it breaks through the filesystem cache in memory, and forces slow disk writes before returning.

But then I actually tested the suggested change on two different machines, and what I was seeing didn’t make any sense. I had access to two different Macs for testing: one M4 Max with 16 CPU cores, and one M3 Ultra with 32 CPU cores.

When I tried SSD vs RAM on 16 cores, the difference was huge. When I tried SSD vs RAM on 32 cores, the difference was… much smaller. That’s confusing. The cargo nextest run command will (by default) run one test binary on every core available on the machine. Why would running the tests on more cores make the tests slower?

Since it seemed like I was getting inconsistent results, I eventually used hyperfine to systematically run the entire test suite 10 times using 1 core, 2 cores, 3 cores, 4 cores, etc, all the way up to the full 32 cores in my M3 Ultra testing box.

The results I saw for using the SSD made sense, mostly. Adding more cores made the tests run faster… up to about 4 cores. Cores 5 to 32, on the other hand, don’t seem to do anything at all. From the outside, that makes it look like the APFS filesystem on the SSD has some kind of mutex or lock that really only allows 4 cores to actually run at the same time. Running similar tests on the M4 Max produced similar results—APFS on SSD seems to test faster up to about 4 cores, and then get stuck there no matter how many more cores you add.

Where things started to get weird was using tmpfs on a ramdisk. On the M4 Max things went roughly how you might expect, with each additional core decreasing the overall runtime but with diminishing returns. The full test suite on one core takes about 327 seconds, and with 16 cores takes about 37 seconds. 15 cores is just a hair slower at 38, and so on.

On the M3 Ultra, though, using a ramdisk and testing from 1 to 32 cores produced worse results for every core added beyond the 12th. I’ve created a gist with raw benchmark output, but you can see the summary chart below.

Whatever is going on with the jj test suite and the ramdisk creates so much contention that 32 cores will all run full out at 100% while taking 3x longer than 12 cores running at 100%.

That’s pretty wild! In the end, it doesn’t seem to be a story about in-memory filesystems exactly. Instead, it’s about some kind livelock contention between the running cores and some shared, limited resource. I’m not sure if that resource is memory itself, some shared CPU cache, the IO bus, or what. But it sure is dramatic.

In the meantime, I’m looking forward to spending my new free time focusing on projects that I’m truly excited about, like rv. We’d love to have your help as we work to build next-generation tools for Ruby.