Ruby, RVM, MacPorts and OpenSSL on macOS (Monterey)

This is not so much a blog post but a log for the sake of future Me:

I have been using MacPorts on macOS for many years and am 99% happy with it: it allows installing binary packages and source ones with custom options, allows multiple versions to live side by side, has a lot of packages, can dump the whole list of installed stuff so you can re-install it elsewhere easily and many other things. It’s great.

I have also for a long time used RVM as my ruby version manager of choice. Again, it just works.

An issue with installing older rubies in recent macOS is that the system-provided OpenSSL is version 3, and ruby before 3.1 cannot build against it.

You can get OpenSSL1.0 or 1.1 via MacPorts, but there’s another problem: when you try to install anything recent via MacPorts you will end up with OpenSSL3, and at that point you will have problems building old ruby versions or old versions of gems like eventmachine or puma, expecting to have OpenSSL1.

This problem manifests itself when installing ruby 2.7 via RVM with a build/install log that ends like this

/Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- openssl (LoadError)
        from /Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/specification.rb:2430:in `to_ruby'
        from ./tool/rbinstall.rb:846:in `block (2 levels) in install_default_gem'
        from ./tool/rbinstall.rb:279:in `open_for_install'
        from ./tool/rbinstall.rb:845:in `block in install_default_gem'
        from ./tool/rbinstall.rb:835:in `each'
        from ./tool/rbinstall.rb:835:in `install_default_gem'
        from ./tool/rbinstall.rb:799:in `block in <main>'
        from ./tool/rbinstall.rb:950:in `block in <main>'
        from ./tool/rbinstall.rb:947:in `each'
        from ./tool/rbinstall.rb:947:in `<main>'

the ruby binary is actually built correctly but the openssl extension failed to build, which you can tell by looking back in the build history

*** Following extensions are not compiled:
openssl:
         Could not be configured. It will not be installed.
         Check ext/openssl/mkmf.log for more details.

If you look into .rvm/src/ruby-2.7.6/ext/openssl/mkmf.log you will find

/Users/riffraff/.rvm/src/ruby-2.7.6/ext/openssl/extconf.rb:111: OpenSSL >= 1.0.1, < 3.0.0 or LibreSSL >= 2.5.0 is required

So it seems to have ruby use OpenSSL1 you should uninstall OpenSSL3 and everything that depends on it.

FWIW, I think that works, but it’s not ideal.

But fear not! There’s a better solution! You just need to tell RVM (and ruby) to use look for headers and libraries in the right place.

To do this you need to both specify where pkgconfig is and which openssl directories to use.

Luckily, this is not difficult, follow this incantation

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig rvm reinstall 2.7.6 --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

You effectively have to tell ruby how to configure itself, and how to find OpenSSL1 at compile time and at link time.

In theory --with-openssl-dir should be enough, but AFAIU MacPorts puts stuff in separate directories which does not play well with ruby-build. Which is fair, as ruby-build does not explicitly support MacPorts, but rather homebrew.

Anyway, once this is solved you may still encounter issues when installing gems that depend on OpenSSL, as those will also need to be told where OpenSSL is.

For example,. this is the error I get installing puma 5.6.4

compiling puma_http11.c
linking shared-object puma/puma_http11.bundle
Undefined symbols for architecture arm64:
  "_SSL_get1_peer_certificate", referenced from:
      _engine_peercert in mini_ssl.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [puma_http11.bundle] Error 1

make failed, exit code 2

The equivalent error for eventmachine would be something like

linking shared-object rubyeventmachine.bundle
Undefined symbols for architecture arm64:
  "_SSL_get1_peer_certificate", referenced from:
      SslBox_t::GetPeerCert() in ssl.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [rubyeventmachine.bundle] Error 1

Luckily you the same technique works to install older gems that need OpenSSL1. Notice you need an extra -- to separate the arguments provided to the gem configuration script from the ones to the gem command.

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig gem install puma -v '5.6.4' -- --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

(also worth noting: puma 5.6.5 builds without problems, see this and this)

So, hopefully I will be able to forget old code that relies on OpenSSL1 at some point, but in case I need it again: hey future Me, this is how you workaround the issue!

A note on ruby 3.0

If you build ruby 3.0 with MacPorts it will not explode, but it will not build the OpenSSL extesion, in /Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/mkmf.log you will find the same error you’ve just seen

/Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/extconf.rb:113: OpenSSL >= 1.0.1, < 3.0.0 or LibreSSL >= 2.5.0 is required
	/Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/extconf.rb:113:in `<top (required)>'
	./ext/extmk.rb:214:in `load'
	./ext/extmk.rb:214:in `block in extmake'
	/Users/riffraff/.rvm/src/ruby-3.0.4/lib/mkmf.rb:331:in `open'
	./ext/extmk.rb:210:in `extmake'
	./ext/extmk.rb:572:in `block in <main>'
	./ext/extmk.rb:568:in `each'
	./ext/extmk.rb:568:in `<main>'

and this in turn will mean you can’t build puma or eventmachine or other things that depend on the ruby openssl extension

$ gem install puma
ERROR:  While executing gem ... (Gem::Exception)
    OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

But you’re good, the same solution as before will allow you to install that version of ruby too

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig rvm install 3.0 --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

From ruby 3.1 onwards, ruby will build just fine with the system-provided OpenSSL3, so you will be able to avoid these shenanigans.

You can still build against MacPorts’ version of OpenSSL1 or OpenSSL3 by using the right paths.

Happy hacking future Me!

Building ruby 2.7 on macOS with MacPorts and OpenSSL3

Recently I got a new apple box (an M1 MacBook Pro, which is nice, if a bit bulky), and found myself in the need to re-establish my dev environment.

This meant re-compiling various old ruby versions which I need for some projects. The problem is old ruby releases before ruby 3 required an equally ancient version of OpenSSL

I have used RVM for many years and it has a nice integration with MacPorts, my favorite macOS package manager. Getting the ports I need on a new install is straightforward, you just get a list of installed packages from one machine and reinstall them on the other. And RVM knows how to get rubies when you encounter a .ruby-version file, so no issue there.

But there’s a catch: I am using OpenSSL3 for most of my stuff, but I need OpenSSL1.0 or 1.1 to build ruby 2.7.5 and older.

If you google this, you will find plenty of bug reports against RVM, ruby-install, puma and plenty of others, with varying suggestion to use --with-openssl-dir, or setting PKG_CONFIG_PATH, overriding LD_FLAGS, and other incantations.

These may work in some cases, but not all: it seems an underlying problem is that if you have multiple versions of OpenSSL the ruby configure script may end up overriding the openssl-dir setting, and still end up linking against the incorrect library.

Luckily, the solution is pretty straightforward if you ignore all those recommendations 🙂

Try this

  • Install OpenSSL3: sudo port install openssl3
  • Install Ruby 3: rvm install 3.1
  • Check that it works and it loads openssl fine, by running something like ruby -ropenssl -e 'p [RUBY_VERSION, OpenSSL::VERSION]'
  • Install OpenSSL1 sudo port install openssl1
  • Remove v3: sudo port uninstall openssl3 (keep things that depended on it if you want)
  • Install older rubies: rvm install 2.7.5
  • Check those work too
  • Reinstall openssl3

Everything should now work fine, because MacPorts has no issues with multiple library versions sitting next to each other, and each ruby is linked to the appropriate one. Packages that depend on V1 or V3 will also just be happy next to each other.

Notice, I did this once already almost a year ago, and I had totally forgotten about it.

So here’s this small post, in the hope it may help someone, or at least my future self.

Advent of Code 2021: Day 6

I’ve been a bit busy so I skipped doing the tasks in Raku for a couple days, but today’s seemed fun. As usual SPOILERS ahead.

So the problem is basically: you have certain amounts of fish which have a “spawn-timer”, when they reach it they get reset to 6, and create a new fish with a spawn timer set at 8. Given an initial list of fish, we want to know how many fish we have after some days.

You, Clever Reader will certainly have noticed that we have an exponential behaviour: at each step we generate more fish, which will generate more more fish next time, and more more more fish and so on§.

The abstract solution in my mind would be:

  • set the initial values in some container structure
  • call some step function the given amount of days
  • count the total entries in the container

This, more or less

sub solve(@ints, $days) { # ints is something like [1, 2, 3, 3, 1, 4..]
  <whatever>
  step(<whatever>) for ^$days;
  [+] <whatever>;
}

the only interesting thing here is the construct

<do-something> for ^$number

where ^$number is a short form to define a range from 0 to $number and the rest is a short form to do a loop over the right hand size of the for keyword.

Extended this might look like

for 0..$days -> $day { <do-something> }

which I find less entertaining§.

The question to answer next is: what should we use as container?

Raku obviously has a Hash class, and we could initialize it with a default and then iterate over the initial values to build.

> my @ints= [1, 3, 3, 3, 3, 2]
[1 3 3 3 3 2]
> my %counts is default(0);
{}
> %counts{$_}++ for @ints
Nil
> %counts
{1 => 1, 2 => 1, 3 => 4}

The only new bit we see is the is default annotation that builds a container with a default value. But it seemed weird I have to do this, surely Raku has some builtin way to count things, like Ruby’s Enumerable#tally ?

And of course it does. It has More Than One Way To Do It, of course. For example, it provides a Bag class which is an immutable container for counting items, and can be extracted from another sequence trivially

> my @ints = [1, 3, 3, 3, 3, 2]
[1 3 3 3 3 2]
> @ints.Bag
Bag(1 2 3(4))

And a corresponding BagHash which returns a mutable container for counting.

Being a map from Any to Int these also come with a nice default, so when we access a missing item we get 0

> my $bh = @ints.BagHash
BagHash(1 2 3(4))
> $bh{1}
1
> $bh{2}
1
> $bh{3}
4
> $bh{99}
0

Wait, you might say, why are you using a scalar for that object, instead of a hash? Shouldn’t it be %bh rather than $bh?

The answer is: I don’t really know#raku @ libera.chat for teaching me.">§

> my %bh-as-hash = $bh
{1 => 1, 2 => 1, 3 => 4}
> %bh-as-hash{99}  # no default in the hash!
(Any)
> $bh{99}          # still default in the BagHash!
0
> $bh{99} = 3      # you can update the BagHash
3
> $bh{99}
3
> %bh-as-hash{99}  # but the hash is unaffected
(Any)

You can alternatively use the := binding operator, which performs an assignment without casting:

> my %bh-as-hash-casted := $bh  # notice the representation
BagHash(1 2 3(4) 99(3))
> %bh-as-hash-casted{100} = 100 # it's the same object!
100
> $bh{100}
100

So our full solve can be

sub solve(@ints, $days) {
  my $counts = @ints.BagHash;
  step($counts) for ^$days;
  [+] $counts.values;
}

Now, how does the step function look? It is pretty straightforward, but it has a tricky bit

sub step($counts) {
  my $old = $counts.clone;
  for 8...0 -> $idx {
    $counts{$idx} = $old{$idx + 1};
    if $idx == 0 {
      $counts{8} = $old{0};
      $counts{6} += $old{0};
    }
  }
}

Do you see the… oddity? Look at this:

> for 1..3 -> $i { say $i }
1
2
3
> for 1...3 -> $i { say $i }
1
2
3
> for 3..1 -> $i { say $i }
Nil
> for 3...1 -> $i { say $i }
3
2
1

you cannot iterate in reverse one of those! This is because the .. operator builds a Range while ... builds a Seq. A Range is list of consecutive numbers, with a beginning and an end, while a Seq is something that can be iterated over.

So there are no numbers in an inverse Range, and if you iterate over it you get zero iterations. In my very humble rubyist opinion, this is very confusing™ , but I guess real Raku people are used to it.

Back to our task: of course, the Astute Reader will have noticed we can do this more tersely by simply shifting the array and adding it to itself, and then brining back the extra items. This is a simpler solution, and it’s pretty nice, but we need to build a list rather than a BagHash:.

sub solve2(@ints, $days) {
  my @list is default(0);      # create a list with default value 0
  @list[$_]++ for @ints;       # add all the fish to it
  step2(@list) for ^$days;     # do the steps
  [+] @list;                   # add up the values in the list
}

I have a feeling the first two lines can be collapsed, but I don’t know how, if you have an idea please let me know in the comments.

Our step function becomes very small

sub step2(@counts) {
  @counts.rotate;
  @counts[6] += @counts[8];
}

And honestly, I feel this is as expressive as it can be.

See you next time!

Advent of Code 2021: Day 2

Hello again!

So, Day 2 is not a particularly interesting problem: basically we have a stream of commands, and we need to move a submarine correspondingly, then in the last step we need to compute the product of the depth and horizontal distance

For problem 2, we are going to have “up N”, “down N” and “forward N”, which will respectively affect the aim of the submarine and the position.

This would be pretty trivial to do with a loop and a case/switch statement (which in Raku is called given/when) but I think it’s a cool chance to try out Raku’s multimethods.

A class definition in Raku looks like this

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;
  method process($operation, $num) { ... }
}

The $. is a twigil which says “this is a scalar value and it’s private but has accessor methods“.§

Methods (and Routines in general) support multiple dispatch which can happen through class, traits, or any specific attribute of the parameter (see the Signature class). In our case, we want to dispatch through the operation name, so we can do this§

multi method process("forward", $num) {
  ...
}
multi method process("up", $num) {
  ...
}
multi method process("down", $num) {
  ...
}

The actual implementation would be: when we go “up” or “down, we want to alter the aim, and when we go “forward” we move by the given amount horizontally, and by aim * num vertically.

We could alter the values in the given clasinstance, but… it’s uglier! The declared instance variables are immutable, if we want them to be mutable we should have used the rw trait, like this

class Sub {
  has $.aim = 0 is rw;
  has $.depth = 0 is rw;
  has $.horizontal = 0 is rw;
}

Raku is pushing gently against us so we prefer immutability, and who am I to go against that?

EDIT: as noted by mscha in the comments, this is incorrect. The variables are mutable, we could always assign them from within the instance, referring to them with the $! twigil, e.g. $!aim += $num. The is rw only refers to generating setter methods so they’re accessible from outside. Sorry, I’m still learning.

So, each of our process() calls will instead produce a new object by adjusting the current one. Luckily, the language has a pretty nice functionality to help with that: the clone() method accepts an argument to tweak the cloned object, so our code would look like this

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;

  multi method process("forward", $num) {
    self.clone(horizontal => $.horizontal + $num, depth => $.depth + $num * $.aim)
  }
  multi method process("up", $num) {
    self.clone(aim => $.aim - $num)
  }
  multi method process("down", $num) {
    self.clone(aim => $.aim + $num)
  }
}

This looks pretty nice to me! Now we only need to plug this into processing the commands. But I thought, well, we decided to be immutable, right? So let’s do this with reduce.

Rather than use the reduce metaoperator like last time, we can use the actual reduce routine. It works similar to other similar function in Python/Ruby/JavaScript/whatever.

my @words = 'day2.txt'.IO.words; # read all words
my @pairs = @words.rotor(2); # pair them up
my $sub = Sub.new # get a sub
<something with $sub and @pairs>.reduce(<block>) # iterate on values

The problem, is that, from what I understand, we cannot pass an initial value (our $sub) as an argument. The way to handle that is through using a Slip.

A Slip allows us to treat a list “as if it was typed there”. For example we can use it to prepend items to a list

> my @list = 1,2,3
[1 2 3]
> my @list2 = 3, @list
[3 [1 2 3]]
> my @list2 = 3, |@list
[3 1 2 3]

Or to use a list as an argument to a multi-argument routine

> my @pair = 1,2
[1 2]
> sub f($a, $b) { say "a: $a, b $b" }
&f
> f @pair
Too few positionals passed; expected 2 arguments but got 1
  in sub f at <unknown file> line 1
  in block <unit> at <unknown file> line 1

> f |@pair
a: 1, b 2

This is akin to the concept of “splat” in ruby, tho it’s somewhat more formalized.

So, we can pass an initial value to reduce by simply prepending it to the commands array:

($sub, |@pairs).reduce(<something>)

something should be a routine that takes two arguments (the accumulator and the next element) and returns a new value for the accumulator. This means it will call process(), and we can once again use a Slip!

($sub, |@pairs).reduce({ $^a.process(|$^b) })

Notice the $^ sourcery: in Raku you can use the ^ twigil to access variables implicitly, they will just be assigned from the arguments in order of definition. This call is effectively equivalent to

($sub, |@pairs).reduce(-> $a, $b { $a.process($b[0], $b[1]) } )

Now we just have to add a method to compute the final value, which is depth * horizontal, and putting all together we get

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;

  multi method process("forward", $num) {
    self.clone(horizontal => $.horizontal + $num, depth => $.depth + $num * $.aim)
  }

  multi method process("up", $num) {
    self.clone(aim => $.aim - $num)
  }

  multi method process("down", $num) {
    self.clone(aim => $.aim + $num)
  }

  method value {
    $.horizontal * $.depth
  }

}

my @words = '2.txt'.IO.words;

say (Sub.new, |@words.rotor(2)).reduce({ $^a.process(|$^b) }).value

which I find somewhat nice. I have to say, I still don’t like twigils, and I think I’m missing some way to write objects more tersely, if you know Raku better than me please let me know in the comments.

See you next time, happy hacking!

Advent of Code 2021: Day 1

Oh oh oh! Finally it’s that time of the year again!

I love Advent of Code and I’ve played it with friends and colleagues for years. Usually, I try to solve it in Ruby, since I’m very familiar with it, and so the effort is only on task solving rather than on learning the language and tooling.

But this year, I thought I might do it both in Ruby and in Raku (The Language Formerly Known As Perl 6), by first solving it in Ruby, and then if I have time trying to solve it again in Raku.

So, here’s a quick view of a couple solutions to the problems in Day 1. Stick to the end for the cool solution past the initial line noise 🙂

SPOILERS AHEAD

The first problem is basically: given a list of values, count the number of times they increase

In Ruby 3, this could look like this

def solve1(array)
  array.                # [1, 2, 5, 4]
    .each_cons(2).      # [(1, 2), (2, 5), (5, 4)]
    .count { _1 < _2 }. # 2
end

AKA: get a sliding window of pairs from the initial array, then count the occurrences where the first value in the pair is less than the second value in the pair. Notice the fancy anonymous positional block arguments!

You can do more or less the same in Raku, but I’m not aware of a .count method that takes a block§, but well, we can still do this in three steps:

  • build the sliding window
  • select the items matching the condition
  • get the size of the filtered list

Raku does not have each_slice, instead it has a .rotor method which is a generalization of “iterate and group“, you can define in one go both the size of the group and the offset, i.e.

> my @ints = [1, 2, 3, 4, 5, 6, 7, 8, 9]
[1 2 3 4 5 6 7 8 9]
> @ints.rotor(2)  # group by 2 
((1 2) (3 4) (5 6) (7 8))
> @ints.rotor(2 => 1)  # group by 2, and move by one after 
((1 2) (4 5) (7 8))
> @ints.rotor(2 => -1)  # group by 2, and move back by one after 
((1 2) (2 3) (3 4) (4 5) (5 6) (6 7) (7 8) (8 9))

The actual signature of this method is even more flexible, but this should be enough for us.

To filter we can use .grep, which takes a block. We can do this javacript-like, passing a block with a pointed arrow.§

Then we get the size of the list, which is done with prefix '+' . This is an odd perlism, just accept it.

+ @ints.rotor(2 => -1).grep(-> ($a, $b) { $a < $b })

So, that should work. But at this point it’s just un uglier Ruby! We can do better!

To get there, let’s first see a different way we can build the list of pairs: we can slice the list and merge it with itself, an operation which most languages call zip.

Raku also calls it zip, but it has something cooler than a function: it has a zip metaoperator! §

A metaoperator is basically an operator which takes another operator as argument, and augments it’s behaviour.

The simplest I can think of is the reduce metaoperator. Suppose you want to get the product of all items in a list. You know the operation you want, which is *, and in Ruby, Python, Smalltalk etc.. you would call a method or function called reduce or inject passing a function to it.

In Raku, you apply the prefix reduce metaoperator ([]) to the binary product operator (*):

> [*] [1,2,3]
6

The Zip metaoperator is the same:, it takes a binary operator and returns a binary operator, which applies its argument to items from both sides zipping them up:

> [1, 2, 3] Z* [10, 20, 30]
(10 40 90)

See where we’re going? Yep, we can apply the first metaoperator to the result of the second!§

> [Z*] [1, 2, 3], [10, 20, 30]
(10 40 90)

So, if we want to know “is the item in the left list larger than the corresponding in the right list?” we can just choose the right operator

> [Z>] [1, 2, 3, 1000], [10, 20, 30, 0]
(False False False True)

This means we can apply it to the slices of the original input to obtain a list of Booleans. Then we just have to count them, and you can do this easily by knowing that in Raku you can add Booleans, and they will be treated as 1 and 0.

Do you know how to add up all items in a list? Yep, reduce metaoperator and binary plus (+)!

So the full solution is

[+] [Z<] @ints, @ints[1..*];

Or in even shorter form

[+] [Z<] @ints[*,1..*]

This is at a glance impossible to understand, but it’s actually pretty expressive, and I guess it should get easier with habit, we’ll see 🙂

See you another day, happy hacking!

PS

Problem 2 left as an exercise for the reader. You may want to invest time in learning how to use hyper operators, they are cool!