Integrating Wise and Google Sheets

I am a long time user and fan of Wise, since it was called Transferwise. I even used it to transfer money from Italy to Hungary (€->HUF) when I got married, so I’m emotionally attached to it.

It’s cheap, reliable, the app works fine and the site is simple and fast. <plug> Go sign up and earn us both a reward. </plug>

Anyway, one my few gripes with it is that I keep track of my finances in Google Sheets, and Wise does not have any native integration with it.

But it does support a generic form of webhooks. And what do you know, Google Sheets support webhooks! Kinda!

Setting up a sheet to receive POST requests

It’s somewhat insane that you can make a spreadsheet the database for a web app, but it works.

First, create a new sheet. You don’t need to do this, but I feel it’s saner to have a sheet which is just a dump of events, and keep your “important” stuff in a different one.

Now go to Extensions -> Apps Script and it will start up the scripts mini-IDE. Usually you use this to add macros or some such, but you can build a web app in there!

The code you need is something like this, put it in code.gs

// yes, this code sucks, I don't care.

function doGet(e) {
  var params = JSON.stringify(e);
  addRecord([[params]]);
  // you need this or you will get an error
  return HtmlService.createHtmlOutput("ok")
}

function doPost(e) {
  // make both POST and GET work the same
  return doGet(e);
}


function addRecord(values) {
  // use this for debugging
  console.log("values", values)
  // sheet name and range where you want to put your data
  range = 'Sheet1!A:A'
  // the id you see in the browser URL bar
  const spreadsheetId = "your spreadsheet id"

  try {
    let valueRange = Sheets.newRowData();
    valueRange.values = values;

    let appendRequest = Sheets.newAppendCellsRequest();
    appendRequest.sheetId = spreadsheetId;
    appendRequest.rows = [valueRange];

    const result = Sheets.Spreadsheets.Values.append(
      valueRange, 
      spreadsheetId,
      range, {valueInputOption: "RAW"});
    console.log('Result', result);
  } catch (err) {
    console.log('Failed with error %s', err.message);
  }
}

Beyond the oddity of the Google APIs, this is basically

  • listen to a POST request
  • add a new entry as JSON to a sheet

Once you have the code, hit the “Deploy” button on the top right, and choose “new deployment”.

In the modal that opens up, use the gear button to choose the kind of deployment, which should be “Web app”

Fill in the details:

  • set “Execute as” to your account, which will allow the app to write to your sheet
  • set “Who has access” to “Anyone“, so that Google will not require authentication. The paranoid person in me does not love this, but I don’t know of a way to secure this better. The URL is pretty unguessable tho.

At this point Google will deploy your web app and provide you with a URL, something like

https://script.google.com/macros/s/ABUNCHofGibberishWithL3tt3rsAndNUMB3rZ_LOLWATHAVEY0u533nThis-there4r3DaSh35t00/exec

You can even access this via a browser, and you should see the profound message that many developers have used to prove that God’s in his Heaven, all is right in the world: ok

You can also test a POST request via curl

curl -d'{"wat":"wat"}' https://script.google.com/macros/s/ABUNCHofGibberish...

If all is working correctly, you should be able to see some HTML response.

Now, if you go to the Sheet, you’ll find that there is nothing there! I tricked you, but it’s so you can learn. If you want to use any of the myriads of Google services, you need to first include it in the project.

If you go back to the Apps Script studio and check out the Executions entry in the sidebar, you will see the logs of your request, and indeed, you should have an error like

Failed with error Sheets is not defined

Notice that checking this logs is your main form of debugging, get used to logging a ton of stuff.

To fix this error, go back to the Editor, click on Services and find Google Sheets, select the V4 API, and use “Sheets” as the name.

Now you can deploy this again (create a new Deployment, and notice you get a new URL). This time, you should get an authorization screen, and since your app is not approved, you need to navigate it it so it lets you past the OAuth screen.

Now you can try to access this again via browser and curl (remember to use the new URL!) and finally you should see some records in your sheet!

Configuring Wise

In the Wise web app go to Settings -> Developer Tools -> Webhooks, and create a new one. Pick the events you want according to the documentation, put the URL of your app in there and hit the “Test webhook” button.

If all went well, you should see a new event in your sheet.

Notice that Wise does not currently allow you to edit a Webhook, so if you want to update it you need to delete it and create it again.

Feel free to send them feedback 😉

Conclusion

I had no idea that Google allows you to use Sheets as the database for a web app, and I’m honestly impressed. The debugging facilities are not that obvious, and when you get an error it’s pretty hard to understand what went wrong, but it works.

Wise webhooks are not that obvious either, and it would be nice to be able to send test requests with various payloads, rather than wait one month to see if the deposit data you’re getting matches what you expect, but it works and it’s still better than many banks.

If you use this, or something better, or have suggestions for improvements let me know in the comments. Happy hacking!

how to fix “something wrong with LDFLAGS” when install Ruby on macOS

From time to time I have to reinstall something (often Ruby) and in the meantime my system has somewhat deviated from the last time and things don’t work just as well.

FWIW, I’m on an M1 MacBook Pro, and I use macports.

This time the issue showed up when doing rvm install 3.2.4

Inspecting configure.log I found this message

If this happens to you you can track it down by going to the directory containing this file and looking at a different one, config.log, which contains both the command line and the test C program.

Common issues here are having something messed up in LDFLAGS or LIBRARY_PATH or the directory may not exist, but in my case the problem was

for some reason I had a broken libunwind, probably leftover from some experiment or migration. The test binary actually builds and runs fine, but for some reason the configure script is not happy with the ld warning, although it seemed fine for other conftests

I solved this by doing

After which my build proceeded just fine. I hope this helps someone in the future.

Tinker Tailor Engineer Spy

When I was in University I had a few good teachers, and I remember fondly Professor Lorenzo Farina.

Once, he§ came to class and ranted a bit about some job advertisement he had read: the job was for a software engineer, and mentioned they were looking for someone with a smanettone mindset.

Smanettone is an Italian term which in context meant something like “hacker” (in the Jargon File sense) or “tinkerer“. Back then, it was a bit of a shibboleth for people who were into cyber culture, *nix, FLOSS, security and so on. Those of us in the audience who were into that felt immediately positive about the company who had used this.§

Our teacher didn’t know that, so he interpreted it in a negative sense: who the heck are these people looking for? Someone who’ll turn knobs and flip switches without any sort of understanding of what’s going on? That is the opposite of what the engineer mindset should be!

The Tinkerer Mindset

Since the teacher welcomed feedback, we approached him to change his mind at the end of the lesson: that’s not what this is about! Being a a smanettone is about the joy of playing with something, push it beyond what it is designed for, and actually trying to understand it better. It’s about having fun while you build something because of your knowledge, not for lack of it!

There is a definition in the Treccani dictionary now, which roughly translates to

In computer jargon, someone who’s passionate and has fun experimenting, creating and changing the contents of their computer or of the software installed on it.

Vocabolario on line Treccani

It’s an ok definition! It’s not the one I’d write, but it’s close to what I think of as the tinkerer mindset.

(In praise of my teacher, he updated his world view, and I seem to remember he started using the term himself later on.)

The Engineer Mindset

So, I liked to think of myself as a smanettone. But I also like to think of myself as an engineer. I don’t think the two things are at odds.

I think the engineer mindset is about having a method. It does not actually matter what method exactly, but you should act with a plan.

Which is another factor: your actions should be intentional, you do X because you expect Y, not because you have no idea. When you get (or don’t get) a result, you try to understand why.

And you should be able to work logically. Logic is not all one needs in life or on a job, but it is a defining part of an engineering approach to problem solving and solution design.

The engineering mindset and the tinkerer mindset may overlap, but are separated and independent and not mutually exclusive.

I hate ChatGPT

ChatGPT and its ilk are incredible things. If someone had told me a few years back that we’d be able to have quasi-conversations with a bot I’d not believe it§.

But I hate the feeling of working with it, and I despise the concept of prompt engineering.

Prompt engineering is, to put it bluntly, not engineering. It is, at best, prompt tinkering, as it shares exploratory and fiddling aspects.

But it’s actually worse than that, because people do not have actual insights on what is happening, nor do they gain them by fiddling.

You hear stuff like “you should talk directly to the LLM” or “you should talk kindly and say please” or “you should ask to explain itself, and sometimes it works better“. In the OpenAI forums you’ll see people telling each other “oh yeah that works terribly with GPT3.5 try GPT4” but also the opposite.

There is actual engineering going on on top of LLMs. Stuff like RAG or HyDE or ReAct is cool and smart, and you can certainly build things of value on top of these things.

But a large part of it is prompt fiddling. You can’t even know how different LLMs (or versions of the same LLM) will react until you try them out, and when you do, you only know if something worked for whatever input you used, but you have no knowledge of how it will generalize. Sometimes a single different word may cause drastically different outcomes, for no discernible reasons.

You don’t even know if the same LLM will continue to work the same over time! They may be getting dumber while you use them!

Prompt fiddling can be fun, and may work at times, but it’s also deeply frustrating and inefficient.

Shamans and Medics

I once listened to a mythology class, which explained that in many mythologies people or animals are born out of a deity’s armpit.§

The explanation given was that, well, primitive people did not really understand procreation. An armpit is angular, hairy and humid. So, not that much different from female genitalia, right? If one thing gives birth, why not the other?

People did not understand things, they told each other vague explanations and did things that seemed to make sense to them.

There’s a beautiful story of an Arab doctor called to help a Frankish knight. The knight had an abscess to a leg, and the Arab doctor, armed with Greek medicinal theory, wanted to administer him some balm.

But a Frankish doctor came and suggested to chop his leg, “better to live with one leg than die with both!”. Of course the knight died.

Those Frankish idiots! They should have stuck with real science and a proper doctor: it would have been enough to rebalance his humors, and the knight would have been just fine. Or not.

The Frankish medics were not idiots, and they are more or less the reason surgery became the real thing in Europe. And the Arab doctors were not idiots either, they had centuries old theories, and sometimes their remedies worked, too.

But they didn’t actually have a proper understanding, and no way of gaining it, they were limited to vague explanations and things that sometimes worked, and sometimes killed people.

And I have the feeling this is where we’re at right now with LLM usage.

I can’t wait for the modern era to arrive.

Solving Advent of Code Day 21 using Z3 and Ruby

This year (2022) I decided to finally try to use Z3 to solve an Advent Of Code challenge, Day 21, to be precise.

Z3 is a theorem prover/SAT/SMT solver, which means “it will solve problems for you“. This is what computers are supposed to do, and I loved this stuff since I did an Operational Research exam in University.

Z3 can solve a variety of problems, but for this problem we just need to understand how it solves simple equations.

How to do algebra with Z3/Ruby

The Ruby bindings for Z3 are not as popular as, say, the python ones, but they work just fine. Install Z3 using your favorite tool (mine is MacPorts), and then the gem

$ port install z3
... compiling stuff
$ gem install z3
... compiling more stuff
$ ruby -r z3 -e 'p Z3.class'
Module

You can start coding now.

To solve an equation with Z3 you define a model composed of variables, values, and constraints, and then ask the solver to fill in the blanks.

The simplest thing you can do is a basic math expression

require 'z3'
solver = Z3::Solver.new

# the argument is the name of the variable in the model
a = Z3.Int('a')
b = Z3.Int('b')
x = Z3.Int('x')

solver.assert a == 1
solver.assert b == 3

solver.assert x == a + b

if solver.satisfiable?
  p solver.model
else
  puts "I can't solve that, Dave"
end

running this should give you

$ ruby additionz3.rb
Z3::Model<a=1, b=3, x=4>

Behold! You can do basic math!

The interesting bit is that Z3.Int is a Z3::IntExpr object, and when you mix it with numbers or other expressions you get back more of the same, you can explore this in irb

>> require 'z3'
=> true
>> a = Z3.Int('a')
=> Int<a>
>> a.class
=> Z3::IntExpr
>> b = a+1
=> Int<a + 1>
>> b.class
=> Z3::IntExpr

You can of course use other operators, such as relational operators. This is how you solve the problem of finding a number between two others

require 'z3'
solver = Z3::Solver.new

a = Z3.Int('a')
b = Z3.Int('b')
x = Z3.Int('x')

solver.assert a == 1
solver.assert b == 3

solver.assert x > a
solver.assert x < b

if solver.satisfiable?
  p solver.model # Z3::Model<a=1, b=3, x=2>
else
  puts "I can't do that Dave"
end

you can solve a system of equations in pretty much the same way. The classic puzzle SEND+MORE=MONEY where each letter is a different digit can be solved like this

require "z3"
solver = Z3::Solver.new

variables = "sendmoremoney".chars.uniq.each_with_object({}) do |digit, hash|
  # All variables are digits
  var = Z3.Int(digit)
  solver.assert var >= 0
  solver.assert var <= 9
  hash[digit] = var
end

# define the words in terms of the digits
send = variables["s"] * 1000 + variables["e"] * 100 + variables["n"] * 10 + variables["d"]
more = variables["m"] * 1000 + variables["o"] * 100 + variables["r"] * 10 + variables["e"]
money = variables["m"] * 10000 + variables["o"] * 1000 + variables["n"] * 100 + variables["e"] * 10 + variables["y"]

# the leftmost digit is never zero
solver.assert variables['s'] > 0
solver.assert variables['m'] > 0

# all digits are different
solver.assert Z3.Distinct(*variables.values)

# define the actual expression
solver.assert money == send + more

if solver.satisfiable?
  # get the values from the model, indexed by their name
  values = solver.model.to_h
  # map each letter to the variable and find each variable in the model
  "send + more = money".chars.map do |char|
    var = variables[char]
    print values[var] || char
  end
  puts
else
  p "Impossibru!"
end

should print

$ ruby smm.rb
9567 + 1085 = 10652

Solving Day 21 with Z3

Spoiler alert: this covers part 2 which is not visible unless you solve part 1.

The problem is: you have a set of monkeys shouting at each other, defined like this:

root: pppw = sjmn
dbpl: 5
cczh: sllz + lgvd
zczc: 2
ptdq: humn - dvpt
dvpt: 3
lfqf: 4
humn: ?
ljgn: 2
sjmn: drzm * dbpl
sllz: 4
pppw: cczh / lfqf
lgvd: ljgn * ptdq
drzm: hmdt - zczc
hmdt: 32

to the left of the : you have the monkey name, and to the right something they shout. A monkey shouts either a number, or the result of an operation based on numbers shouted by other monkeys, except for two of them: the monkey named root will check if the two numbers are equal, and humn represents you: you need to yell the right number so the equality check returns true.

My Clever Reader will have realized this is a simple algebraic problem, but the real input is large, to solve it you would have to think, determine the order of the operations, build a tree.. yeah I got bored already.

But executing instructions is what the Ruby interpreter does. And there is a clear mapping from the input a ruby+z3 instruction. So… let’s just transpile the input to a ruby program!

Each input line becomes an assert, we add a prologue, eval it, and then ask for a solution:

require 'z3'
input = <<~INPUT.lines
    root: pppw + sjmn
    dbpl: 5
    cczh: sllz + lgvd
    zczc: 2
    ptdq: humn - dvpt
    dvpt: 3
    lfqf: 4
    humn: 5
    ljgn: 2
    sjmn: drzm * dbpl
    sllz: 4
    pppw: cczh / lfqf
    lgvd: ljgn * ptdq
    drzm: hmdt - zczc
    hmdt: 32
  INPUT

# create a hash monkey-name -> variable 
# so we can look them up by name
env = Hash.new { |h, k| h[k] = Z3.Int(k) }

# convert the input to a ruby program 
prog = input.map do |l|
  # root checks equality
  l = l.sub(/root: (.*) \+ (.*)\n/, 'solver.assert env["\1"] == env["\2"]' + "\n")
  # this is our incognita, just get rid of it
  l = l.sub(/humn: (.*)\n/, "\n")
  # assert a variable as an exact number
  l = l.sub(/(\w+): (\d+)\n/, 'solver.assert(env["\1"] == \2)' + "\n")
  # assert a variable as the result of a binary operation
  l = l.sub(/(\w+): (\w+) (.) (\w+)\n/, 'solver.assert(env["\1"] == (env["\2"] \3 env["\4"]))' + "\n")
  l
end

solver = Z3::Solver.new
# ger the program in here
eval(prog.join)

# solve and show our solution
p solver.satisfiable?
# ask for a solution
p solver.model.to_h[env["humn"]].to_i

If you remove the input, this boils down to ~10 lines of code.

Is it ugly? Yes. Is it cheating? Probably. Is it running eval on random input I just downloaded from the internet? You bet it is.

But I had a lot of fun.

Ruby, RVM, MacPorts and OpenSSL on macOS (Monterey)

This is not so much a blog post but a log for the sake of future Me:

I have been using MacPorts on macOS for many years and am 99% happy with it: it allows installing binary packages and source ones with custom options, allows multiple versions to live side by side, has a lot of packages, can dump the whole list of installed stuff so you can re-install it elsewhere easily and many other things. It’s great.

I have also for a long time used RVM as my ruby version manager of choice. Again, it just works.

An issue with installing older rubies in recent macOS is that the system-provided OpenSSL is version 3, and ruby before 3.1 cannot build against it.

You can get OpenSSL1.0 or 1.1 via MacPorts, but there’s another problem: when you try to install anything recent via MacPorts you will end up with OpenSSL3, and at that point you will have problems building old ruby versions or old versions of gems like eventmachine or puma, expecting to have OpenSSL1.

This problem manifests itself when installing ruby 2.7 via RVM with a build/install log that ends like this

/Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- openssl (LoadError)
        from /Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /Users/riffraff/.rvm/src/ruby-2.7.6/lib/rubygems/specification.rb:2430:in `to_ruby'
        from ./tool/rbinstall.rb:846:in `block (2 levels) in install_default_gem'
        from ./tool/rbinstall.rb:279:in `open_for_install'
        from ./tool/rbinstall.rb:845:in `block in install_default_gem'
        from ./tool/rbinstall.rb:835:in `each'
        from ./tool/rbinstall.rb:835:in `install_default_gem'
        from ./tool/rbinstall.rb:799:in `block in <main>'
        from ./tool/rbinstall.rb:950:in `block in <main>'
        from ./tool/rbinstall.rb:947:in `each'
        from ./tool/rbinstall.rb:947:in `<main>'

the ruby binary is actually built correctly but the openssl extension failed to build, which you can tell by looking back in the build history

*** Following extensions are not compiled:
openssl:
         Could not be configured. It will not be installed.
         Check ext/openssl/mkmf.log for more details.

If you look into .rvm/src/ruby-2.7.6/ext/openssl/mkmf.log you will find

/Users/riffraff/.rvm/src/ruby-2.7.6/ext/openssl/extconf.rb:111: OpenSSL >= 1.0.1, < 3.0.0 or LibreSSL >= 2.5.0 is required

So it seems to have ruby use OpenSSL1 you should uninstall OpenSSL3 and everything that depends on it.

FWIW, I think that works, but it’s not ideal.

But fear not! There’s a better solution! You just need to tell RVM (and ruby) to use look for headers and libraries in the right place.

To do this you need to both specify where pkgconfig is and which openssl directories to use.

Luckily, this is not difficult, follow this incantation

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig rvm reinstall 2.7.6 --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

You effectively have to tell ruby how to configure itself, and how to find OpenSSL1 at compile time and at link time.

In theory --with-openssl-dir should be enough, but AFAIU MacPorts puts stuff in separate directories which does not play well with ruby-build. Which is fair, as ruby-build does not explicitly support MacPorts, but rather homebrew.

Anyway, once this is solved you may still encounter issues when installing gems that depend on OpenSSL, as those will also need to be told where OpenSSL is.

For example,. this is the error I get installing puma 5.6.4

compiling puma_http11.c
linking shared-object puma/puma_http11.bundle
Undefined symbols for architecture arm64:
  "_SSL_get1_peer_certificate", referenced from:
      _engine_peercert in mini_ssl.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [puma_http11.bundle] Error 1

make failed, exit code 2

The equivalent error for eventmachine would be something like

linking shared-object rubyeventmachine.bundle
Undefined symbols for architecture arm64:
  "_SSL_get1_peer_certificate", referenced from:
      SslBox_t::GetPeerCert() in ssl.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [rubyeventmachine.bundle] Error 1

Luckily you the same technique works to install older gems that need OpenSSL1. Notice you need an extra -- to separate the arguments provided to the gem configuration script from the ones to the gem command.

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig gem install puma -v '5.6.4' -- --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

(also worth noting: puma 5.6.5 builds without problems, see this and this)

So, hopefully I will be able to forget old code that relies on OpenSSL1 at some point, but in case I need it again: hey future Me, this is how you workaround the issue!

A note on ruby 3.0

If you build ruby 3.0 with MacPorts it will not explode, but it will not build the OpenSSL extesion, in /Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/mkmf.log you will find the same error you’ve just seen

/Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/extconf.rb:113: OpenSSL >= 1.0.1, < 3.0.0 or LibreSSL >= 2.5.0 is required
	/Users/riffraff/.rvm/src/ruby-3.0.4/ext/openssl/extconf.rb:113:in `<top (required)>'
	./ext/extmk.rb:214:in `load'
	./ext/extmk.rb:214:in `block in extmake'
	/Users/riffraff/.rvm/src/ruby-3.0.4/lib/mkmf.rb:331:in `open'
	./ext/extmk.rb:210:in `extmake'
	./ext/extmk.rb:572:in `block in <main>'
	./ext/extmk.rb:568:in `each'
	./ext/extmk.rb:568:in `<main>'

and this in turn will mean you can’t build puma or eventmachine or other things that depend on the ruby openssl extension

$ gem install puma
ERROR:  While executing gem ... (Gem::Exception)
    OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

But you’re good, the same solution as before will allow you to install that version of ruby too

$ PKG_CONFIG_PATH=/opt/local/lib/openssl-1.1/pkgconfig rvm install 3.0 --with-openssl-lib=/opt/local/lib/openssl-1.1 --with-openssl-include=/opt/local/include/openssl-1.1

From ruby 3.1 onwards, ruby will build just fine with the system-provided OpenSSL3, so you will be able to avoid these shenanigans.

You can still build against MacPorts’ version of OpenSSL1 or OpenSSL3 by using the right paths.

Happy hacking future Me!

Building ruby 2.7 on macOS with MacPorts and OpenSSL3

Recently I got a new apple box (an M1 MacBook Pro, which is nice, if a bit bulky), and found myself in the need to re-establish my dev environment.

This meant re-compiling various old ruby versions which I need for some projects. The problem is old ruby releases before ruby 3 required an equally ancient version of OpenSSL

I have used RVM for many years and it has a nice integration with MacPorts, my favorite macOS package manager. Getting the ports I need on a new install is straightforward, you just get a list of installed packages from one machine and reinstall them on the other. And RVM knows how to get rubies when you encounter a .ruby-version file, so no issue there.

But there’s a catch: I am using OpenSSL3 for most of my stuff, but I need OpenSSL1.0 or 1.1 to build ruby 2.7.5 and older.

If you google this, you will find plenty of bug reports against RVM, ruby-install, puma and plenty of others, with varying suggestion to use --with-openssl-dir, or setting PKG_CONFIG_PATH, overriding LD_FLAGS, and other incantations.

These may work in some cases, but not all: it seems an underlying problem is that if you have multiple versions of OpenSSL the ruby configure script may end up overriding the openssl-dir setting, and still end up linking against the incorrect library.

Luckily, the solution is pretty straightforward if you ignore all those recommendations 🙂

Try this

  • Install OpenSSL3: sudo port install openssl3
  • Install Ruby 3: rvm install 3.1
  • Check that it works and it loads openssl fine, by running something like ruby -ropenssl -e 'p [RUBY_VERSION, OpenSSL::VERSION]'
  • Install OpenSSL1 sudo port install openssl1
  • Remove v3: sudo port uninstall openssl3 (keep things that depended on it if you want)
  • Install older rubies: rvm install 2.7.5
  • Check those work too
  • Reinstall openssl3

Everything should now work fine, because MacPorts has no issues with multiple library versions sitting next to each other, and each ruby is linked to the appropriate one. Packages that depend on V1 or V3 will also just be happy next to each other.

Notice, I did this once already almost a year ago, and I had totally forgotten about it.

So here’s this small post, in the hope it may help someone, or at least my future self.

Advent of Code 2021: Day 6

I’ve been a bit busy so I skipped doing the tasks in Raku for a couple days, but today’s seemed fun. As usual SPOILERS ahead.

So the problem is basically: you have certain amounts of fish which have a “spawn-timer”, when they reach it they get reset to 6, and create a new fish with a spawn timer set at 8. Given an initial list of fish, we want to know how many fish we have after some days.

You, Clever Reader will certainly have noticed that we have an exponential behaviour: at each step we generate more fish, which will generate more more fish next time, and more more more fish and so on§.

The abstract solution in my mind would be:

  • set the initial values in some container structure
  • call some step function the given amount of days
  • count the total entries in the container

This, more or less

sub solve(@ints, $days) { # ints is something like [1, 2, 3, 3, 1, 4..]
  <whatever>
  step(<whatever>) for ^$days;
  [+] <whatever>;
}

the only interesting thing here is the construct

<do-something> for ^$number

where ^$number is a short form to define a range from 0 to $number and the rest is a short form to do a loop over the right hand size of the for keyword.

Extended this might look like

for 0..$days -> $day { <do-something> }

which I find less entertaining§.

The question to answer next is: what should we use as container?

Raku obviously has a Hash class, and we could initialize it with a default and then iterate over the initial values to build.

> my @ints= [1, 3, 3, 3, 3, 2]
[1 3 3 3 3 2]
> my %counts is default(0);
{}
> %counts{$_}++ for @ints
Nil
> %counts
{1 => 1, 2 => 1, 3 => 4}

The only new bit we see is the is default annotation that builds a container with a default value. But it seemed weird I have to do this, surely Raku has some builtin way to count things, like Ruby’s Enumerable#tally ?

And of course it does. It has More Than One Way To Do It, of course. For example, it provides a Bag class which is an immutable container for counting items, and can be extracted from another sequence trivially

> my @ints = [1, 3, 3, 3, 3, 2]
[1 3 3 3 3 2]
> @ints.Bag
Bag(1 2 3(4))

And a corresponding BagHash which returns a mutable container for counting.

Being a map from Any to Int these also come with a nice default, so when we access a missing item we get 0

> my $bh = @ints.BagHash
BagHash(1 2 3(4))
> $bh{1}
1
> $bh{2}
1
> $bh{3}
4
> $bh{99}
0

Wait, you might say, why are you using a scalar for that object, instead of a hash? Shouldn’t it be %bh rather than $bh?

The answer is: I don’t really know§

> my %bh-as-hash = $bh
{1 => 1, 2 => 1, 3 => 4}
> %bh-as-hash{99}  # no default in the hash!
(Any)
> $bh{99}          # still default in the BagHash!
0
> $bh{99} = 3      # you can update the BagHash
3
> $bh{99}
3
> %bh-as-hash{99}  # but the hash is unaffected
(Any)

You can alternatively use the := binding operator, which performs an assignment without casting:

> my %bh-as-hash-casted := $bh  # notice the representation
BagHash(1 2 3(4) 99(3))
> %bh-as-hash-casted{100} = 100 # it's the same object!
100
> $bh{100}
100

So our full solve can be

sub solve(@ints, $days) {
  my $counts = @ints.BagHash;
  step($counts) for ^$days;
  [+] $counts.values;
}

Now, how does the step function look? It is pretty straightforward, but it has a tricky bit

sub step($counts) {
  my $old = $counts.clone;
  for 8...0 -> $idx {
    $counts{$idx} = $old{$idx + 1};
    if $idx == 0 {
      $counts{8} = $old{0};
      $counts{6} += $old{0};
    }
  }
}

Do you see the… oddity? Look at this:

> for 1..3 -> $i { say $i }
1
2
3
> for 1...3 -> $i { say $i }
1
2
3
> for 3..1 -> $i { say $i }
Nil
> for 3...1 -> $i { say $i }
3
2
1

you cannot iterate in reverse one of those! This is because the .. operator builds a Range while ... builds a Seq. A Range is list of consecutive numbers, with a beginning and an end, while a Seq is something that can be iterated over.

So there are no numbers in an inverse Range, and if you iterate over it you get zero iterations. In my very humble rubyist opinion, this is very confusing™ , but I guess real Raku people are used to it.

Back to our task: of course, the Astute Reader will have noticed we can do this more tersely by simply shifting the array and adding it to itself, and then brining back the extra items. This is a simpler solution, and it’s pretty nice, but we need to build a list rather than a BagHash:.

sub solve2(@ints, $days) {
  my @list is default(0);      # create a list with default value 0
  @list[$_]++ for @ints;       # add all the fish to it
  step2(@list) for ^$days;     # do the steps
  [+] @list;                   # add up the values in the list
}

I have a feeling the first two lines can be collapsed, but I don’t know how, if you have an idea please let me know in the comments.

Our step function becomes very small

sub step2(@counts) {
  @counts.rotate;
  @counts[6] += @counts[8];
}

And honestly, I feel this is as expressive as it can be.

See you next time!

Advent of Code 2021: Day 2

Hello again!

So, Day 2 is not a particularly interesting problem: basically we have a stream of commands, and we need to move a submarine correspondingly, then in the last step we need to compute the product of the depth and horizontal distance

For problem 2, we are going to have “up N”, “down N” and “forward N”, which will respectively affect the aim of the submarine and the position.

This would be pretty trivial to do with a loop and a case/switch statement (which in Raku is called given/when) but I think it’s a cool chance to try out Raku’s multimethods.

A class definition in Raku looks like this

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;
  method process($operation, $num) { ... }
}

The $. is a twigil which says “this is a scalar value and it’s private but has accessor methods“.§

Methods (and Routines in general) support multiple dispatch which can happen through class, traits, or any specific attribute of the parameter (see the Signature class). In our case, we want to dispatch through the operation name, so we can do this§

multi method process("forward", $num) {
  ...
}
multi method process("up", $num) {
  ...
}
multi method process("down", $num) {
  ...
}

The actual implementation would be: when we go “up” or “down, we want to alter the aim, and when we go “forward” we move by the given amount horizontally, and by aim * num vertically.

We could alter the values in the given clasinstance, but… it’s uglier! The declared instance variables are immutable, if we want them to be mutable we should have used the rw trait, like this

class Sub {
  has $.aim = 0 is rw;
  has $.depth = 0 is rw;
  has $.horizontal = 0 is rw;
}

Raku is pushing gently against us so we prefer immutability, and who am I to go against that?

EDIT: as noted by mscha in the comments, this is incorrect. The variables are mutable, we could always assign them from within the instance, referring to them with the $! twigil, e.g. $!aim += $num. The is rw only refers to generating setter methods so they’re accessible from outside. Sorry, I’m still learning.

So, each of our process() calls will instead produce a new object by adjusting the current one. Luckily, the language has a pretty nice functionality to help with that: the clone() method accepts an argument to tweak the cloned object, so our code would look like this

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;

  multi method process("forward", $num) {
    self.clone(horizontal => $.horizontal + $num, depth => $.depth + $num * $.aim)
  }
  multi method process("up", $num) {
    self.clone(aim => $.aim - $num)
  }
  multi method process("down", $num) {
    self.clone(aim => $.aim + $num)
  }
}

This looks pretty nice to me! Now we only need to plug this into processing the commands. But I thought, well, we decided to be immutable, right? So let’s do this with reduce.

Rather than use the reduce metaoperator like last time, we can use the actual reduce routine. It works similar to other similar function in Python/Ruby/JavaScript/whatever.

my @words = 'day2.txt'.IO.words; # read all words
my @pairs = @words.rotor(2); # pair them up
my $sub = Sub.new # get a sub
<something with $sub and @pairs>.reduce(<block>) # iterate on values

The problem, is that, from what I understand, we cannot pass an initial value (our $sub) as an argument. The way to handle that is through using a Slip.

A Slip allows us to treat a list “as if it was typed there”. For example we can use it to prepend items to a list

> my @list = 1,2,3
[1 2 3]
> my @list2 = 3, @list
[3 [1 2 3]]
> my @list2 = 3, |@list
[3 1 2 3]

Or to use a list as an argument to a multi-argument routine

> my @pair = 1,2
[1 2]
> sub f($a, $b) { say "a: $a, b $b" }
&f
> f @pair
Too few positionals passed; expected 2 arguments but got 1
  in sub f at <unknown file> line 1
  in block <unit> at <unknown file> line 1

> f |@pair
a: 1, b 2

This is akin to the concept of “splat” in ruby, tho it’s somewhat more formalized.

So, we can pass an initial value to reduce by simply prepending it to the commands array:

($sub, |@pairs).reduce(<something>)

something should be a routine that takes two arguments (the accumulator and the next element) and returns a new value for the accumulator. This means it will call process(), and we can once again use a Slip!

($sub, |@pairs).reduce({ $^a.process(|$^b) })

Notice the $^ sourcery: in Raku you can use the ^ twigil to access variables implicitly, they will just be assigned from the arguments in order of definition. This call is effectively equivalent to

($sub, |@pairs).reduce(-> $a, $b { $a.process($b[0], $b[1]) } )

Now we just have to add a method to compute the final value, which is depth * horizontal, and putting all together we get

class Sub {
  has $.aim = 0;
  has $.depth = 0;
  has $.horizontal = 0;

  multi method process("forward", $num) {
    self.clone(horizontal => $.horizontal + $num, depth => $.depth + $num * $.aim)
  }

  multi method process("up", $num) {
    self.clone(aim => $.aim - $num)
  }

  multi method process("down", $num) {
    self.clone(aim => $.aim + $num)
  }

  method value {
    $.horizontal * $.depth
  }

}

my @words = '2.txt'.IO.words;

say (Sub.new, |@words.rotor(2)).reduce({ $^a.process(|$^b) }).value

which I find somewhat nice. I have to say, I still don’t like twigils, and I think I’m missing some way to write objects more tersely, if you know Raku better than me please let me know in the comments.

See you next time, happy hacking!

Advent of Code 2021: Day 1

Oh oh oh! Finally it’s that time of the year again!

I love Advent of Code and I’ve played it with friends and colleagues for years. Usually, I try to solve it in Ruby, since I’m very familiar with it, and so the effort is only on task solving rather than on learning the language and tooling.

But this year, I thought I might do it both in Ruby and in Raku (The Language Formerly Known As Perl 6), by first solving it in Ruby, and then if I have time trying to solve it again in Raku.

So, here’s a quick view of a couple solutions to the problems in Day 1. Stick to the end for the cool solution past the initial line noise 🙂

SPOILERS AHEAD

The first problem is basically: given a list of values, count the number of times they increase

In Ruby 3, this could look like this

def solve1(array)
  array.                # [1, 2, 5, 4]
    .each_cons(2).      # [(1, 2), (2, 5), (5, 4)]
    .count { _1 < _2 }. # 2
end

AKA: get a sliding window of pairs from the initial array, then count the occurrences where the first value in the pair is less than the second value in the pair. Notice the fancy anonymous positional block arguments!

You can do more or less the same in Raku, but I’m not aware of a .count method that takes a block§, but well, we can still do this in three steps:

  • build the sliding window
  • select the items matching the condition
  • get the size of the filtered list

Raku does not have each_slice, instead it has a .rotor method which is a generalization of “iterate and group“, you can define in one go both the size of the group and the offset, i.e.

> my @ints = [1, 2, 3, 4, 5, 6, 7, 8, 9]
[1 2 3 4 5 6 7 8 9]
> @ints.rotor(2)  # group by 2 
((1 2) (3 4) (5 6) (7 8))
> @ints.rotor(2 => 1)  # group by 2, and move by one after 
((1 2) (4 5) (7 8))
> @ints.rotor(2 => -1)  # group by 2, and move back by one after 
((1 2) (2 3) (3 4) (4 5) (5 6) (6 7) (7 8) (8 9))

The actual signature of this method is even more flexible, but this should be enough for us.

To filter we can use .grep, which takes a block. We can do this javacript-like, passing a block with a pointed arrow.§

Then we get the size of the list, which is done with prefix '+' . This is an odd perlism, just accept it.

+ @ints.rotor(2 => -1).grep(-> ($a, $b) { $a < $b })

So, that should work. But at this point it’s just un uglier Ruby! We can do better!

To get there, let’s first see a different way we can build the list of pairs: we can slice the list and merge it with itself, an operation which most languages call zip.

Raku also calls it zip, but it has something cooler than a function: it has a zip metaoperator! §

A metaoperator is basically an operator which takes another operator as argument, and augments it’s behaviour.

The simplest I can think of is the reduce metaoperator. Suppose you want to get the product of all items in a list. You know the operation you want, which is *, and in Ruby, Python, Smalltalk etc.. you would call a method or function called reduce or inject passing a function to it.

In Raku, you apply the prefix reduce metaoperator ([]) to the binary product operator (*):

> [*] [1,2,3]
6

The Zip metaoperator is the same:, it takes a binary operator and returns a binary operator, which applies its argument to items from both sides zipping them up:

> [1, 2, 3] Z* [10, 20, 30]
(10 40 90)

See where we’re going? Yep, we can apply the first metaoperator to the result of the second!§

> [Z*] [1, 2, 3], [10, 20, 30]
(10 40 90)

So, if we want to know “is the item in the left list larger than the corresponding in the right list?” we can just choose the right operator

> [Z>] [1, 2, 3, 1000], [10, 20, 30, 0]
(False False False True)

This means we can apply it to the slices of the original input to obtain a list of Booleans. Then we just have to count them, and you can do this easily by knowing that in Raku you can add Booleans, and they will be treated as 1 and 0.

Do you know how to add up all items in a list? Yep, reduce metaoperator and binary plus (+)!

So the full solution is

[+] [Z<] @ints, @ints[1..*];

Or in even shorter form

[+] [Z<] @ints[*,1..*]

This is at a glance impossible to understand, but it’s actually pretty expressive, and I guess it should get easier with habit, we’ll see 🙂

See you another day, happy hacking!

PS

Problem 2 left as an exercise for the reader. You may want to invest time in learning how to use hyper operators, they are cool!