My Experience Migrating a Rails API to Crystal and AWS Lambda – Part II

Last Updated on May 29, 2023

Hey, this is the second part of my post, giving an overview of how I migrated a Rails API to Crystal with hosting on AWS Lambda.

In the first part, I have given an introduction about how our Rails API was designed and the technologies that we used. Now, I will explain why we chose to use Crystal, the different solutions we stumbled across, and some technologies we used to finish our microservice.

Some of you should be asking yourselves now: What is Crystal, and why use it?

This is what I am going to explain now!

The Crystal Lang

Crystal is a programing language with syntax similar to Ruby, statically type-checked, and most importantly: it is a compiled language!

class Code
  def initialize(@name: String)
  end

  def run
    puts "#{@name} is running!"
  end
end

c = Code.new("hello_world.cr")
c.run # => "hello_world.cr is running!"

Their first appearance was nine years ago, with the purpose of creating a language with Ruby’s efficiency for writing code and C’s efficiency for running code.

"We don’t want to write C code to make the code run faster."

Another cool thing about Crystal is that you rarely need to specify types, thanks to a type inference algorithm. You can check out more details and see how you can create a cool application with Crystal here.

So, what reasons led us to choose Crystal over Ruby?

Why Crystal?

Like I mentioned in the first post, we struggled to make our Rails API run faster but we did not succeed. Even with all the optimizations and extra threads (via Sidekiq,) the Genetic Algorithm (GA) stilled quite slow to give the results.

We started to think about alternative solutions. Then, the senior developer that was watching my colleague and I said:

"Let’s find alternatives to Ruby."

To be honest, this scared me a little bit. We were running out of time, and we were supposed to learn a new language to finish our microservice? Yes! The good part: we did not have to learn all about this language because it is similar to Ruby in almost everything (at least syntactically)… I’m talking about Crystal!

Ok, the learning curve will be smaller, but how can Crystal help on improving our microservice’s speed? The answer is compiling code. Crystal is a compiled language, and this makes it very fast performing heavy calculations.

This table is a benchmark of a matrice allocation and calculation. It is clear how Crystal overcomes Ruby on heavy computations, and our GA needed this power to run faster. You can find more details about the benchmark here.

Unfortunately, I don’t have a comprehensive benchmark of Crystal and Ruby running our GA, but the time taken was around 6 and 22 seconds, respectively. It was a considerable improvement, and we decided to link our almost finished Rails API to Crystal in order to build a hybrid solution with the best of the two worlds.

The Hybrid Solution

Since our API was basically finished, we thought about preserving it to receive the requests and communicate with Google’s Distance Matrix API. After this, an asynchronous job would be enqueued to be processed on Crystal, all this through Sidekiq.

This solution was comprised of three steps.

The First Step

In the first step, the client makes a request to our Rails API with the data containing the locations. The data is a JSON with origin, destination, and a list of places, each with latitude and longitude.

{
  "origin": 0,
  "destination": 5,
  "waypoints": [
    {"id": 0, "lat": -29.83635, "lng": -51.17556},
    {"id": 1, "lat": -30.83635, "lng": -52.17556},
    {"id": 2, "lat": -31.83635, "lng": -53.17556},
    {"id": 3, "lat": -32.83635, "lng": -54.17556},
    {"id": 4, "lat": -33.83635, "lng": -55.17556},
    {"id": 0, "lat": -34.83635, "lng": -56.17556}
  ]
}

After this, Rails returns a Job ID (JID) to be used thereafter to consult the GA result.

After receiving the JSON request and enqueueing the job, Rails returns the JID to the client, as I said before. Later on, the client would be able to make requests polling the results.

The Second Step

And where is Crystal? Right here!

The big difference is that our Sidekiq Worker was written using Crystal, not Ruby! When a worker is ready, it processes the job and returns a result: a JSON with the route distance and the order of the places provided by our GA. Note that the job is enqueued by Sidekiq Client on Rails, but processed by Sidekiq Server on Crystal.

When the worker finishes running the GA, it saves the result wherever it wants (I’m using Postgres here just as an example) and tells Sidekiq Client that the job is over.

If you want to learn more about Sidekiq’s Crystal implementation, check out the repository here.

The Third (and last!) Step

And finally, the final step.

When the result of our GA is ready and saved to a database, the client can make a request to a different endpoint with a JID parameter to fetch the optimized route.

But, here is a fun fact: we never implemented this solution (even though the deadline was looming!).

After realizing that that would be unnecessary work, we abandoned this idea and started to think about a Crystal-only solution.

The Final Solution

We decided to forgo the hybrid solution and move forward with a Crystal-only solution.

One of the problems that we faced is that, unlike Rails, Crystal does not have a library to interact with the Distance Matrix API, which forced us to implement it from scratch in an ad-hoc fashion. However, the issue that was concerning us since the beginning kept looping over and over: we did not have time!

With regards to hosting, Heroku was our first thought, but for this we would need to build a full API written in Crystal with authentication support and all that stuff.

Then, all of a sudden, a solution for the API stuff dawned on us: AWS Lambda!

AWS Lambda

"AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of Amazon Web Services. It is a computing service that runs code in response to events and automatically manages the computing resources required by that code."

With this service, we provide the code we want to run and pay only for the computation time taken, without the need for provisioning or managing servers.

Also, AWS likes security layers. Authentication and Authorization would not be a problem since everything we use in AWS needs specific clearances. Just as an example, I think that I needed to ask three or four times for more permissions to create the Lambda function and to execute it on the client’s AWS account, thus security is not a problem.

I will not give more details about Lambda, but you can check how cool this service is here.

With this in mind, we just needed to finish the code, create the tests, and deploy to Lambda. Now, I will talk about two tools we used to test our microservice: Spectator and VCR.cr

Spectator

In the Ruby world, we have useful libraries called "gems." With Crystal it is not different, but instead of gems, we have "shards." Shards are a collection of Crystal libraries, frameworks, etc.

One of these shards is called Spectator.

"Spectator is a fully-featured spec-based test framework for Crystal. It mimics features from RSpec."

The cool thing about this shard is that we increased a lot of our test creation speed because the syntax is pretty similar to RSpec, the most used testing gem for Ruby.

Spectator.describe String do
  subject { "hey" }

  describe "#==" do
    context "with the same value" do
      let(value) { subject.dup }

      it "is true" do
        is_expected.to eq(value)
      end
    end

    context "with a different value" do
      let(value) { "ho" }

      it "is false" do
        is_expected.to_not eq(value)
      end
    end
  end
end

OK, we wrote the specs, but we had one last problem to solve: we needed to test the Distance Matrix API on our side. It would not be cool to make countless requests every time we ran our tests. That would entail more costs to our client’s Google account, and we didn’t want that to happen.

How can we save the request made in one test and use the same response every time? The answer is cassettes.

VCR.cr

VCR.cr is another cool shard we used due to our previous experience with Ruby on Rails.

Like a video cassette recorder, we can record the HTTP requests present in our tests and use them again on future test runs, thus speeding up our test suite and making it more deterministic.

require "vcr"
require "http/client"

load_cassete("casette-one") do
  response = HTTP::Client.get("https://codeminer42.com")
end

We reduced our test suit’s runtime from minutes to seconds with VCR.cr.

There are several implementations of this shard in other programming languages like Javascript, Python, Go, Clojure, etc.

You can check more details about VCR.cr here.

Crambda

Last but not least, Crambda is a shard that allows you to create a custom runtime for Lambda.

You just need to provide a JSON parameter to a handler function, and after processing, another JSON with the result should be returned.

require "json"
require "crambda"

def handler(event : JSON::Any, context : Crambda::Context)
  pp context
  JSON.parse("[1, 2]")
end

Crambda.run_handler(->handler(JSON::Any, Crambda::Context))

You can check out more details about Crambda here. Their documentation also gives information about how to compile correctly for AWS. I confess that I had much trouble during this step, and their documentation helped me out a lot.

The End

Here is our final structure! As you can see, migrating all the code to Crystal and using Lambda to be our "API" allowed our flow chart to be much more straightforward and easier to understand what, where and when everything is happening.

This also allowed us to build a synchronous solution without background jobs and a pretty good time of around 8 seconds to get a response.

Conclusion

Crystal is a powerful language. Following its description, it has the best of two worlds in one language: easy to write and fast to execute. Even with less visibility than Ruby, Crystal already has a good community and excellent libraries and tools that help new users (or even experienced ones) a lot when they set out to learn the language.

It was my first time using Crystal, but I definitely would be more than happy in using it again on another project.

We want to work with you. Check out our "What We Do" section!