---
title: "How to debug a GenServer like Sherlock using IEx"
subtitle: "A Sherlock Holmes approved guide 🕵"
date: 2019-11-08
draft: false
author: Wilfried
tags: [dev, programming, elixir, production, debug]
---

# Elixir, the path to functional programming ⚗️

With my [**Rust** rediscovery](https://blog.papey.fr/post/03-bites-the-rust/)
a lot of _functional programming_ concepts became less obscure. In order to
go deeper into this paradigm I chose _Elixir_ a **powerful** general purpose, _functional
programming_ language, compiled and executed inside the _Erlang_ virtual
machine (**BEAM**).

But why _Elixir_, if _Erlang_ exists ? Because :

    Elixir, is Erlang with underpants - Athoune

Unlike _Erlang_, the _Elixir_ syntax is clear and elegant. The language comes
with everything you need, included : interactive shell, deps tooling, building tooling
and more…

Even if _José Valim_, the _Elixir_ creator was a core _Ruby_ dev, _Elixir_ is
completely different since this is a _functional programming_ language. No
mutation, no inheritance, no classes, only **pure** functions. Pray your only
true god, the pipe operator `|>` operator, used to compose functions.

Now, I know, the only thing you want is code and examples, so let's take a
quick look at some basic _Elixir_ stuff using the interactive shell **IEx**.

A simple addition :

{{< highlight exs >}}
iex(1)> 2 + 2
4
{{< /highlight >}}

A simple addition, inside a list

{{< highlight exs >}}
iex(1)> 2 + 2
iex(5)> list = [1,2,3,4][1, 2, 3, 4]
iex(6)> Enum.map(list, fn e -> e + 1 end)
[2, 3, 4, 5]
{{< /highlight >}}

List splitting (head and tail)

{{< highlight exs >}}
iex(7)> [head | tail] = list
[1, 2, 3, 4]
iex(8)> head
1
iex(9)> tail
[2, 3, 4]
{{< /highlight >}}

Another important aspect of Elixir is _pattern matching_ used to match
values, data structures and much more, let's try it :

{{< highlight exs >}}
iex(11)> x = {:this, :is, :a, :test}
{:this, :is, :a, :test}
iex(12)> {a, b, c, d} = x
{:this, :is, :a, :test}
iex(13)> a
:this
iex(14)> b
:is
iex(15)> c
:a
iex(16)> d
:test
{{< /highlight >}}

Using _pattern matching_ you can assign values but also destructure data to
simplify interaction with it.

Feels the hype growing ? Nice !

As usual, I like a real project to experiment on something. In the _Elixir_
case, I started a Discord bot project called
[o2m](https://github.com/papey/o2m). The main idea of this bot is to send
alert messages on a specific channel when a new episode from a selected
podcast is available. Today, I was struggling with a bug on a production
instance of _o2m_. The last episode was not fetched correctly and the action
that write a _"There is a new episode_" message was not triggered correctly.
To debug and inspect the current state of the application I used IEx **in
production**.

# Base ingredient of a good Elixir : Mix 🧙

Combined with IEx, there is _Mix_. Shipped with _Elixir_, this a build tool
used for the following application related tasks :

- creating
- setting up needed deps
- testing
- compiling

The combo killer here is to start IEx inside the _Mix_ project, in order to have all
the dependencies imported inside the interactive shell

{{< highlight exs >}}
iex -S mix
{{< /highlight >}}

After that, there is a lot of useful commands like `recompile` to refresh and
recompile new code **while your code is running**. Yes, this is something
that _Elixir_ do by default, hot code reloading.

When your code is ready, you want to deploy it. With _Mix_ the standard way
is to used the `release` command

{{< highlight sh >}}
MIX_ENV=production mix release
{{< /highlight >}}

This will create a precompile and packaged unit, with runtime included. With
this, you do not have to install _Erlang_ or _Elixir_ on your production
server. Boom.

If you look carefully, there is an interesting message and the end of the command output :

{{< highlight sh >}}
Release created at \_build/prod/rel/o2m!

    # To start your system
    _build/prod/rel/o2m/bin/o2m start

Once the release is running:

    # To connect to it remotely
    _build/prod/rel/o2m/bin/o2m remote

    # To stop it gracefully (you may also send SIGINT/SIGTERM)
    _build/prod/rel/o2m/bin/o2m stop

To list all commands:

    _build/prod/rel/o2m/bin/o2m

{{< /highlight >}}

Particularly,

    To connect to it remotely

This means I can have access to an interactive shell on production while my
code is running, hooray 🎆

# Sherlocking a GenServer 🔎

Did you read the title ? We are talking about _GenServer_ here ! So _what the fuck_ is this ?

Taken from the [documentation](https://hexdocs.pm/elixir/GenServer.html) :

    A GenServer is a process like any other Elixir process and it can be used
    to keep state, execute code asynchronously and so on. The advantage of using
    a generic server process (GenServer) implemented using this module is that it
    will have a standard set of interface functions and include functionality for
    tracing and error reporting. It will also fit into a supervision tree.

In _Elixir_, this is the default and common module used to implement client-server behaviors.

Here is the example from the documentation

{{< highlight elixir >}}
defmodule Stack do
use GenServer

    # Callbacks

    @impl true
    def init(stack) do
    {:ok, stack}
    end

    @impl true
    def handle_call(:pop, \_from, [head | tail]) do
    {:reply, head, tail}
    end

    @impl true
    def handle_cast({:push, element}, state) do
    {:noreply, [element | state]}
    end

end
{{< / highlight >}}

Basically, this is a data structure with a state responding to triggers using
handlers that update or retrieve the current state

To launch and interact with the server, in IEx :

{{< highlight exs >}}

iex(41)> {:ok, pid} = GenServer.start_link(Stack, [:hello])

iex(42)> GenServer.call(pid, :pop)
:hello

iex(43)> GenServer.cast(pid, {:push, :world})
:ok

iex(44)> GenServer.call(pid, :pop)
:world
{{< /highlight >}}

What if, I want to access the current state, of the GenServer ?

In _Erlang_, there is the `sys` module with the `get_state/2`
[function](http://erlang.org/doc/man/sys.html#get_state-2) available

But, this is _Erlang_, and we use _Elixir_ ! We're doomed ? No ! Because
everything available in _Erlang_ is available inside _Elixir_ using the `:`
operator.

{{< highlight erl >}}
iex(18)> :io.format("Hello World~n")
Hello World
:ok
{{< /highlight >}}

So, with our `sys` example :

{{< highlight exs >}}
iex(19)> :sys.get_state(pid)
[:!, :world]
{{< /highlight >}}

Nice ! Now we can get state of a GenServer using the PID !

Now, let's move from local machine to prod. In this case, processus are
handled by a **Supervisor** but first things first, we want a IEx shell
inside our running app. In order to illustrate this last part, I will be
using one of my prod instances of _o2m_

{{< highlight sh >}}
o2m@16557a733b59:/opt/o2m\$ ./prod/rel/o2m/bin/o2m remote
Erlang/OTP 22 [erts-10.5.3][source] [64-bit][smp:2:2] [ds:2:2:10][async-threads:1] [hipe]

Interactive Elixir (1.9.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(o2m@16557a733b59)1>
{{< /highlight >}}

Ok but, what is the _pid_ of the _GenServer_ I want to inspect ? I don't know !
Here the first solution could be outputting _pid_ to _stdout_ but, imagine that
our application kills and restart GenServer, on demand or our application is
flooding _stdout_ because of some errors, this is not a viable solution.

First, we could list all processus,

{{< highlight exs >}}
iex(o2m@16557a733b59)3> Supervisor.which_children(O2M.Supervisor)
[
{O2M, #PID<0.2368.0>, :worker, [O2M]},
{"jobs-https://feed.ausha.co/bj5li17QPONy", #PID<0.2365.0>, :worker, [Jobs]},
{"jobs-https://feed.ausha.co/yJeEUGlLVq0o", #PID<0.2362.0>, :worker, [Jobs]},
{"jobs-https://anchor.fm/s/b3b7468/podcast/rss", #PID<0.2358.0>, :worker,
[Jobs]}
]
{{< /highlight >}}

Where `O2M.Supervisor` is the dedicated _Supervisor_ of my application

Now I can identify the GenServer I want to inspect, let's take the
`jobs-https://anchor.fm/s/b3b7468/podcast/rss` one, with associated pid
`0.2358.0` (the last element of the list, here), here comes the fun

{{< highlight exs >}}
iex(o2m@16557a733b59)13> {_, pid, _, \_} = Supervisor.which_children(O2M.Supervisor) |> List.last
{"jobs-https://anchor.fm/s/b3b7468/podcast/rss", #PID<0.2358.0>, :worker,
[Jobs]}
iex(o2m@16557a733b59)14> pid
#PID<0.2358.0>
{{< /highlight >}}

Then, I can use the pid value to get _GenServer_ state and inspect it to see
if everything is ok :

{{< highlight exs >}}
iex(o2m@16557a733b59)15> :sys.get_state(pid)
{"https://anchor.fm/s/b3b7468/podcast/rss",
%{
date: "Mon, 04 Nov 2019 09:00:00 GMT",
show: "Harry Cover, le podcast des meilleures reprises",
title: "Yes we can work it out !",
url: "https://anchor.fm/leotot8/episodes/Yes-we-can-work-it-out-e8n4d3"
}}
{{< /highlight >}}

How ! Impressive ! It was a little bit Sherlock Holmes oriented debugging but
that was fun. Keep in mind that this is a really simple operation here, from
IEx everything is possible from starting new GenServers to modify state of a
specific one and much more.

I said it at the beginning, **Elixir is powerful**.