joeellis.la

Using CircleCI 2.0 with Elixir and Phoenix

Tue, 12 Sep 2017 00:00:00 -0500

I recently took advantage of CircleCI’s new 2.0 features in a Phoenix application and thought it was worth sharing here. CircleCI 2.0 boasts many new features but the most interesting ones to me were the native support for Docker images and their advanced caching features.

It turns out to be a great move - both of these features helped cut our build times by at least 50%. Using a Docker image meant CircleCI no longer needed to compile Elixir, Erlang, and Node for each job. And the advanced caching features went a step further by giving us control over how our build and deps directories were cached, saving us on the compilation time between jobs. It did take some research through their docs and forums to figure out how to create a complete, working CircleCI yaml file, so I wanted to write up what I did in case my example helps save time for other people.

To start, my example config assumes the following:

Elixir 1.4.2, Erlang 19.x, Node 7.x, and yarn compiled into a single Docker image
Phoenix 1.3.x app
PostgreSQL 9.6

Your stack may differ from the above, and that’s ok. The strategy outlined here should still work for the majority of Phoenix and Elixir projects. Just know there may be a few things you’ll have to change to make this config work for your project.

To start, here is what the full CircleCI yaml config file looks like:

# .circleci/config.yml

version: 2
jobs:
  build:
    docker:
      - image: joeellis/elixir-phoenix-node:1.0
        environment:
          - MIX_ENV=test
      - image: postgres:9.6.2-alpine
        environment:
          - POSTGRES_USER=my-database-user
          - POSTGRES_PASSWORD=my-database-password
          - POSTGRES_HOST=localhost
    working_directory: ~/app
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
            - v1-mix-cache-{{ .Branch }}
            - v1-mix-cache
      - restore_cache:
          keys:
            - v1-build-cache-{{ .Branch }}
            - v1-build-cache
      - run: mix do deps.get, compile
      - save_cache:
          key: v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
          paths: "deps"
      - save_cache:
          key: v1-mix-cache-{{ .Branch }}
          paths: "deps"
      - save_cache:
          key: v1-mix-cache
          paths: "deps"
      - save_cache:
          key: v1-build-cache-{{ .Branch }}
          paths: "_build"
      - save_cache:
          key: v1-build-cache
          paths: "_build"
      - restore_cache:
          keys:
            - v1-yarn-cache-{{ .Branch }}-{{ checksum "assets/yarn.lock" }}
            - v1-yarn-cache-{{ .Branch }}
            - v1-yarn-cache
      - run:
          working_directory: assets
          command: yarn install && yarn test
      - save_cache:
          key: v1-yarn-cache-{{ .Branch }}-{{ checksum "assets/yarn.lock" }}
          paths: assets/node_modules
      - save_cache:
          key: v1-yarn-cache-{{ .Branch }}
          paths: assets/node_modules
      - save_cache:
          key: v1-yarn-cache
          paths: assets/node_modules
      - run: mix ecto.create && mix ecto.migrate
      - run: mix phoenix.digest
      - run: mix test

That is a bunch of config, so let’s break it down piece by piece and see what’s going on:

# .circleci/config.yml

version: 2
jobs:
  build:
    docker:
      - image: joeellis/elixir-phoenix-node:1.0
        environment:
          - MIX_ENV=test
      - image: postgres:9.6.2-alpine
        environment:
          - POSTGRES_USER=my-database-user
          - POSTGRES_PASSWORD=my-database-password
          - POSTGRES_HOST=localhost
    working_directory: ~/app

First we tell CircleCI that we’d like our build to execute under the joeellis/elixir-phoenix-node:1.0 docker image. This is a simple docker image I built with Elixir 1.4.2, Erlang 19.x, node 7.x, and yarn already installed. Any Docker image will do though - CircleCI even offers pre-built Elixir images if you’d rather not create your own.

The config also downloads a second docker image, postgres:9.6.2-alpine to create a database container and with our app’s database credentials (see the official docker image docs for more supported options). Lastly, it sets a working directory folder called app in the CircleCI user’s home directory.

Next, the build checks out our git repo, and restores any caches that may already exist:

- checkout
- restore_cache:
    keys:
    - v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
    - v1-mix-cache-{{ .Branch }}
    - v1-mix-cache
- restore_cache:
    keys:
    - v1-build-cache-{{ .Branch }}
    - v1-build-cache
- run: mix do deps.get, compile

The restore_cache and cache keys here make look funny to you. What are they and where do they come from? In short, this is part of CircleCI’s new caching mechanism, and before you read the rest of this article, I highly recommend you read and understand their caching docs because you will need to understand it to create the best caching strategy for your own app. After reading that, read below about the save_cache steps first and we’ll cirle back to how this restore_cache stuff works in a minute.

steps:
  - save_cache:
      key: v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
      paths: "deps"
  - save_cache:
      key: v1-mix-cache-{{ .Branch }}
      paths: "deps"
  - save_cache:
      key: v1-mix-cache
      paths: "deps"

For our deps directory, we create three types of CircleCI caches:

A cache keyed against the branch name and the mix.lock checksum.
- This cache is used for most commits to a branch.
- It also uses a mix.lock checksum as part of its key. If new dependencies are added and the mix.lock file changes, this will make sure to cause a cache miss and force a recompilation of the new dependencies.
A cache keyed against just the branch name.
- CircleCI uses this cache as a fallback if the first cache can’t be found, usually when the mix.lock file has changed.
A cache with just a generic key of v1-mix-cache.
- This cache is used if CircleCI can’t find any of the above caches, like in the case of the first commit of a new branch. Also, if you are creating small commits often, then you’ll find this cache is very useful at saving on compilation times between branches.

For our build directory, you can see it’s very similar to the deps caching strategy:

- save_cache:
    key: v1-build-cache-{{ .Branch }}
    paths: "_build"
- save_cache:
    key: v1-build-cache
    paths: "_build"

One small difference is that we only use two types of caches as there is no ‘lockfile’ for the build directory to cache against. This makes sense though, as we want your project to always recompile itself with the new changes made in each commit.

After you understand how the save_cache works, then the restore_cache keys in the steps above make more sense:

- checkout
- restore_cache:
    keys:
    - v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
    - v1-mix-cache-{{ .Branch }}
    - v1-mix-cache
- restore_cache:
    keys:
    - v1-build-cache-{{ .Branch }}
    - v1-build-cache
- run: mix do deps.get, compile

All we are doing here is declaring which of our saved caches to check and which order should check them.

Next, we use the exact same caching strategy to install our frontend node_modules using yarn:

- restore_cache:
    keys:
    - v1-yarn-cache-{{ .Branch }}-{{ checksum "assets/yarn.lock" }}
    - v1-yarn-cache-{{ .Branch }}
    - v1-yarn-cache
- run:
    working_directory: assets
    command: yarn install && yarn test
- save_cache:
    key: v1-yarn-cache-{{ .Branch }}-{{ checksum "assets/yarn.lock" }}
    paths: assets/node_modules
- save_cache:
    key: v1-yarn-cache-{{ .Branch }}
    paths: assets/node_modules
- save_cache:
    key: v1-yarn-cache
    paths: assets/node_modules

If you are using npm, the setup is roughly the same - instead of keying against a yarn.lock checksum, you would key against package-lock.json. I just prefer yarn as it is very fast and deterministic. Also, the latest npm (5.3.0 at the time of this writing) was having some issues compiling correctly in production.

- run: mix ecto.create && mix ecto.migrate
- run: mix phoenix.digest
- run: mix test

This last step should look familiar to anyone who has run a Phoenix application before. Just create the database, create a digest for your assets if needed, and finally run mix test.

Hopefully, this rundown of CircleCI’s 2.0 features helps someone out there. The config file looks long, but as you can see, the bulk of it is just repeated use of the same caching pattern. Give it a try, and if you run into trouble, free to tweet at me or ping me on the elixir-lang Slack channel!

Debugging Mix Dependencies Locally

Sat, 08 Jul 2017 00:00:00 -0500

Have you ever needed to debug a dependency, perhaps to track down a bug or maybe just to see how it works? In scripted languages like Ruby or JS, you can usually add debug statements, like puts "test" or console.log("test"), to the dependency’s code and see the output the next time you run your app.

This does not quite work in an Elixir / Mix project as dependencies are compiled and usually compiled just once, when they are fetched. Fortunately for us, mix (Elixir’s build tool) gives us an option called path, to automatically recompile a dependency whenever it changes. Simply declare the path option with your dependency in your project’s mix.exs like so:

defp deps do
  [
    ...
    {:my_dep, '~> 1.0.0', path: 'deps/my_dep'}
    ...
  ]
end

This tells mix to look at the deps/my_dep folder for that dependency when compiling, and recompile it (and related project files) anytime a change is detected. Now you can open up the code at deps/my_dep, add your debug statements, and explore away!

Absolute & Relative Paths

Note that the path option isn’t limited to looking at the deps folder, and also accepts relative and absolute system paths, like so:

defp deps do
  [
    ...
    {:my_dep, '~> 1.0.0', path: '../path/to/my/local/my_dep'}
    ...
  ]
end

This is particularly useful if you are developing a new Hex package along side your project, or if you want to clone a package locally to track your changes and submit patches to the original author later.

Reset the Dependency

Once you’re finished debugging and want to restore the dependency to it’s normal state, simply clean it out and re-fetch it by running:

mix do deps.clean my_dep, deps.get, compile

And everything will be reset back to how it was in the beginning.

Testing Redux: Async Action Creators

Wed, 05 Jul 2017 00:00:00 -0500

One of the biggest (and often overlooked) advantages of Redux apps is how inherently testable they are. Redux apps are composed of mostly plain old JavaScript objects and functions, so testing rarely requires special tricks or methods.

And probably the most valuable tests you can write for a Redux app are action creator tests. Actions tend to collect most of the messy, complex code in a Redux app, and learning how to test them well can make future maintenance and refactoring easier for you and your team.

In this article, I’ll explain the pattern I use for testing Redux actions (asynchronous action creators in particular) and hopefully show that testing even complex actions can be pretty easy!

The Setup

Packages to install

For action testing, I use the following packages:

mocha - my JS testing framework of choice. If you don’t use mocha, however, keep reading - the strategies in this article are not mocha specific and can work with any JS testing framework.
nock - an HTTP mocking library. This is only used if your app is making HTTP requests (but at some point, your app probably will).
redux-mock-store - a simple, but effective Redux store mock.

Configuring nock

I usually create a base nock setup file in my app that I can import when needed. It looks something like this:

import nock from 'nock';

nock.disableNetConnect();

// Configure the base url to your endpoint here
export default nock("http://localhost/api/");

I recommend adding nock.disableNetConnect() as it will force disable all non-mocked HTTP requests during your test. This ensures your tests won’t accidentally hit a real API endpoint during a test run, a bad practice that is both dangerous and slow.

An Example Action Creator

Let’s start with a look at a real world example, this set of action creators for fetching users from an API endpoint:

const fetchUsersRequest = () => {
  return { type: Actions.FETCH_USERS_REQUEST }
}

const fetchUsersSuccess = (users) => {
  return {
    type: Actions.FETCH_USERS_SUCCESS,
    payload: users
  }
}

const fetchUsersFailure = (err) => {
  return {
    type: Actions.FETCH_USERS_FAILURE,
    payload: err
  }
}

export const fetchUsers = () => {
  return (dispatch) => {
    dispatch(fetchUsersRequest());

    return fetch("/users")
    .then(({ users }) => {
      dispatch(fetchUsersSuccess(users));
    })
    .catch((err) => {
      dispatch(fetchUsersFailure(err));
    })
  }
}

Here, fetchUsers is the main action creator we want to test. When executed, it will send an HTTP request to the API, and dispatch two out of these three actions:

fetchUsersRequest - an action used to represent a request in progress (handy if your UI needs to show some kind of loading state)
fetchUsersSuccess - an action that fires if the API call succeeds
fetchUsersFailure - an action that fires if the API call fails

We want our test to check that these action creators are all invoked in the correct order, with the correct data, for both successful and unsuccessful API calls. To do this, we’ll do the following:

Setup a mock response for our HTTP API endpoint. If the action makes a request to /users/123, then we should make sure it hits our nock server and have nock return a mock response that we can control and test against.
Setup an ordered array of the actions we expect our action creator to dispatch and send to the reducers.
Dispatch the action creator function, assert it received our mock data, and assert it called all of our expected actions in the correct order.

Testing The Happy Path

Given the strategy above, here is what testing the success path of fetchUsers might look like:

import assert from "assert";
import mockStore from "redux-mock-store";
import nock from "../path/to/nockSetup";
import Constants from "../path/to/Constants";
import { fetchUsers } from "../path/to/UserActions";

describe("UserActions", function() {
  beforeEach(function() {
    this.store = mockStore({});
    this.usersData = [ {id: 1}, {id: 2} ];
  });

  context("#fetchUser", function() {
    it("can fetch users from the api", function() {
      nock.get("/users").reply(200, this.usersData);

      const expectedActions = [
        {
          type: Constants.FETCH_USERS_REQUEST
        },
        {
          type: Constants.FETCH_USERS_SUCCESS,
          payload: this.usersData
        }
      ];

      return this.store
        .dispatch(fetchUsers())
        .then(() => {
          assert.deepEqual(this.store.getActions(), expectedActions)
        })
        .catch(err => {
          throw(err)
        });
    });
  });
});

Let’s break this test down into the important parts:

beforeEach(function() {
  this.store = mockStore({});
  this.usersData = [ {id: 1}, {id: 2} ];
});

Here, we setup both our mock store and some fake data that we’ll use later for our mock HTTP response.

context("#fetchUser", function() {
  it("can fetch users from the api", function() {
    nock.get("/users").reply(200, this.usersData);
    ...

Next, we use nock to create a mock HTTP route for our Redux app to hit. This will return back our mock user data (this.usersData) whenever a successful GET request is made to the /users route.

...
const expectedActions = [
  {
    type: Constants.FETCH_USERS_REQUEST
  },
  {
    type: Constants.FETCH_USERS_SUCCESS,
    payload: { users: this.usersData }
  }
];
...

Then we create an expectedActions array, an array of the actions we expect to be dispatched in the order we expect them to be dispatched in. The first object is the action created when fetchUsersRequest is invoked, and the second object is the action created when fetchUsersSuccess is invoked. Note: the expectedActions array here does not contain the action for fetchUsersFailure as we are only testing the success path right now.

...
return this.store
  .dispatch(fetchUsers())
  .then(() => {
    assert.deepEqual(this.store.getActions(), expectedActions)
  })
  .catch(err => {
    throw(err)
  });
...

Finally, we kick off the test by dispatching our fetchUsers action creator using our mock store’s dispatch function. redux-mock-store comes with a handy store.getActions function, so we can see which actions were called, and make sure it equals our expectedActions array.

Testing The Not So Happy Path

We should also test that our action creators handle errors appropriately. Here is what testing the failure path of fetchUsers might look like:

...

it("handle errors when fetching users from the api", function() {
  nock.get("/users").reply(500, this.usersData);

  const expectedActions = [
    {
      type: Constants.FETCH_USERS_REQUEST
    },
    {
      type: Constants.FETCH_USERS_FAILURE,
      payload: new Error()
    }
  ];

  return this.store
    .dispatch(fetchUsers())
    .then(() => {
      throw("This test should not have succeeded.")
    })
    .catch(err => {
      assert.deepEqual(this.store.getActions(), expectedActions)
    });
});

...

It has a very similar structure to the success test above. The main difference is our HTTP request mock now replies with a HTTP 500 status code (used for internal server errors) and our expectedActions array includes the results from the fetchUsersFailure action creator. Also, we run our assert in the catch portion of our promise (as we expect a failure here), and we throw an error in the success path in case our action creator incorrectly manages to succeed.

And that’s it! You just learned how to fully test an async action creator for a Redux app!

Conclusion

With this testing strategy, even as your action creators become more complex, the testing strategy itself stays the same. They still follow the same pattern of creating a mock HTTP response, creating a list of expected actions, and comparing both to the result of the action creator function call. I recommend giving this testing strategy a try, especially if you’ve never tested your Redux actions before - I think you’ll find them to be invaluable addition to your Redux app.

Redux Architecture Guidelines

Tue, 20 Jun 2017 00:00:00 -0500

I’ve written many Redux apps over the past couple of years, and it is by far my favorite JS framework. The only downside is, unlike other frameworks, Redux is far less opinionated in how to structure an app. I prefer this freedom, but it does make for a steeper learning curve, especially if you’re new to Redux. So I decided to write up some of the higher level thinking and structure I’ve picked up and often use when building a Redux app. Hopefully it comes in handy for someone out there.

State

Plan your state shape

In terms of saving time down the road, planning the structure of your state object upfront is most valuable thing you can do for your app. A poorly formed state object will make your app difficult to maintain and is avoidable with some planning. I run through this quick checklist when planning out state objects:

How will it store multiple resources from an API (users, accounts, items, etc)?
How will it handle loading states (showing loading spinners when fetching /updating data)?
How will it handle showing and clearing of UI success and error notifications?
Does it feel consistent and predictable? Could another team member easily work with it?
Is it easy to access data within it? Does it nest properties unnecessarily?
Is it serializable? Could it easily be stored away in localstorage or in a database?
Are there any properties you could pull from the URL instead of in the state?
Is there any duplicated data in here? If so, is that really needed?

There are many different ways to answer these questions - it depends on your app. But in my experience, having at least an answer for each will save you time in the long run.

Avoid nesting state objects

Some Redux apps have deeply nested state structures, i.e. shapes that look like this:

{
  foo: {
    bar: {
      baz: {
        qux: ...
      }
    }
  }
}

This often happens when we work with relational data as it feels natural to use nesting to represent those relationships. Unfortunately, nested data structures create complexity. At the component level, you’ll have to reach even deeper into the state to get certain information. And at the reducer level, merging new data into your state will become far more complex. On top of all of that, nested data can even cause performance issues with React / Redux itself.

Consider instead to flatten and normalize your state shape. In Redux land, the shallower the nesting, the easier it is is to fetch and update state data in your app. Normalized states help solve the problems listed above, and make your state much more flexible overall.

Storing only raw data in the state

It’s tempting to use Redux’s state as a vehicle to store any and all information you think you may need later. Yet, doing so will increase your app’s complexity in the form of state bloat and redundant properties. This, in turn, increases complexity in your actions, reducers, and tests. So what should and shouldn’t be stored?

In Redux apps, there are really two types of data. The first is raw data, data your app requires to run. User data fetched from an API is an example of raw data - without it, your app won’t have the information it needs to run. The second is derived data, or data created from other existing data. Using the firstName and lastName properties to display a user’s name as Jane Doe is an example of derived data.

I recommend persisting only raw data in your state. It helps reduces state bloat and makes it easier to reason about what data is important in your app. All other derived data should be created using functions that accept that raw data from the state return back the information you need.

Before adding something new to the state object, ask yourself this question, “Can I create this from data that already exists in the state?” If the answer is “yes”, then create that data with a function. If the answer is “no”, then you may have a good case to add this data to the state. You may be surprised over time how often the answer is “yes.”

Prefer Redux state over React state

React comes with its own system for managing state inside of components. In a Redux app, though, prefer to use Redux’s state for the majority of your app data and inter-component communication. It is overall much easier to reason about your app when there is one accepted way for components to set and access state, especially if you are working within a team.

Note that there are reasonable exceptions to this guideline. It can be beneficial for complex UI components to persist local properties using React component state, especially when those properties aren’t globally important to the app. When doing this, just try to keep that React state management localized to that component. Using two separate state systems too much, especially for inter-component communication, is likely to cause confusion for the developer after you.

Actions

Standardize action payloads

When working with a team, having a standard object shape for your actions is very helpful. Doing so reduces bikeshedding and creates maintainable and testable code. I highly recommend adopting some kind of standard with your team. I use the Flux Standard Action spec because it is straightforward and simple to understand. But whatever you use, make sure it’s consistent and easy to work with.

Ensure action creators are composable

Many example apps and tutorials I run across use simple action creator functions when teaching Redux concepts. This is great for illustrating a point, but real world apps are complex. It’s inevitable that you will need to compose higher-level complex actions, preferably from existing action creators you have already written.

Start a habit of making sure all your action creator functions are composable in some way. It’s a simple rule that really pays off when you need it. I personally wrap each action creator in a promise so they can be easily chained together using the then function.

Component Architecture

Containers & presentational components

The most useful concept I’ve come across for building stable and easily maintainable Redux apps is the container & presentational component paradigm as described by Dan Abramov in the official Redux documentation. I will not dive into it here as the docs already do a great job at explaining the concept with great examples. But understanding this paradigm may be one of the most useful things you can learn about in Redux land. It is very difficult to maintain and iterate on an app of even moderate complexity without it. Learn it well.

Use intermediary containers

While the container / presentational component paradigm works, it’s not always clear when containers should be introduced. I’ve seen (and written) apps with a single top-level container that fetches the whole world and then passes down everything to its component’s children and their children’s children. This results in props ‘passing through’ multiple components before they are ever even used. As your app grows, this becomes an annoying problem as even simple changes, like renaming props, involves changing many other non-related components. Definitely a code smell that something is not right.

Instead, create containers when you notice multiple props ‘passing through’ multiple components. There is no need to pass props from one end to the other when a container in the middle can access the state and create those props for you. Intermediary containers also have added benefits, such as encapsulating sections of your component tree making their children easier to maintain and test. Don’t be afraid to use them if the situation calls for it.

There are No Rules

All the guidelines I’ve listed are just patterns I’ve found worth repeating. However, do not consider any these points as the only way to do things. After all, one of biggest advantages of Redux is its free form structure, so know when you should ‘break’ the rules and try something new.

Connect to Running Elixir Applications with IEx Remote Shell

Sat, 10 Jun 2017 00:00:00 -0500

Note: --remsh has security issues you should be aware of when using it on your local machine. Please be sure to read the security section at the bottom of this article.

Recently I learned about a fun option built into IEx called --remsh or “remote shell”. It creates an IEx shell in the context of an Elixir node, allowing you to debug and reproduce issues inside a running application! Also note that despite the name, --remsh can connect IEx to either local or remote nodes / applications. Here’s an example of how it works:

# when running Elixir app on a server called example.com...
> iex --name "joe@iex_test" --cookie secret_cookie --remsh "node@example.com"

iex(node@example.com)1> Node.list
[:joe@iex_test]

--remsh works in combination with two other options, --name and --cookie. The first is required and creates a named IEx shell to identify yourself to other nodes in the cluster. The second is not required but usually needed - it’s the the security cookie needed to access the node (often found under your home directory at ~/.erlang.cookie).

You could also wrap this up in a simple bash script to make it even easier to use:

function reiex() {
 if [ $1 = "staging" ]; then
  iex --name "$(whoami)@reiex" --cookie secret_cookie --remsh "node@staging.example.com"
 elif [ $1 = "prod" ]; then
  iex --name "$(whoami)@reiex" --cookie secret_cookie --remsh "node@prod.example.com"
 else
  echo "Environment not found!"
 fi
}

Just run reiex <environment name> to connect to your staging or production servers.

Security Issues

--remsh has security implications you should be aware of before using it. Remember - Elixir / Erlang nodes have complete access to all other nodes in a given cluster. So when you connect from your laptop via --remsh, other nodes in that cluster can now access and make RPC calls on your local workstation. If you connect to a remote node that has been compromised from your local machine, your private files (even private SSH keys) would be up for grabs.

If you plan to do any serious work with remsh, I recommend reading Alex Weber’s writeup which goes in further detail about this issue. Alex also offers us a more secure alternative: create an SSH connection to your remote machine first and remsh locally to the node from there. This can at least prevent node RPC calls from being made against your workstation as well as establish a more secure connection between your machine and the node.

Happy --remsh-ing everyone!

Streaming Data with Ecto

Sun, 21 May 2017 00:00:00 -0500

Note: if you are not familiar with Elixir streams, you should read the documentation on streams first or this post may be hard to follow.

It’s a given that in your programming career, you will need to fetch a dataset and run some set of operations on it. With small datasets, you can often get away with fetching the records at once and doing what you need to do. But if you’re trying to fetch a large dataset, you will run into some common problems. It’s very slow, creates a blocking database connection, and most database libraries will load the dataset into memory which can slow down or kill your server.

Fortunately, Ecto has a fantastic tool for handling this situation, Repo.stream/2. Repo.stream/2 avoids fetching everything at once and instead fetches data in iterative cycles, performing operations on each record along the way.

In this example, we will work through a typical app feature: exporting database records as CSV files. It’s a useful feature and is a good example to show how versatile Elixir streams can be.

First, let’s look at some code:

defmodule UserExporter do
  @columns ~w( id email inserted_at updated_at )a

  def export(query) do
    path = "/tmp/users.csv"

    Repo.transaction fn ->
      query
      |> Repo.stream
      |> Stream.map(&parse_line/1)
      |> CSV.encode
      |> Enum.into(File.stream!(path, [:write, :utf8]))
    end
  end

  defp parse_line(user) do
    # order our data to match our column order
    Enum.map(@columns, &Map.get(user, &1))
  end
end

Here we have a module with one public function, export/1, that accepts an Ecto query and returns back a file stream.

At the top, we declare a @columns module attribute to define the user data to show in our file. We’ll need to refer to this @columns list later to reorder our data for the CSV file.

Next is the meat of the exporter. The first thing we do is open up a database transaction. This is a good idea when iterating over a large dataset and also a requirement when using Repo.stream/2 with either a MySQL or Postgres. All work using Repo.stream/2 needs to take place inside a transaction.

Having opened a transaction, we pass our query to Repo.stream/2 which creates the initial stream. Remember, this only creates the stream; streams are lazily evaluated, so we haven’t starting making any calls to the database yet.

We then pass the stream to a Stream.map function which will fetch and transform data from each user struct into a list of user data matching the order of our columns.

Using the wonderful CSV library, we then pass the stream to the CSV.encode/1 function to do all the hard work. CSV.encode/1 both accepts and return streams, so it fits right into our pipeline setup here.

Finally, a call to Enum.into runs the whole stream, and the actual database calls start. By default, Repo.stream/2 pulls down 500 records at a time, and each is parsed and written to a file at tmp/users.csv. Lastly, our function returns a file stream that points to our newly created file. We can use this new stream to do more file-based operations or start some other stream-based work, like uploading the file to S3, etc.

As a test, we can see the results of our CSV in an iex console:

iex> {:ok, file} = UserExporter.export(from u in User)
{:ok,
 %File.Stream{line_or_bytes: :line,
  modes: [:write, {:encoding, :utf8}, :binary], path: "/tmp/users.csv",
  raw: false}}
iex> File.read!(file.path)
#=> "1,joe@example.com,2017-04-27 21:57:42.972524,2017-05-20 18:48:16.235083\r\n2,jane@example.com,2017-04-27 18:36:32.053556,2017-04-27 18:36:32.065434\r\n3,jill@example.com,2017-04-27 18:37:43.503567,2017-04-27 18:37:43.503575\r\n"

Pretty neat, eh? Using streams, we created a simple and ordered pipeline to efficiently fetch data, transform it, and write it into a file - 500 records at a time. Repo.stream/2 is a very handy tool dealing with large datasets, I highly recommend you give it a try if you haven’t heard of it before.

Nested Modules and Auto-Aliasing in Elixir

Wed, 10 May 2017 00:00:00 -0500

Recently, I was coding a module to create CSV files from database records when I came across an unexpected error in Elixir. Using the CSV library, I wrote something like this:

# mix.exs
defp deps do
  [
    {:csv, "~> 1.4.2"}
    ...
  ]
end

# csv.ex
defmodule Foo do
  defmodule CSV do
    def export(file) do
      CSV.encode(file)
    end
  end
end

I expected Foo.CSV.export(file) to trigger the CSV library to encode the file I passed into the function. To my surprise, the app came back with this error:

(UndefinedFunctionError) function Foo.CSV.encode/1 is undefined
or private

This error kind of makes sense. I’m calling a function on a CSV module, and I just defined a named CSV module right above it. Naturally, the app should be confused as to which CSV module I’m referring to. But I really didn’t want to change my module’s name, so how can I tell the app which module I want to use?

After some experimenting, I found something even more interesting - switching from a nested module to a single namespaced module made it work:

# csv.ex
defmodule Foo.CSV do
  def export(file) do
    CSV.encode(file)
  end
end

What?? I thought nested modules and namespaced modules compiled to the same thing? Shouldn’t the first example also compile to Foo.CSV? And why does the second example not throw the same error as the first one?

Why this happens

After asking around, I found this behavior puzzled other Elixir developers as well. Thankfully, Bryan Joseph helped me figure out what was happening and also pointed me to part in the docs that explains this behavior.

While compiling, when Elixir reaches a nested module, it creates an “auto-alias” for that nested module. In other words, this code:

defmodule Foo do
  defmodule CSV do
    def export(file) do
      CSV.encode(file)
    end
  end
end

actually compiles to something more like this behind the scenes:

defmodule Foo do
  defmodule Foo.CSV do
    def export(file) do
      CSV.encode(file)
    end
  end

  alias Foo.CSV, as: CSV
end

This is why my original example threw an error - when CSV.encode(file) is called, the auto-alias directs the app to instead look for an encode/1 function on the Elixir.Foo.CSV module.

Yet, when modules are not nested, the compiler does not create an auto-alias, so there is no confusion about which CSV module I’m trying to reference.

So if you are hitting a similar error with nested modules, consider either changing the name or switching over to a namespaced module instead.