Setting Up Fast AI on RC's Community Cluster

January 13, 2019

Setting up fast.ai on the cluster (a rough outline)

RC's community cluster is a great resource to take advantage of! I had a rough time getting the Fast AI course set up, so I wrote this rough guide to (hopefully) help others save some time.

  1. Make an account on recurse.com/settings/cluster - this requires setting up and copying over an SSH key (instructions here)[https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/].

  2. SSH onto one of the GPU-enabled machines - I chose to use mercer (both Mercer and Crosby have powerful graphics cards. You can find more info on the machine specifics here)

    ssh your-username@mercer.cluster.recurse.com
  3. Once on the machine, you should be in your own home directory. The command prompt should look something like this:

    mari@mercer:~$
  4. Install Conda so that you can create virtual environments to install software that won't affect others using the machine.

  5. Install the fast.ai library (instructions from the course forum)

    git clone https://github.com/fastai/fastai.git
    cd fastai
    conda env create -f environment.yml
  6. Activate the environment for fast.ai

    source activate fastai
  7. On your local machine, in a separate tab, set up port forwarding so that the port you're serving up the jupyter notebook to can be opened on a browser on your local machine.

For example, if I want to open up the notebooks on port 8888 on my local machine, and use port 6565 on the server:

ssh -N -f -L localhost:8888:localhost:6565 mari@mercer.cluster.recurse.com
  1. Back on the cluster, inside the fast.ai directory:

    jupyter notebook --no-browser --port 6565

The --no-browser flag prevents jupyter from trying to open a browser inside the remote server, and the port flag specifies the port that the server will use.

  1. In your browser, navigate to the link that the jupyter notebook gives you, but change the port that you used above. In my case, jupyter served up

    http://localhost:6565/?token=#####

so I would navigate to

http://localhost:8888/?token=#####

Notes

  • Sometimes your env won't be automatically populated by conda, so you might have find your env elsewhere to run it:

    source activate anaconda/envs/fast-ai
  • Make sure you're using python3
  • To see if your port forwarding is running/working correctly:

    ps aux | grep ssh

Helpful resources: