Atlas CE on AWS¶
Being able to run Atlas CE on AWS means access to GPUs and faster results. In this tutorial we'll walk through setting up and running jobs on AWS with Atlas CE.
- An AWS account
First, let's go to the AWS developer console. Sign in, and then we'll create a new instance: AWS console
Go to Services > EC2, then click the "Launch Instance" button to start setup.
The custom AMI we'll be using for this tutorial is only available in us-east (Ohio), Canada (Central), and EU (Ireland), so you'll need to switch to one of those regions before creating an instance.
Step 1: Choose an Amazon Machine Image (AMI)¶
Let's use the custom Atlas CE AMI, to do this:
- Select "Community AMIs" on the left sidebar
- Search for "Atlas CE"
- Select the AMI that says "Atlas Community Edition on Deep Learning AMI for Ubuntu "
Our custom AMI is just AWS's Deep Learning AMI (Deep Learning AMI (Ubuntu 16.04) Version 24.2) with Atlas CE added on top of it.
Once the instance has been finalized and spun up, costs will be incurred. Please follow the tear down steps at the bottom of this guide to minimize this.
Step 2: Configure an Instance Type¶
For this page we'll select a p2.xlarge instance type, as we want some a GPU to train with. We recommend a P2 instance as it a good balance between performance and cost, but if you are looking for more power you can select a P3 instance.
Make sure to click "Next: Configure Instance Details" to continue provisioning the instance.
Step 3: Configure Instance Details¶
- If your instance does require additional configuration adjust as needed
- Otherwise there are no changes needed here, just click "Next: Add Storage"
Step 4: Add Storage¶
We recommend increasing the size of your storage to at least 250GB, as the base AMI image is already ~80GB. Adjust as needed for your expected data.
Step 5: Add Tags¶
- No changes, just click Next: Configure Security Group
Step 6: Configure Security Group¶
Next, we'll create a new security group to allow for Atlas to securely use a few different ports on our instance. Specifically to allow the GUI, REST API, and archive server.
To allow for this:
- Add 3 new rules, with ports for 5555, 5557, and 5959
- Adjust the "Source" to My IP for each rule
Now, click Review and Launch to go review our instance before launching.
Review Instance Launch¶
Our instance setup should now look similar to the below.
- Click "Launch" which should then ask you to create a new key pair
- Select "Create a new key pair" and give it the name "atlas-ce" (note: if you've already used this key name before you can either re-use of, or create another unique key name)
- Download the key pair
- Click "Launch Instances"
- It should then redirect you to the Launch Status page
- Beside "The following instance launches have been initiated" you should see your new instance ID. Click the ID to go to the Instances page
- You can find the IPv4 address of the instance in the lower Description panel.
You should then be able to use your downloaded
.pem key to then SSH into the instance to check that everything is setup.
Be sure to run
chmod 400 <key_name>.pem on your key before SSH'ing to give it the proper permissions. We also recommend that you move the key to your
ssh -i /path/to/key/<key_name>.pem firstname.lastname@example.org
You should expect to wait ~30 seconds when you first SSH in.
The instance will run a few commands which includes activating a conda environment that contains
foundations. It also runs
atlas-server so all the services for Atlas CE are running.
When you SSH into the instance,
atlas-server will start, but you can put it into a background process with
ctrl + c, to then be able to run other commands.
You should also now be able to see the GUI. In your browser go
<ipv4.of.ec2.instance>:5555 and you should see the Atlas CE project page with two projects that have each been run once. These projects were baked into the AMI to help get started.
Run our first Atlas CE job¶
If this is your first time using Atlas CE, you can try run a simple job with the following steps. Or you can just skip ahead to setting up AWS to remotely work with VSCode below.
We can run one of our demo projects to try out how the
foundations CLI works.
foundations submit scheduler . driver.py
- Go back to the Atlas CE GUI in your browser, and you should see the new job we've just run
For more information on the Foundations CLI read our CLI section, or run
We've now successfully setup a P2 instance with Atlas! In the next section we'll go over running jobs on AWS with VSCode.
Running jobs remotely via VSCode¶
A really easy to develop with Atlas locally in VSCode, while having the power of a cloud GPU, is the setup remote deployment in VSCode. While we also love PyCharm's remote deployment, it's not free, so we'll just use VSCode. This will allow for our code to be synced with our instance, and simply fun
foundations submit ... commands to run jobs.
- Install VSCode
First, open up VSCode, and we'll install the Remote - SSH plugin that will allow us to open code from the remote instance in VSCode:
- Open extensions via the left bar menu (or by pressing
shift + cmd + X)
- Search for
- install the plugin
Before we connect to the host in VSCode, let's open the SSH config file, to add this instance to our SSH Config, to easily allows VSCode to connect to our instance.
~/.ssh/config(Create it if it does not exist)
Add the following lines:
Host <my.aws.ip.address> IdentityFile ~/.ssh/<key_name>.pem User ubuntu
Let's also give the right permission to our
pem file using the following command:
chmod 400 ~/.ssh/<key_name>.pem
Once the plugin is installed and the SSH config is updated, open the Command Palette and select "Remote-SSH: Connect to Host".
- Open command prompt in VSCode:
Cmd + Shift + P(or click green "Open Remote Window" botton, bottom left of VSCode)
- Search for
Remote-SSH: Connect to Hostfrom the menu
- Select your instance ip address
It should open a new VSCode window where we'll be able to select "File" > "Open..." and then select a directory from our instance to open. In our case we can select our
atlas_tutorials/auction_price_regression_tutorial project which will then open up in VSCode in a new window.
At the bottom the VSCode window we'll select the "Terminal" tab, and click the "+" symbol to open a new shell tab. Choose "bash" from the drop down that will give us bash shell access to AWS.
- cd into
- Let's test that we can run a job with
foundations submit scheduler . driver.py
- You should see this job in the Atlas CE GUI
We're now set! You can now open the files and adjust as you wish, or you can start on your own project. To understand more about the
auction_price_regression project, you can check out the full tutorial in the Tutorial section of the docs.
If you're looking for more detailed docs on setting up VSCode with AWS, Microsoft has a good setup guide with much more detail.
Tear down AWS instance¶
It's important to be aware that after the instance has been spun up, you will incur costs on your account. There are two options: either stop the instance (which will mean costs are minimal to keep your storage around), or throw away the instance completely.
To stop or terminate the instance:
- Go to your AWS console
- Right click on our P2 instance > "Instance State" > either Terminate or Stop
That's it! We've successfully spun up a GPU instance, and run jobs remotely, from VSCode.
You can now either start your own projects, or look at some of our more advanced tutorials to explore more of Atlas CE.
If you have any thoughts or feedback about setting up Atlas CE on AWS, we're always happy to help answer questions on our Dessa Community Slack!