Tools for AWS Cloud Management in MagellanMapper

MagellanMapper provides a few tools for basic management of an Amazon Web Servicies (AWS) platform for running pipelines with cloud computing. These tools serve as wrappers to the AWS Boto3 Python-based client management to simplify common tasks related to MagellanMapper.

Dependencies

  • awscli: AWS Command Line Interface for basic up/downloading of images and processed files S3. Install via Pip.

  • boto3: AWS Python client to manage EC2 instances.

Launch EC2 Instances

You can launch a new EC2 instance of a custom AMI via MagellanMapper, such as an AMI with MagellanMapper pre-installed as described below.

To start an instance, use this command:

python -u -m magmap.io.aws --ec2_start "Name" "ami-xxxxxxxx" "m5.4xlarge" \
  "subnet-xxxxxxxx" "sg-xxxxxxxx" "UserName" 50,2000 [2]

  • Name is your name of choice

  • ami is your previously saved AMI with MagellanMapper

  • m5.4xlarge is the instance type, which can be changed depending on your performance requirements

  • subnet is your subnet group

  • sg is your security group

  • UserName is the user name whose security key will be uploaded for SSH access

  • 50,2000 creates a 50GB swap and 2000GB data drive, which can be changed depending on your needs

  • 2 starts two instances (optional, defaults to 1)

To log into the server with graphical support, SSH into your server instance with port forwarding to allow VNC access:

ssh -L 5900:localhost:5900 -i [your_aws_pem] ec2-user@[your_server_ip]

Start a graphical server (eg vncserver) to run ImageJ/Fiji for stitching or for Mayavi dependency setup. Now you can access the server graphically using a VNC cient pointing to localhost with port 5900.

Set Up Volumes

The setup_server.sh script sets up volumes on a new server instance:

bin/setup_server.sh -d [path_to_data_device] -w [path_to_swap_device] \
    -f [size_of_swap_file] -u [username]

  • -d is the main data drive, typically a drive large enough for your full image file

  • -w is a swap drive or file path

  • -f is a swap file size in GB, used if swap is set to a path rather than a device

  • -n will map device names to NVMe names, which allows drive names such as sdf to be mapped to the corresponding NVMe-style (eg /dev/nvme0n1) names

  • -u is the username, used to change ownership of the mounted drives; defaults to ec2-user and should be changed to ubuntu for Ubunut-based AMIs

  • -s to set up fresh drives including formatting; exclude when re-mounting drives that have already been formatted

Set up drives on a new server instance to format and mount data and swap drives or create swap files:

bin/setup_server.sh -d [path_to_data_device] -w [path_to_swap_device] \
    -f [size_of_swap_file] -u [username]

Set Up a Server with MagellanMapper

After launching a server, you can set up MagellanMapper by downloading it within the server or by deploying custom files such as a local branch using the depoy.sh script.

Install/Update From Main Respository

Typically graphical support (eg via vncserver) is required during installation for Mayavi and stitching in the standard setup, but you can alternatively run a lightweight install without GUI (see Readme).

See Installation for downloading and install MagellanMapper within the server.

Install/Update From Local Branch

The deployment script allows deploying MagellanMapper using local files including custom modifications. It also downloads Fiji for the image stitching pipeline. Use this command to deploy local files to a server:

bin/deploy.sh -p [path_to_your_aws_pem] -i [server_ip] \
    -d [optional_file0] -d [optional_file1]

  • This script by default will:

    • Archive the MagellanMapper Git directory and scp it to the server, using your .pem file to access it

    • Download and install ImageJ/Fiji onto the server

    • Update Fiji and install BigStitcher for image stitching

  • To only update an existing MagellanMapper directory on the server, add -u

  • To add multiple files or folders such as .aws credentials, use the -d option as many times as you’d like

  • After running this script, log in and install MagellanMapper if it has not been previously set up

Run MagellanMapper on Server

When returning to a server with MagellanMapper already set up, you’ll need to perform the following tasks:

  • Re-mount drives using setup_server.sh

  • Activate the Conda environment set up during installation

Now you can use the pipelines.sh script to perform tasks as whole images, such as this command to fully process a multi-tile image with tile stitching, import to Numpy array, and cell detection, with AWS S3 import/export and Slack notifications along the way, followed by server clean-up/shutdown:

bin/process_nohup.sh -d "out_experiment.txt" -o -- bin/pipelines.sh \
  -i "/data/HugeImage.czi" -a "my/s3/bucket" -n \
  "https://hooks.slack.com/services/my/incoming/webhook" -p full -c