# Environments

cnvrg is designed to make running and using remote computes easier than ever.

Part of that mission is affording you the flexibility to set your environment exactly the way you need it.

  • cnvrg lets you manage and utilize a versatile set of computes that meet a range of needs. Read more here.
  • cnvrg provides support for using custom Docker images for running experiments, launching workspaces, building flows and configuring endpoints and apps.

Every team and user has full control to build or use custom Docker images. In some cases, you would want to use a custom library without building a Docker image. To do so, you can use requirements.txt and prerun.sh files.

The topics in this page:

# requirements.txt (Python)

There are some cases when you would like to use a different version of a library or a package that isn't installed on the machine or included in the Docker image.

To support this use case, cnvrg offers an easy solution - if a requirements.txt file exists in the project root directory, cnvrg will install the packages listed in the file before the workspace or experiment starts.

To use a requirements.txt file:

  1. Create a file named requirements.txt and save it in the project head tree (in the main project directory).
  2. Specify which Python libraries you would like to use.

WARNING

If you are creating the file in your local environment, make sure you sync the file to cnvrg or your git repo.

Here is an example requirements.txt file:

docutils==0.11
Jinja2==2.7.2
MarkupSafe==0.19
Pygments==1.6
Sphinx==1.2.2
numpy
tensorflow

To generate this kind of format for your own project, run:

pip freeze > requirements.txt

# prerun.sh Script

When using cnvrg to run experiments or to start a notebook, there are some cases where you would like to run a bash script before starting the experiment or notebook.

To support this use case, cnvrg offers an easy solution. When the job starts, it will search for a prerun.sh file and automatically run it before the experiment/notebook starts.

To use a prerun.sh script:

  1. Create a file name prerun.sh and save it in the project head tree (in the main project directory). Include all the commands you wish to execute before the experiment starts.
  2. Make sure to sync or push the file before starting an experiment or a notebook.

# Add a Custom Docker Image

A Docker image is one of the main building blocks that cnvrg requires to execute a job. A job can be a workspace, experiment, web app, endpoint, or flow.

While cnvrg supplies capable, ready-to-use Docker images by default, if you prefer, you can use your own images.

NOTE

The working directory of any images to be used in cnvrg will be /cnvrg, regardless of what is set in the Dockerfile. To find out all the requirements for your custom images, read the requirements for custom images section.

cnvrg lets you:

  • Build your own Docker image: You upload a custom Dockerfile. cnvrg builds the Docker image and pushes it to the private registry that you choose.

    • While the image is being built, the build log is streamed live.
    • Once the image is ready, it is added to the table of images.
    • You receive an email notification about the success or failure of the image build.

    NOTE

    You can only build and push an image once you have connected cnvrg to a custom registry.

  • Pull a Docker image: You specify the information of an existing Docker image.

After successfully adding a docker image, you can select the newly created Docker image when you run a workspace or an experiment. When selected, the image will be pulled Kubernetes node and be used to run that specific job. If the image is already available locally, it will not be pulled but just loaded. For AWS, cnvrg creates a new AMI. For on-premise machines, the Docker image must be available locally on the machine.

NOTE

You can pull images from public registries without connecting to them.

Some registries come pre-configured in cnvrg.

TIP

You can also access the build image and pull image screens from within the page of the relevant registry.

# Build a custom Docker image

  1. Go to the Containers tab of your organization.
  2. Click Add Image.
  3. In the pane that appears, click Build Image.
  4. In the drop-down list, select the Registry that the image will be pushed to.
  5. Type in the Repository name and Tag of the image that will be created.
  6. In the Compute drop-down list, click one or more compute engines. cnvrg will attempt to run the job on the first available compute engine that you selected.
  7. Paste in your custom Dockerfile. Each time you edit the Dockerfile and click Add, cnvrg pushes it again to the relevant repository.
  8. (Optional) Type in a Readme.
  9. (Optional) Click Change Logo and select an image from the list provided.
  10. Click Add.

cnvrg builds the image and pushes it to the relevant repository and adds the image to the list of your available Docker images.

# Pull a custom Docker image

  1. Go to the Containers tab of your organization.
  2. Click Add Image.
  3. In the pane that appears, click Pull Image.
  4. Either:
    • Type in the full Docker URL; or
    • Select the registry in the drop-down list, type in the repository and type in the tag.
  5. (Optional) Type in a Readme.
  6. (Optional) Click Change Logo and select an image from the list provided.
  7. Click Add.

cnvrg adds the image to the list of your available Docker images.

# Custom images with on-premise compute

If you are using on-premise hardware with cnvrg, there are a few differences in the Docker image process.

Building a Docker image

If you build an image on on-premise compute, you have the option to not push the image to a registry. In this case, the image will be built and available for use on on-premise hardware locally.

NOTE

You can still push the image if you so choose.

Pulling a Docker image

You cannot pull a Docker image to on-premise hardware. To use a specific Docker image on on-premise hardware you must first ensure it is available locally on the machine.

Then, add the details of the Docker image to cnvrg and choose the Pull from existing repository option.

# Requirements for custom images

cnvrg is designed to make using custom Docker images as easy as possible, however there are a few important notes.

  • You must install the code language of your choice (Python, R and others).
  • To use R Studio or R Shiny you must install them yourself.
  • You must have pip installed (version > 10.0).
  • To use your custom image with a Visual Studio Code workspace, you must install tar.
  • While cnvrg will clone your git repository for you when stating a job (when your project is linked wit git), if you want to use git commands (for example, git push), you must install git and ssh yourself.
  • When using the container in cnvrg, the project will be downloaded or cloned into /cnvrg. You do not need to set this as the WORKDIR in your Dockerfile.

If you have the Install Job Dependencies Automatically organization setting enabled, your custom Docker image will automatically work with all of the cnvrg features (workspaces, serving, experiments and so on). This includes the cnvrg SDK. The toggle for this feature can be found in your organization Settings.

# Recommendations when Install Job Dependencies Automatically is enabled

While not required, we suggest you install the cnvrg CLI in your Docker image so that you can use all of the CLI commands. Otherwise, the CLI will only be usable in the folders /cnvrg (the workdir) and /data (where datasets are mounted).

To use git easily inside a JupyterLab workspace we recommend you install the JupyterLab git extension.

# Required packages when Install Job Dependencies Automatically is disabled

If you have the Install Job Dependencies Automatically organization setting disabled, you will need to install the required packages yourself. You will also be unable to use the cnvrg CLI and SDK unless you install them yourself.

Here is a table outlining the required packages to install for each feature you want to use your custom Docker image with.

You may need to scroll horizontally to see the full table.

Package Experiments JupyterLab Workspace R Studio Workspace Shiny App Dash App Voila App Tensorboard Compare Serving
tensorboard
butterfly
jupyterlab
jupyterlab-git
RStudio
dash
dash-daq
voila
pygments
gunicorn
flask

# Installation guides for packages

WARNING

Creating a Dockerfile and building a working custom image can be a very technical process. Each situation can be different and will require you to do your own research while building your Dockerfile.

The following code snippets are examples and may not work in your own Dockerfile.

# cnvrg CLI

RUN apt-get install -y ruby ruby-dev
RUN gem install cnvrg --no-ri --no-rdoc

# pip (version > 10)

RUN pip install --upgrade pip

# tar

RUN apt-get install tar

# cnvrg SDK

RUN pip install cnvrg

# TensorBoard

RUN pip3 install tensorboard

# Butterfly

RUN pip3 install butterfly

# JupyterLab

RUN pip3 install jupyterlab

# JupyterLab-git

RUN jupyter labextension install @jupyterlab/git
RUN pip3 install jupyterlab-git && jupyter serverextension enable --py jupyterlab_git

# Dash

RUN pip3 install dash

# Dash DAQ

RUN pip3 install dash-daq

# Voila

RUN pip3 install voila

# Pygments

RUN pip3 install Pygments

# Gunicorn

RUN pip3 install gunicorn

# Flask

RUN pip3 install flask

# R Studio

The simplest method is to build your image based on a Rocker image.

FROM rocker/rstudio:latest

# R Shiny

The simplest method is to build your image based on the official R Shiny Rocker image.

docker pull rocker/shiny

# Using Docker Images in Jobs

There are three ways to choose a Docker image for a job:

  • Using the image selector in the web UI
  • Using the CLI
  • Using the SDK

NOTE

cnvrg will always pull the latest version of the Docker image and tag as chosen for the job.

# Using the image selector in the web UI

Experiment image choice

You can use the image selector to choose a Docker image when starting a workspace, preparing a flow task, creating a new experiment, or deploying an endpoint from the web UI.

  1. Click Start Workspace, New Experiment, choose your Task or Publish. Fill in the other relevant details in the pane that appears.
  2. To choose your image, click the Image drop-down list.
  3. First, click the image repository toggle list you want to use.
  4. Then click the tag of your choice from the list.
    The repository and tag will now be selected in the Image drop-down list.
  5. Click Start Workspace, Run, Save Changes or Deploy Endpoint.

The selected image will be used as the virtual environment.

TIP

The image selector is found in the Environment subsection for experiments and in the Advanced tab for flow tasks.

# Using the cnvrg CLI

To use your new image through the CLI, add the --image flag in the run command. For example:

cnvrg run --image="tensorflow:19.07" python3 mnist.py

# Using the Python SDK

In the run() SDK call, pass in the name of the image. For example:

from cnvrg import Experiment()
e=Experiment()
e.run(python3 train.py,
      image="tensorflow:19.07")

# Connect to a Docker Registry

In cnvrg, you can connect to other Docker registries, beyond the registries provided by default.

This allows you to pull from these registries and easily add new images to cnvrg from them. In the case of a private registry, you can also build and push new images to them.

# Default registries

# cnvrg

Your system is automatically connected to cnvrg's Docker registries. These contain the default images that are built and maintained by the cnvrg team.

# NVIDIA NGC

cnvrg has seamless integration with NVIDIA's NGC platform. Without any further setup, you can pull all of NVIDIA's Docker images.

# Docker Hub

cnvrg is connected with the official Docker Hub registry. You can pull from any public repositories within Docker Hub by simply adding the details to cnvrg. Follow the instructions here.

TIP

To pull an image from a public registry, you do not need to connect to it beforehand. However, it is however best practice to connect to it.

# Add a registry to cnvrg

To add a private or public registry to cnvrg:

  1. Go to the Containers tab of your organization.

  2. Click Add registry.

  3. From the list of registries, select the registry you wish to connect to.

  4. In the Registry URL field, type in the full registry URL.

  5. Type in a title for the new registry.

    NOTE

    If you require authentication for the new registry, click Authentication, then provide a username and password.

    Authentication is optional, because most cloud providers can be connected directly to the Kubernetes cluster and additional authentication is unnecessary. It may also be a public registry.

  6. Click Save.

# Explore your Docker Images

cnvrg provides a set of images that meet most of the needs of data scientists working in machine learning settings. Using a ready-made image saves you the often complex task of building Docker images that cnvrg supports.

You can also add your own custom Docker images to use within cnvrg.

# Viewing your available Docker images

To view the images and registries that are known to cnvrg, navigate to the Containers tab of your organization.

# Registries

Along the top of the pane are the registries that cnvrg has been configured to connect to.

A registry is a storage and content delivery system, holding repositories, within which are the Docker images, available in different tagged versions.

There are two types of registries in cnvrg:

  • Public: cnvrg, Docker Hub, NVIDIA, and others. These registries are marked with a globe.
    Images can only be pulled from the public registries provided with cnvrg. You will not be able to push new images to them from within cnvrg.
  • Private: ACR, ECR, GCR, NVIDIA private, Docker Hub private and others.
    Images can be both pulled from and pushed to a private registry.

You can connect cnvrg to any additional registry to which you have access.

Clicking an entry will display the registry's page. From there, you can edit and delete the registry as well as add images (build and pull).

# Images

In the lower area is a table showing the Docker images that are known to cnvrg: both the images that are provided by default with the product (they reside in the cnvrg registry), as well as images that you have added.

For each image, the following information is provided:

  • Status
  • Repository name
  • Tag
  • Registry name
  • Creation date
  • Author name

You can sort and filter the data in the table.

Clicking an entry in the table displays a summary of the image and its Readme. From there, you can edit and delete the image.

General

# Custom Python Modules

You can use your own custom Python modules within cnvrg. To avoid any issues, you should follow best practices when setting up your custom modules:

  1. Create and add a file named __init__.py to the modules folder in your file structure.
  2. Add the following code snippet to the beginning of the code you are executing:
import os, sys
from os.path import dirname, join, abspath
sys.path.insert(0, abspath(join(dirname(__file__), '..')))

# Git Submodules

If your project is connected to a git repository that contains submodules, you must add the following to the prerun.sh file in your repo or Files. You must also have an OAuth git token set for your user account.

If you do not already have a prerun.sh file, create one, and then add the following code.

Add to your prerun.sh file:

git config --global url."https://x-access-token:${GIT_REPO_CLONE_TOKEN}@github".insteadOf https://github
git submodule init
git submodule update

TIP

A prerun.sh script allows you to run commands to further customize your environment at the start of every job.

# Automate cnvrg Authentication

If you want to automate cnvrg CLI commands with a script you can easily authenticate with cnvrg.

First use a machine that has the cnvrg CLI installed, and find the password/token from your .netrc file. You can find it using the terminal with the cat command:

cat ~/.netrc

In the script you are creating, add the following lines:

export CNVRG_USER="username"
export CNVRG_EMAIL="email@address.com"
export CNVRG_TOKEN="password_from_previous_step"
export CNVRG_OWNER="organization_name"
export CNVRG_API="your_API_URL"
cnvrg auth

Then any cnvrg CLI commands will work as part of the script. For example, you could use this authentication to automate the following commands:

mkdir data2
cd data2 && cnvrg data init --title="test_data" && cnvrg data sync

TIP

If you are unsure of your API URL, go to the CLI docs to find out more.

Last Updated: 6/16/2020, 6:37:22 AM