# cnvrg CLI

As data scientists, we use the terminal a lot - to run experiments, to write code, to change parameters, and more.

This was the main reason that we built a simple and easy-to-use CLI tool, so you can keep doing what you love the way you know it, just faster and in a more organized fashion.

With the cnvrg CLI installed, you can create projects, run experiments, and sync projects and more, all from the comfort of your terminal.

In the following guide, we'll walk you through installing Ruby and the cnvrg CLI gem. This page also has information on all of the different CLI commands that you can use after you've installed!

The topics in this page:

# Download and Install the cnvrg CLI

# Install Ruby

cnvrg CLI was built with Ruby. Ruby is a great programming language and can be very easily installed.

WARNING

cnvrg currently supports Ruby versions 2.3 - 2.6.6. Ruby 2.7.0 is not currently supported.

    # Install the cnvrg CLI

    Now that you have Ruby configured, move to the next step and install cnvrg CLI.

    Check your version of Ruby by running the following command:

    ruby -v
    

    Open the terminal and run the corresponding command for your version of Ruby.

      Verify cnvrg is installed:

      cnvrg --version
      

      If the output was a version number, then the cnvrg CLI has been succesfully installed.

      # Setup and log in

      After successfully installing the CLI, you now need to set the API URL and log in.

      To allow your CLI to communicate with your cnvrg environment, set the API URL using the cnvrg --api command:

      cnvrg --api '/api'
      

      You can copy the above code exactly to set your correct API. If you're still not sure what your unique Web URL is, log on via the Web and copy the URL from your browser, or contact support for additional help.

      Now log in and authenticate using the cnvrg login command:

      cnvrg login
      

      You will be prompted to enter your email and password.

      After succesfully logging in, you will have everything set up and can start using all the CLI commands.

      # CLI Help

      To find out information about any of the CLI commands, use cnvrg help [command] or cnvrg --help [command]

      # Environment Setup Commands

      # Connect to your private environment (set the API URL)

      To ensure the CLI can communicate with your cnvrg environment, you must set the API URL using the cnvrg --api command:

      cnvrg --api '/api'
      

      You can copy the above code exactly to set your correct API. If you're still not sure what your unique Web URL is, log on via the Web and copy the URL from your browser, or contact support for additional help.

      # Log in

      Log in to cnvrg and authenticate using your credentials:

      cnvrg login
      

      # Log out

      To unauthenticate your system:

      cnvrg logout
      

      # Check the logged in user

      Use the me command to see which account is currently logged on to cnvrg CLI:

      cnvrg me
      

      # Check the cnvrg CLI version

      Use the version command to check the currently installed version of cnvrg CLI:

      cnvrg version
      

      # Project Commands

      # Create an empty project

      To simply create a new, empty cnvrg project, use the new command. This command will create a new folder with the default tree of files:

      cnvrg new 'project_name'
      

      You can create a project and sync a directory to it with a single command.

      Navigate into the directory containing the files to upload. From there, run the the link command:

      cnvrg link --title='project_name' --sync
      
      Flags Type Description
      --title=TITLE string Title of project. Default is the directory name.
      --sync boolean Syncs the files with the online directory.

      If you have created a project online already but now would like to link a directory connected with Git, use the link_git command:

       cnvrg link_git "cnvrg_project_url"
      

      The project url is the full url path to the project in your cnvrg deployment.

      If your project was at https://app.cnvrg.io/org/project/project_name/, in the browser then this url is what you use in the cli command. :::

      # Clone an existing project

      To clone or replicate an already existing project:

       cnvrg clone "cnvrg_project_url"
      
      Flags Type Description
      --commit=COMMIT string specify which commit the new project is cloning from

      Use the unlink command to unlink your local directory from the cnvrg project:

       cnvrg unlink
      

      # Other project commands

      • Use the jump command to change to a specific commit:

         cnvrg jump commit_sha1
        

      File Commands:

      • Use the download command to retrieve files from the cnvrg server:

        cnvrg download
        
      • Use the upload command to send files from your local machine to the cnvrg server:

        cnvrg upload
        
      • Use the sync command to update the server about any file changes in your local directory:

        cnvrg sync
        

        TIP

        You can use cnvrg sync -f to commit directly to master.

      ::: note Run cnvrg sync command from your project root directory, and not from any of its subfolders. :::

      # Dataset Commands

      # Dataset ID

      In many commands, you will need to use a dataset ID. The dataset ID is the name used for the dataset in its URL.

      For example, if you have a dataset that lives at: https://app.cnvrg.io/my_org/datasets/dataset_name, the dataset ID is dataset_name.

      Then to use cnvrg data link, the command would be cnvrg data link dataset_name.

      # Initialize a new dataset with a local directory

      Navigate into the directory for the dataset and use the data init command:

      cnvrg data init
      

      Using this command will create a new empty dataset with the same name as your local folder.

      NOTE

      If you would like to set a custom title of the dataset, simply add --title="custom_title":

      cnvrg data init --title="custom_title"
      

      Keep in mind that you cannot use whitespaces in the dataset's title.

      First, go to the dataset inside cnvrg's Web UI and copy the dataset ID from the URL.
      Then navigate into the local directory that you are linking to the existing directory and use the data link command:

      cnvrg data link dataset_id
      

      You can now use commands like cnvrg data sync to push changes from your local directory to the remote one.

      TIP

      The dataset id is usually the full name of the dataset. You can also use the full URL instead.
      If your dataset was at https://app.cnvrg.io/org/datasets/dataset_name, you could either use the full URL or just dataset_name.

      # Upload files to a dataset

      To upload a file or files into a remote dataset, use the data put command:

      cnvrg data put [--commit=SHA1/latest] [--message='MESSAGE'] [--threads=1000] dataset_id file_path/file
      
      Flags Type Description
      --commit=COMMIT string The commit to upload the file or directory to. Can be either the sha1 of a specific commit or latest. If the flag is not used, it will create a new commit.
      --message='MESSAGE' string Add a message to the commit.
      --threads=THREADS integer Configure the amount of CPU threads to use for the upload. Default is 15.
      --chunk-size=CHUNK-SIZE integer The size of each portion of the upload that files are split into. Default is 1000.
      --force flag Create a new commit with only the files included in the current upload.
      --override flag Use this flag to replace existing copies even if they are identical. Every file included in the command will be uploaded.
      --dir string Upload the files to a specific subdirectory in the remote dataset. For example, --dir="images" will upload the files to the images/ folder. It will create the directory if it does not exist.

      The data put command will upload the specified files to the remote dataset.

      NOTE

      In CLI versions below 1.9.5, you must use the full dataset URL. For example: cnvrg data put https://app.cnvrg.io/my_org/datasets/mnist image.png

      Examples:

      • Upload the entire current folder recursively (including all files & sub-folders & folder structure):

        cnvrg data put dataset_id .
        
      • Upload a specific folder and its content (not including sub-folders):

        cnvrg data put dataset_id folder/*.*
        
      • Upload a specific file from a path (the path will be matched):

        cnvrg data put dataset_id folder1/folder2/file.txt
        

      # Remove files from a dataset

      To remove a file that exists in a dataset, use the data rm command:

      cnvrg data rm [--message='MESSAGE'] dataset_id file_path/file
      
      Flags Type Description
      --message='MESSAGE' string Add a message to the commit.

      The data rm will remove the matching files from the dataset.

      Examples:

      • Delete an entire folder and all its contents:

        cnvrg data rm dataset_id folder/
        
      • Delete a specific file:

        cnvrg data rm dataset_id file_name
        

      # Commit caching and clearing commands

      To cache a commit onto an NFS disk, use the data cache command:

      cnvrg data cache commit_sha1 disk_slug
      

      To clear a commit currently cached on an NFS disk, use the data clear command:

      cnvrg data clear commit_sha1 disk_slug
      

      You can also list the connected external disks using the external_disk_list command:

      cnvrg external_disk_list
      

      # Other dataset commands

      • To update a local version of a dataset with files from the remote version, use the data download command:
      cnvrg data download
      
      • To clone the dataset from the server into your local environment:

        cnvrg data [--commit=SHA1/latest] clone dataset_id [--only_tree] [--query='query_name']
        
        Flags Type Description
        --commit=COMMIT string the commit to clone the dataset from
        --only_tree boolean whether to download only the tree of directories
        --query string downloads only the files that match this query
      • To see a list of all datasets, use the data list command:

        cnvrg data list
        
      • To see a list of the files associated with a specific dataset, use the data files command:

        cnvrg data files dataset_id
        
      • To retrieve a list of existing commits (and cache information) for the current dataset, use the data commits command:

        cnvrg data commits dataset_id
        
      • To retrieve a list of existing subsets and queries for the current dataset, use the data queries command:

        cnvrg data queries dataset_id
        
      • To see a list of the files associated with a specific query on the current dataset, use the data query_files command:

        cnvrg data query_files dataset_id 'QUERY_NAME'
        

      # Workspace Commands

      # Run a remote SSH session

      To start a remote SSH session with an open workspace inside cnvrg, use the cnvrg ssh start command:

      cnvrg ssh start [--kubeconfig=/path/to/kubeconfig] [--user=username] [--port=desired_local_port] [--password=desired_password] workspace_id
      
      Flags Type Description
      --kubeconfig=/path/to/kubeconfig path Set a specific kubeconfig. By default the kubeconfig is taken from the KUBECONFIG environment variable. If you have multiple contexts in the kubeconfig, maje sure you have the current context set to the workspace's cluster.
      --user=username string Choose a specific username for SSH session. Default is root. The user must exist in the workspace's container.
      --port=desired_local_port int Choose a specific local port to connect the SSH session to. Default is 2222.
      --password=desired_password string Choose a specific password for the SSH session. The default is a randomly generated string. You can use this flag to make the password consistent between sessions.

      The SSH session will stay live until the command is interrupted. If the workspace stops, the command will not automatically stop and the user must stop it themselves.

      NOTE

      To disable host key checking for ssh by default add the following lines in your ssh config file (usually located at ~/.ssh/config):

      Host localhost
        StrictHostKeyChecking no
      

      If you would like to disable it for all hosts, change localhost to *, such that:

      Host *
        StrictHostKeyChecking no
      

      TIP

      The workspace_id can be found in the URL for the workspace.

      For example, if your workspace is live at https://app.cnvrg.io/acme/projects/recommender/notebook_sessions/show/ekqjsnzerv9x5t62y7yd, then the workspace_id would be ekqjsnzerv9x5t62y7yd.

      The corresponding example command would be: cnvrg ssh start ekqjsnzerv9x5t62y7yd.

      # Experiment Commands

      # Run a new experiment

      Use the run command to run a new experiment. Ensure the experiment command (for example python3 train.py) is the last parameter of the command and any flags you are using are entered before the command.

      For example:

      cnvrg run [-t='experiment_title'] [--data='dataset_name'] [--machine='machine_name'] command
      
      Flags Type Description
      --machine=MACHINE string Set the compute. For Kubernetes compute it should be in the format: cluster_name.template_name. For example, workers.medium. You can leave out cluster_name to use the default cluster. For an on-premise machine, simply write the name of the machine. For example, worker1.
      --sync_before boolean Sync the directory before running the experiment
      --sync_after boolean Sync the directory after running the experiment. Default is True.
      --image="IMAGE_NAME" string Docker image to use.
      --title=TITLE string Set the title of the experiment.
      --log boolean Display the log of the experiment.
      --wait flag Wait for the experiment to complete. The command will return the exit status of the experiment.
      --commit=COMMIT string Choose a specific commit/version id of the project.
      --data=DATASET_NAME string Choose a dataset to run the experiment with.
      --data_commit=DATA_COMMIT string Choose the specific commit/version id of the dataset to run the experiment on
      --datasets=DATASETS array of dictionaries A list of datasets and commits/queries (that is: [{"id": "dataset_id-1", "commit": "commit_sha1-1", "query": "query_name-1", "tree_only": true, "use_cached": true}, {dataset_dict_2}, ...]) to attach to the experiment. You can use either commit or query and not both. use_cached is only relevant when integrated to NFS and is true by default.
      --grid='grid.yaml' path Run a grid search using a YAML file outlining the parameters.
      --output_dir=OUTPUT_DIR string When running remote experiment in git project, sets the output folder for cnvrg to track experiment artifacts.
      --git-branch=GIT_BRANCH string Specific branch to use when running Experiment in git project.
      --git-commit=GIT_COMMIT string Specific commit to use when running Experiment in git project.
      --git_diff=True boolean When git_diff is enabled cnvrg will also sync the git difference, saving the changes relative to your git index along with the experiment. Default is False.
      --prerun=False boolean Flag for whether or not to run the prerun.sh script. Default is True.
      --requirements=False boolean Flag for whether or not to install the packages in the requirements.txt file. Default is True.
      --schedule="dd.mm.yyyy hh:mm<PM/AM>" or --schedule="yyyy.dd.mm hh:mm<PM/AM>" string Set a time to execute the experiment. The time is relative to the timezone defined for the current user. You can specify the time using either a 24- or 12-hour clock format. For example: To specify 4:25 PM on the 31st of January 2012, you can specify any of these: 31.01.2021 16:25, 31.01.2021 4:25PM, 2021.01.31 16:25, 2021.01.31 4:25PM.
      --recurring="* * * * *" string Set a schedule for executing a recurring experiment. Specify the schedule using Cron format. For example, to execute the experiment at 30 minutes past every hour, specify: "30 * * * *"
      --notify_on_success boolean Receive an email when the experiment ends successfully. Default is False.
      --notify_on_error boolean Receive an email if an error occurs while your experiment is running. Default is False.
      --emails=EMAILS string Add extra recipients for email notifications for the experiment. The string should be a list of comma-separated emails. For example, "person1@email.com,person2@email.com".

      ::: note TIP The --git_diff flag is useful when developing in a workspace. With it, you can fiddle with your code and run experiments without first syncing to git. The code you run with will still be synced so you can eventually push back the code that your are most happy with. :::

      # Example YAML file for grid searches

      parameters:
          - param_name: "learning_rate"
            type: "discrete" # An array of numerical values
            values: [0.1, 0.01 ,0.001]
      
          - param_name: "kernel"
            type: "categorical" # An array of string values
            values: ["linear", "rbf"]
      
          - param_name: "epochs"
            type: "integer"
            min: 10 # inclusive
            max: 200 # not inclusive
            scale: "linear"
            steps: 10 # The number of linear steps to produce.
      
          - param_name: "fractions"
            type: "float" # Precision is 9 after period
            min: 0.1 # inclusive
            max: 0.2 # not inclusive
            scale: "log2"
            steps: 10 # The number of linear steps to produce.
      

      TIP

      The scale can be linear, log2 or log10.

      # Flow Commands

      You can use Flow YAML files to build and run Flows from the CLI. To see how to construct a Flow YAML visit the Flows section.

      # Create a Flow online using a local YAML file

      cnvrg flow import --file='flow.yaml'  
      

      This command will create a new Flow in the Project from the data within the specified file.

      # Download an online Flow as a YAML file

      cnvrg flow export --title='Flow_Title' [--version='version'] [--file='name_of_file_to_save.yaml']
      

      This command will download the specified Flow as a YAML and save it in your local folder.

      Flags Type Description
      --title='TITLE' string Required. Sets the name of the online Flow you will be downloading.
      --version='VERSION' string Sets the version of the Flow you will be downloading. Default is latest.
      --file='FILE_NAME' string Optionally sets the name of the downloaded Flow YAML.

      # Run a Flow using a local Flow YAML file

      cnvrg flow run --file='flow.yaml'
      

      This command will run a flow using the specifications within the supplied Flow YAML file.

      TIP

      If you would like to run a Flow that you created online using the CLI, first use cnvrg flow export to download the Flow's YAML. Then use cnvrg flow run to run the Flow using th downloaded file.

      # AI Library Commands

      You can create a new component or update an existing component in the AI Library using the library import command. This command uses a library.yaml file and other accompanying files to create a library.

      Run the library import command in a directory containing the library.yaml and the files needed to construct your library component.

      cnvrg library import
      

      # Miscellaneous Commands

      • To print the cnvrg CLI logs file to your screen, use the logs command:

        cnvrg logs
        
      • To retrieve a list of comes or machines, use the list_machines command:

        cnvrg list_machines
        
      • To receive usage tips for any cnvrg CLI commands, use the help command:

        cnvrg help [COMMAND]
        
      • To set the compression path, use the set_compression_path command:

        cnvrg set_compression_path 'path'
        
      • To set the current logged on account as the default owner of the current project, use the set_default_owner command:

        cnvrg set_default_owner
        
      • To see the status of the working tree and ensure your local and remote projects are in sync use the status command:

        cnvrg status
        

      # Troubleshooting the cnvrg CLI

      If you experience issues in running CLI commands, troubleshoot by performing the following checks:

      # Update the cnvrg CLI

      Sometimes updating the CLI can help resolve the issue.

      To update the CLI, run the command:

      gem update cnvrg
      

      If you already have the latest version you will get the following response:

      Updating installed gems
      Nothing to update
      

      If you do not have the latest version, the newest version will be downloaded and installed.

      After updating the CLI, check if your issue persists.

      # Check you are using the correct API

      cnvrg needs to communicate with your specific workspace. To do so, you need to provide it a link to the correct URL API.
      As part of the set up process, you should have set your API to the correct URL, but often this may be the root cause of common CLI issues.

      Your specific URL API is: /api

      NOTE

      It should simply be the URL you use to interact with cnvrg online followed by /api.
      For example, https://app.cnvrg.io/api, https://cnvrg.mycompany.com/api and so on.
      Make sure you have not included your organization's name in the API link.

      You can check the URL API you are currently using by running the command: cnvrg me.

      To set your API correctly first log out:

      cnvrg logout
      

      Then set the API:

      cnvrg --api '/api'
      

      Finally, log in again:

      cnvrg login
      

      You should now have correctly set your API. Check if your issue persists.

      # SSL verification error (Windows)

      When you attempt to install the cnvrg CLI on a PC running Windows, you may receive the following error:

      ERROR: SSL verification error at depth 2: self signed certificate in certificate chain (19) ERROR: Root certificate is not trusted
      

      To resolve the issue, run the following two commands:

      gem sources --remove https://rubygems.org
      gem sources --add http://rubygems.org
      

      Now, try again to install the cnvrg CLI by running the following command:

      gem install cnvrg
      

      # Install Xcode Command Line Tools (MacOS)

      If you are having repeated errors installing Ruby, you may not have the Xcode Command Line Tools installed on your MacOS machine. Run the following command and then try again:

      xcode-select --install
      

      NOTE

      Sometimes, updating your version of MacOS requires a reinstallation of the Xcode Command Line Tools to fix common command line issues.

      # Put the gem in your PATH

      After installing the CLI, it is possible that the CLI's gem is not in your PATH. This may be the case, if after installing the CLI, when using a command you get the following error:

      bash: cnvrg: command not found
      

      To fix this issue:

      1. Run the following command:
      gem environment
      
      1. Copy the path from EXECUTABLE DIRECTORY. It should end with /bin. For example, /usr/local/bundle/bin)
      2. Run the following command, with the copied path from the above step:
      export PATH=/path/to/executable/directory/bin:$PATH 
      

      For example, using the above example: export PATH=/usr/local/bundle/bin:$PATH
      4. To ensure this problem is solved (and you do not need to repeat these steps), run the following command (with the same path used above) to automatically export the CLI's gem to the PATH every time you run a new terminal session:

      # if using bash
      echo 'export PATH="/path/to/executable/directory/bin:$PATH"' >> ~/.bashrc 
      
      #if using zsh
      echo 'export PATH="/path/to/executable/directory/bin:$PATH"' >> ~/.zshrc 
      

      # Contacting cnvrg support

      If the above methods have not resolved your issue, please contact the cnvrg support using the online chat bubble at the bottom right of the cnvrg web app.

      For quick and efficient support, please include the output of the following command in your request:

      cnvrg logs
      

      Support will help resolve your issue as quickly as possible.

      Last Updated: 7/1/2021, 10:38:07 PM