# AI Library

AI Library Page

Building ML pipelines and continual learning are key to an effective machine learning workflow. Reproducible and modular code components are core components of any such workflow.

The AI Library in cnvrg facilitates these goals. It is a specially built package manager for ML components designed specifically for machine learning. It helps data scientists and developers build machine learning components and reuse them across projects.

cnvrg comes pre-configured with many AI Library components that you can start using straight away. You can also easily create your own. This allows you to share your work with your team and build a repository of your code and algorithms.

The topics in this page:

# Components

AI Library entry

A component is a piece of reusable code. It can be used for any machine learning task including training, preprocessing, and more.

Each component comprises the following:

  • Code
  • Arguments and parameters
  • A base image
  • A compute template
  • (Optionally) A description
  • (Optionally) Documentation

A component can be created via the web UI or cnvrg CLI.

# Preinstalled Components

cnvrg comes with preinstalled components for many powerful machine learning preprocessing and training algorithms. Our data science experts are constantly improving the library with the latest, cutting-edge algorithms.

You can explore all the prebuilt components from the AI Library page. They are each comprehensively documented and all of the features and algorithms are fully described.

cnvrg includes components for K-nearest neighbors, linear regression, random forest classifier, XGBoost and more.

All prebuilt components come configured with auto-tracking and advanced visualizations such as:

  • Confusion matrices
  • Correlation matrices
  • Loss and accuracy graphs

Experiment tracking

# Use a Component

You can use a component from the AI Library in two ways:

  • As a task in a flow
  • Using the cnvrg SDK

# As a task in a flow

Choose a component

To use the component as a task in a flow within one of your projects:

  1. Go to Projects > Flows.
  2. Open a new or existing flow.
  3. Click the New Task drop-down list on the top right and choose the component you would like to use.
  4. In the panel that appears, configure all the relevant details (parameters, compute, image and so on.)
  5. Click Save Changes.

The task is added to your flow.

Now you can link it up to other tasks, or click the Play button to execute the component.

TIP

If you click Custom Task, your recent AI Library components will appear on the Enter a command or search... panel. You can quickly click them there as well.

# Using the SDK

To run a component using the cnvrg SDK, you use the following format:

from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg1="val1",arg2="val2",...)

You can also run a grid search by using an array for an argument:

from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg=["val1","val2"])

More information on this SDK method can be found here.

# Add a Custom Component

You can turn any code you have written into a component and export it to the AI Library.

There are two ways to add a component to the AI Library:

  • Using the web UI
  • Using the cnvrg CLI

# Using the web UI

Export a task to the library

You can export your code as a component from any of your projects.

  1. Open the project that contains the code you want to export.
  2. Open the Flows section of the project.
  3. Create a new flow or open an existing one.
  4. Click the New Task drop-down list.
  5. Click Custom Task.
  6. Type in the Command for the component.
  7. Fill in and set up any relevent details including:
    • Parameters (both the key and value will be exported as entered)
    • Conditions
    • Compute
    • Image
    • Description (edited by clicking the right gray panel)
  8. Click Publish as Library.
  9. In the confirmation screen that appears, click Save to complete the export.

cnvrg packages your task as a component and exports it to the AI Library for future use. You and your colleagues will now be able to access the new component in the AI Library page.

# Using the CLI

You can create a new component by using the cnvrg library import CLI command. The command should be run in a directory with the following contents:

  • File(s) containing the component code
  • A component YAML file named 'librar.yaml'
  • (Optionally) requirements.txt
  • (Optionally) prerun.sh
  • (Optionally) README.md

Use the command as follows:

$ cnvrg library import

This uses all the files in the directory that the command is run from to create a component in the AI Library of your organization.

# File(s) containing the component code

You can include one or more files with the actual code that makes up your AI Library component.

# Component YAML file

The YAML file for an AI Library component contains all the important setup information for the component. The file must be called library.yml and should be structured as follows:

---                 #Examples:
title:              #xgboost
command:            #python3 xgb.py
description:        #xgboost - AI library made by cnvrg.io data science team.
version:            #1.0.30
docker_images:      #cnvrg
compute:            #small
icon:               #xgboost
arguments:
    arg1:           #output_model:
        value:      #model.sav
    arg2:           #max_depth:
        value:      #value: 3

You can include as many arguments as you want.

WARNING

The following are reserved words and cannot be used as arguments in your library.yml:

command
arguments
grid
title
project
compute
datasets
local
library
working_directory
sync_before
sync_after
prerun
requirements
image

# (Optional) requirements.txt

If your component has specific packages that need to be installed for it to function, include a requirements.txt file in your directory.

When running the component, they will all be installed automatically.

# (Optional) prerun.sh

If you would like to include a prerun script for the component, include a prerun.sh file in your directory.

When running the component, the script will be run on start-up.

# (Optional) README.md

You can include a markdown file called README.md as part of your directory. The contents of this document will be rendered as the documentation for the component on its AI Library entry.

# Using the AI Library Page

AI Library search

All your components are kept within the AI Library Page. There, you can bookmark your most useful tasks and search through your library.

# Favorite a component

To set a component as a favorite, click the star on the left of the entry for the component. To unfavorite a component, click it again.

# Search through components

To search for specific components, click on the Search bar at the top of the AI Library page.

Type in the term you are searching for and as you type, the page wil automatically update live.

NOTE

The search function only searches the names of the components.

# Change the view

On the left-hand-side you can click to apply a filter for which components are displayed. Either:

  • All
  • Favorites
  • Popular
  • Recent

# Opening a component's entry

AI Library entry

You can inspect every component by clicking it on the AI Library page. Doing so will allow you to see information such as:

  • Documentation for the component:
    • The details of the component (produced by the README.md)
    • Metadata (default compute, Docker image, last updated, created by and monthly usage)
    • A code snippet for how t use the library with the cnvrg SDK
  • Source (YAML): The library.yml file for the component
  • Dependencies:
    • The requirements.txt
    • The Prerun.sh
Last Updated: 5/11/2020, 9:30:58 AM