# AI Library

AI Library Page

Continual learning and building machine learning (ML) pipelines are key to an effective ML workflow. Reproducible and modular code are core components of any such workflow.

The cnvrg AI Library facilitates these goals. It is a specially built package manager for ML components designed specifically for machine learning. It helps data scientists and developers build ML components and reuse them across projects.

cnvrg comes preconfigured with many AI Library components that you can immediately start using. You can also easily create your own. This allows you to share your work with your team and build a repository of your code and algorithms.

The topics in this page:

# Components

AI Library entry

A component is a piece of reusable code. It can be used for any ML task, including training and preprocessing.

Each component comprises the following:

  • Code
  • Arguments and parameters
  • A base image
  • A compute template
  • A description (Optional)
  • Documentation (Optional)

A component can be created using the web UI or cnvrg CLI.

# Preinstalled Components

cnvrg comes with preinstalled components for many powerful ML preprocessing and training algorithms. The cnvrg data science experts are constantly improving the library with the latest, cutting-edge algorithms.

You can explore all the prebuilt components from the AI Library page. They are each comprehensively documented and all of the features and algorithms are fully described.

cnvrg includes components for K-nearest neighbors, linear regression, random forest classifier, XGBoost and more.

All prebuilt components come configured with auto-tracking and advanced visualizations such as:

  • Confusion matrices
  • Correlation matrices
  • Loss and accuracy graphs

Experiment tracking

# Component Usage

You can use a component from the AI Library in two ways:

  • As a task in a flow
  • With the cnvrg SDK

# As a task in a flow

Choose a component

Complete the following steps to use the component as a task in a flow within a project:

  1. Go to Projects > Flows.
  2. Open a new or existing flow.
  3. Click the New Task drop-down list on the top right and choose the component to use.
  4. In the displayed panel, configure the relevant details, including parameters, compute, and image.
  5. Click Save Changes.

The task is added to your flow.

Now you can link it up to other tasks, or click the Play button to execute the component.

TIP

If you click Custom Task, your recent AI Library components display on the Enter a command or search... panel. You can quickly click them there as well.

# With the SDK

To run a component using the cnvrg SDK, use the following format:

from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg1="val1",arg2="val2",...)

To run a grid search, use an array for an argument:

from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg=["val1","val2"])

More information on this SDK method can be found here.

# Custom Component Additions

You can turn any written code into a component and export it to the AI Library.

There are two ways to add a component to the AI Library:

  • Using the web UI
  • Using the cnvrg CLI

# Using the web UI

Export a task to the library

Complete the following steps to export your code as a component from any of your projects:

  1. Open the project that contains the code to export.
  2. Open the Flows section of the project.
  3. Create a new flow or open an existing one.
  4. Click the New Task drop-down list.
  5. Click Custom Task.
  6. Enter the Command for the component.
  7. Provide any relevent details including:
    • Parameters (both the key and value are exported as entered)
    • Conditions
    • Compute
    • Image
    • Description (edited by clicking the right gray panel)
  8. Click Publish as Library.
  9. In the displayed confirmation dialog, click Save to complete the export.

cnvrg packages your task as a component and exports it to the AI Library for future use. You and your colleagues can now access the new component in the AI Library page.

# Using the CLI

You can create a new component by using the cnvrg library import CLI command. Run the command in a directory with the following contents:

  • File(s) containing the component code
  • A component YAML file named librar.yaml
  • requirements.txt (Optional)
  • prerun.sh (Optional)
  • README.md (Optional)

Use the command as follows:

$ cnvrg library import

This uses all the files in the directory from where the command is run to create a component in the AI Library of your organization.

# File(s) with the component code

You can include one or more files with the actual code that makes up your AI Library component.

# Component YAML file

The YAML file for an AI Library component contains all the important setup information for the component. Name the file library.yml and structure it as follows:

---                 #Examples:
title:              #xgboost
command:            #python3 xgb.py
description:        #xgboost - AI library made by cnvrg.io data science team.
version:            #1.0.30
docker_images:      #cnvrg
compute:            #small
icon:               #xgboost
arguments:
    arg1:           #output_model:
        value:      #model.sav
    arg2:           #max_depth:
        value:      #value: 3

You can include as many arguments as desired.

WARNING

The following are reserved words and cannot be used as arguments in your library.yml:

command
arguments
grid
title
project
compute
datasets
local
library
working_directory
sync_before
sync_after
prerun
requirements
image

# requirements.txt (Optional)

If your component has specific packages that need to be installed for it to function, include a requirements.txt file in your directory.

When running the component, all requirements are installed automatically.

# prerun.sh (Optional)

To include a prerun script for the component, add a prerun.sh file in your directory.

When running the component, the script runs on start-up.

# README.md (Optional)

You can include a markdown file called README.md as part of your directory. The contents of this document are rendered as the documentation for the component on its AI Library entry.

# AI Library Page Usage

AI Library search

All your components reside within the AI Library Page. There, you can bookmark your most useful tasks and search through your library.

# Favorite a component

To set a component as a favorite, click the star on the left of the entry for the component. To unfavorite a component, click it again.

# Search through components

To search for specific components, click on the Search bar at the top of the AI Library page.

As you begin typing the component name you are searching, the page automatically updates with possible component matches.

# Change the view

On the left side, click one of the following filters to apply and display specific components:

  • All
  • Favorites
  • Popular
  • Recent

# Open a component's entry

AI Library entry

You can inspect every component by clicking it on the AI Library page. Doing so allows you to see information such as:

  • Documentation for the component:
    • The details of the component (produced by the README.md)
    • Metadata (default compute, Docker image, last updated, created by, and monthly usage)
    • A code snippet regarding the library's usage with the cnvrg SDK
  • Source (YAML): The library.yml file for the component
  • Dependencies:
    • The requirements.txt
    • The Prerun.sh
Last Updated: 11/30/2022, 6:32:43 PM