# AI Library
Continual learning and building machine learning (ML) pipelines are key to an effective ML workflow. Reproducible and modular code are core components of any such workflow.
The cnvrg AI Library facilitates these goals. It is a specially built package manager for ML components designed specifically for machine learning. It helps data scientists and developers build ML components and reuse them across projects.
cnvrg comes preconfigured with many AI Library components that you can immediately start using. You can also easily create your own. This allows you to share your work with your team and build a repository of your code and algorithms.
The topics in this page:
# Components
A component is a piece of reusable code. It can be used for any ML task, including training and preprocessing.
Each component comprises the following:
- Code
- Arguments and parameters
- A base image
- A compute template
- A description (Optional)
- Documentation (Optional)
A component can be created using the web UI or cnvrg CLI.
# Preinstalled Components
cnvrg comes with preinstalled components for many powerful ML preprocessing and training algorithms. The cnvrg data science experts are constantly improving the library with the latest, cutting-edge algorithms.
You can explore all the prebuilt components from the AI Library page. They are each comprehensively documented and all of the features and algorithms are fully described.
cnvrg includes components for K-nearest neighbors, linear regression, random forest classifier, XGBoost and more.
All prebuilt components come configured with auto-tracking and advanced visualizations such as:
- Confusion matrices
- Correlation matrices
- Loss and accuracy graphs
# Component Usage
You can use a component from the AI Library in two ways:
- As a task in a flow
- With the cnvrg SDK
# As a task in a flow
Complete the following steps to use the component as a task in a flow within a project:
- Go to Projects > Flows.
- Open a new or existing flow.
- Click the New Task drop-down list on the top right and choose the component to use.
- In the displayed panel, configure the relevant details, including parameters, compute, and image.
- Click Save Changes.
The task is added to your flow.
Now you can link it up to other tasks, or click the Play button to execute the component.
TIP
If you click Custom Task, your recent AI Library components display on the Enter a command or search... panel. You can quickly click them there as well.
# With the SDK
To run a component using the cnvrg SDK, use the following format:
from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg1="val1",arg2="val2",...)
To run a grid search, use an array for an argument:
from cnvrg import Library
library = Library('/cnvrg_libraries/library_name')
experiment = library.run(arg=["val1","val2"])
More information on this SDK method can be found here.
# Custom Component Additions
You can turn any written code into a component and export it to the AI Library.
There are two ways to add a component to the AI Library:
- Using the web UI
- Using the cnvrg CLI
# Using the web UI
Complete the following steps to export your code as a component from any of your projects:
- Open the project that contains the code to export.
- Open the Flows section of the project.
- Create a new flow or open an existing one.
- Click the New Task drop-down list.
- Click Custom Task.
- Enter the Command for the component.
- Provide any relevent details including:
- Parameters (both the key and value are exported as entered)
- Conditions
- Compute
- Image
- Description (edited by clicking the right gray panel)
- Click Publish as Library.
- In the displayed confirmation dialog, click Save to complete the export.
cnvrg packages your task as a component and exports it to the AI Library for future use. You and your colleagues can now access the new component in the AI Library page.
# Using the CLI
You can create a new component by using the cnvrg library import
CLI command. Run the command in a directory with the following contents:
- File(s) containing the component code
- A component YAML file named
librar.yaml
- requirements.txt (Optional)
- prerun.sh (Optional)
- README.md (Optional)
Use the command as follows:
$ cnvrg library import
This uses all the files in the directory from where the command is run to create a component in the AI Library of your organization.
# File(s) with the component code
You can include one or more files with the actual code that makes up your AI Library component.
# Component YAML file
The YAML file for an AI Library component contains all the important setup information for the component. Name the file library.yml
and structure it as follows:
--- #Examples:
title: #xgboost
command: #python3 xgb.py
description: #xgboost - AI library made by cnvrg.io data science team.
version: #1.0.30
docker_images: #cnvrg
compute: #small
icon: #xgboost
arguments:
arg1: #output_model:
value: #model.sav
arg2: #max_depth:
value: #value: 3
You can include as many arguments as desired.
WARNING
The following are reserved words and cannot be used as arguments in your library.yml
:
command
arguments
grid
title
project
compute
datasets
local
library
working_directory
sync_before
sync_after
prerun
requirements
image
# requirements.txt (Optional)
If your component has specific packages that need to be installed for it to function, include a requirements.txt
file in your directory.
When running the component, all requirements are installed automatically.
# prerun.sh (Optional)
To include a prerun script for the component, add a prerun.sh
file in your directory.
When running the component, the script runs on start-up.
# README.md (Optional)
You can include a markdown file called README.md
as part of your directory. The contents of this document are rendered as the documentation for the component on its AI Library entry.
# AI Library Page Usage
All your components reside within the AI Library Page. There, you can bookmark your most useful tasks and search through your library.
# Favorite a component
To set a component as a favorite, click the star on the left of the entry for the component. To unfavorite a component, click it again.
# Search through components
To search for specific components, click on the Search bar at the top of the AI Library page.
As you begin typing the component name you are searching, the page automatically updates with possible component matches.
# Change the view
On the left side, click one of the following filters to apply and display specific components:
- All
- Favorites
- Popular
- Recent
# Open a component's entry
You can inspect every component by clicking it on the AI Library page. Doing so allows you to see information such as:
- Documentation for the component:
- The details of the component (produced by the README.md)
- Metadata (default compute, Docker image, last updated, created by, and monthly usage)
- A code snippet regarding the library's usage with the cnvrg SDK
- Source (YAML): The library.yml file for the component
- Dependencies:
- The requirements.txt
- The Prerun.sh