# AI Library
Building ML pipelines and continual learning are key to an effective machine learning workflow. Reproducible and modular code components are core components of any such workflow.
The AI Library in cnvrg facilitates these goals. It is a specially built package manager for ML components designed specifically for machine learning. It helps data scientists and developers build machine learning components and reuse them across projects.
cnvrg comes pre-configured with many AI Library components that you can start using straight away. You can also easily create your own. This allows you to share your work with your team and build a repository of your code and algorithms.
The topics in this page:
A component is a piece of reusable code. It can be used for any machine learning task including training, preprocessing, and more.
Each component comprises the following:
- Arguments and parameters
- A base image
- A compute template
- (Optionally) A description
- (Optionally) Documentation
A component can be created via the web UI or cnvrg CLI.
# Preinstalled Components
cnvrg comes with preinstalled components for many powerful machine learning preprocessing and training algorithms. Our data science experts are constantly improving the library with the latest, cutting-edge algorithms.
You can explore all the prebuilt components from the AI Library page. They are each comprehensively documented and all of the features and algorithms are fully described.
cnvrg includes components for K-nearest neighbors, linear regression, random forest classifier, XGBoost and more.
All prebuilt components come configured with auto-tracking and advanced visualizations such as:
- Confusion matrices
- Correlation matrices
- Loss and accuracy graphs
# Use a Component
You can use a component from the AI Library in two ways:
- As a task in a flow
- Using the cnvrg SDK
# As a task in a flow
To use the component as a task in a flow within one of your projects:
- Go to Projects > Flows.
- Open a new or existing flow.
- Click the New Task drop-down list on the top right and choose the component you would like to use.
- In the panel that appears, configure all the relevant details (parameters, compute, image and so on.)
- Click Save Changes.
The task is added to your flow.
Now you can link it up to other tasks, or click the Play button to execute the component.
If you click Custom Task, your recent AI Library components will appear on the Enter a command or search... panel. You can quickly click them there as well.
# Using the SDK
To run a component using the cnvrg SDK, you use the following format:
from cnvrg import Library library = Library('/cnvrg_libraries/library_name') experiment = library.run(arg1="val1",arg2="val2",...)
You can also run a grid search by using an array for an argument:
from cnvrg import Library library = Library('/cnvrg_libraries/library_name') experiment = library.run(arg=["val1","val2"])
More information on this SDK method can be found here.
# Add a Custom Component
You can turn any code you have written into a component and export it to the AI Library.
There are two ways to add a component to the AI Library:
- Using the web UI
- Using the cnvrg CLI
# Using the web UI
You can export your code as a component from any of your projects.
- Open the project that contains the code you want to export.
- Open the Flows section of the project.
- Create a new flow or open an existing one.
- Click the New Task drop-down list.
- Click Custom Task.
- Type in the Command for the component.
- Fill in and set up any relevent details including:
- Parameters (both the key and value will be exported as entered)
- Description (edited by clicking the right gray panel)
- Click Publish as Library.
- In the confirmation screen that appears, click Save to complete the export.
cnvrg packages your task as a component and exports it to the AI Library for future use. You and your colleagues will now be able to access the new component in the AI Library page.
# Using the CLI
You can create a new component by using the
cnvrg library import CLI command. The command should be run in a directory with the following contents:
- File(s) containing the component code
- A component YAML file named 'librar.yaml'
- (Optionally) requirements.txt
- (Optionally) prerun.sh
- (Optionally) README.md
Use the command as follows:
$ cnvrg library import
This uses all the files in the directory that the command is run from to create a component in the AI Library of your organization.
# File(s) containing the component code
You can include one or more files with the actual code that makes up your AI Library component.
# Component YAML file
The YAML file for an AI Library component contains all the important setup information for the component. The file must be called
library.yml and should be structured as follows:
--- #Examples: title: #xgboost command: #python3 xgb.py description: #xgboost - AI library made by cnvrg.io data science team. version: #1.0.30 docker_images: #cnvrg compute: #small icon: #xgboost arguments: arg1: #output_model: value: #model.sav arg2: #max_depth: value: #value: 3
You can include as many arguments as you want.
The following are reserved words and cannot be used as arguments in your
command arguments grid title project compute datasets local library working_directory sync_before sync_after prerun requirements image
# (Optional) requirements.txt
If your component has specific packages that need to be installed for it to function, include a
requirements.txt file in your directory.
When running the component, they will all be installed automatically.
# (Optional) prerun.sh
If you would like to include a prerun script for the component, include a
prerun.sh file in your directory.
When running the component, the script will be run on start-up.
# (Optional) README.md
You can include a markdown file called
README.md as part of your directory. The contents of this document will be rendered as the documentation for the component on its AI Library entry.
# Using the AI Library Page
All your components are kept within the AI Library Page. There, you can bookmark your most useful tasks and search through your library.
# Favorite a component
To set a component as a favorite, click the star on the left of the entry for the component. To unfavorite a component, click it again.
# Search through components
To search for specific components, click on the Search bar at the top of the AI Library page.
Type in the term you are searching for and as you type, the page wil automatically update live.
The search function only searches the names of the components.
# Change the view
On the left-hand-side you can click to apply a filter for which components are displayed. Either:
# Opening a component's entry
You can inspect every component by clicking it on the AI Library page. Doing so will allow you to see information such as:
- Documentation for the component:
- The details of the component (produced by the README.md)
- Metadata (default compute, Docker image, last updated, created by and monthly usage)
- A code snippet for how t use the library with the cnvrg SDK
- Source (YAML): The library.yml file for the component
- The requirements.txt
- The Prerun.sh