# Architecture Overview

cnvrg comprises two main components:

  • Application layer: Manages the cnvrg back-end and front-end services, including the database, object storage, metadata services, and more.
  • Compute (workers) layer: This is where the machine learning jobs and workload are executed (experiments, model training, and so on).

This architecture design enables you to scale your machine learning resources quickly and easily.

cnvrg is built to be incredibly flexible and supports many different deployment environments for your cnvrg application layer. After you've deployed cnvrg you can add all of your different workers layers, no matter how they are hosted!

As compute resources exist separately to the Application layer, you can connect multiple different workers layers from all the different supported options. This allows you complete flexibility to leverage all your different infrastructure.

This allows you to use cnvrg on-premise, on the cloud, in multi-cloud setups and in hybrid on-premise + cloud setups.

The supported workers layers are:

  • On-premise: Kubernetes / OpenShift
  • Single-node machines
  • Spark clusters – YARN, Hadoop
  • Cloud Kubernetes clusters: AWS, Azure, GCP
  • Cloud Instances: AWS, Azure, GCP

This allows you to use cnvrg on-premise, on the cloud, in multi-cloud setups and in hybrid on-premise + cloud setups.

# Application Layer Requirements

To be able to run the Application layer, ensure you have the following base infrastructure requirements:

cnvrg Component CPU Memory Storage Description
cnvrg App 8 CPU 32 gb 150 gb cnvrg's main application handles all of cnvrg’s functionality including its unique data science management (projects, datasets, models, users, code, experiments, endpoints) along with the jobs orchestration framework.
cnvrg database 4 CPU 16 gb 80 gb cnvrg-db hosts the main database. It is based on PostgreSQL 11.3.
cnvrg ElasticSearch 4 CPU 16 gb 80 gb cnvrg's ElasticSearch server manages the datasets’ metadata and enables advanced querying capabilities on top of the hosted datasets.
cnvrg Object Storage 4 CPU 16 gb 80 gb cnvrg's object storage hosts the management and versioning of projects’ files, artifacts and datasets’ files. Storage size should be set with regards to datasets and codebase size (Recommended: 1TB). cnvrg's object storage server is based on MinIO - a cloud storage server compatible with Amazon S3, released under Apache License v2.
Last Updated: 3/18/2020, 1:30:24 PM