# Release Notes
# Version v5.2.1 (Oct 15th 2024)
- Improvement: Query metadata of dataset SDK V2 - the get_metadata now returns metadata for multiple specified files, including handling missing or non-existent files by returning empty objects. (23335)
# Version v5.2.0 (Oct 15th 2024)
New Upgrade - Rails 7
- This release requires running the following SQL command for every DB instance:
CREATE INDEX CONCURRENTLY idx_commit_to_blob_versions_commit ON commit_to_blob_versions(commit_id, commit_type, blob_v_type); - Note: It may take a few hours to complete on large databases. For more information, please contact support.
# Version v5.1.1 (Oct 1st 2024)
- Improvement: Compute template - Update kube template with custom resources. (23845)
- Fix: Gaudi jobs won't run without "runtimeClassName: habana" in the pod spec. (23921)
- Fix: Experiments - Download button is not working for multiple files in input and output artifacts. (23801)
- Fix: Experiments - GPU utilization not showing in experiment metrics. (23781)
# Version v5.1.0 (Sep 25th 2024)
- New Feature: Datasource - A datasource is a data repository or storage location that can be connected for importing, exporting, or processing data. The platform supports various types of datasources, including cloud storage solutions such as Amazon S3 and MinIO buckets, which can be integrated and used within workspaces and experiments. (21895)
# Version v5.0.12 (Sep 19th 2024)
- New Feature: Serving - LLM TGI Endpoint - Deploy large language models (LLMs) quickly using a ready-to-use endpoint. Select a pre-defined configuration or customize as needed, with minimal setup. The deployment process is automated, providing an API endpoint ready for application integration. After deployment, you can connect your endpoint to Gradio or Streamlit using the Generic App (22902)
# Version v5.0.11 (Sep 18th 2024)
- New Feature: Add filtering by label - Users can now filter the dashboard, datasets, and projects by tags, allowing for quick access to jobs created with specific labels. (17328)
# Version v5.0.10 (Sep 16th 2024)
- Improvement: Dashboard - Improved clarity for inactive Kubernetes nodes. (22679)
- Fix: SDKv2 Datasets clone fails if query_slug using tags contains more than 1000 files in an experiment. (23680)
# Version v5.0.9 (Sep 15th 2024)
- New Feature: Compute Templates - SecurityContext - Defines security configurations, allowing admins to set capabilities (e.g., IPC_LOCK, NET_RAW) for container interactions to ensure secure execution. (22677)
- New Feature: Compute Templates - Custom Resources - Utilize specific capabilities in the platform, and deployments must specify custom resources in the pod configuration. This is done by defining resource requests under the resources section. (22359)
# Version v5.0.8 (Sep 11th 2024)
- New Branding Announcement - Introducing Intel Tiber AI Studio (formerly cnvrg.io) Our platform has rebranded to Intel Tiber AI Studio, reflecting our commitment to advanced AI and MLOps solutions under Intel.
# Version v5.0.6 (Aug 29th 2024)
- Fix: SDK v2, Experiment from experiments.get, is not the same as experiments.list (23880)
- Fix: SDK v2, when starting an Experiment and then stopping it using the Experiment.finish() command, the Experiment timer keeps running.(23193)
# Version v5.0.4 (Aug 4th 2024)
- New Feature: Project Template - Admins can now create templates from existing projects, allowing them to clone files/Git path, settings, and flows. These templates can be applied across the entire organization, streamlining project setup, simplifying onboarding, and ensuring consistent standards across all initiatives (17688)
# Version v5.0.3 (July 29th 2024)
- New Feature: Generic App - Connect and deploy any type of application within the cnvrg platform, leveraging various technologies and frameworks that best suit your project needs. Start using popular frameworks like Gradio and Streamlit with minimal setup. Create interactive applications for data science and machine learning projects easily. By selecting these examples and clicking apply, the example app will work out of the box. These frameworks were added to support Generative AI use cases (22177)
# Version v5.0.2 (July 25th 2024)
- New Feature: Labels - Ability to create labels at the organization level and apply these labels to project and dataset cards. Assign specific labels to streamline the identification and management of various projects and datasets (17122)
# Version v5.0.1 (July 22nd 2024)
- New Feature: Filters - Adding filters to the Datasets Dashboard and Projects pages. Ability to save customized filters for repeated use, enhancing efficiency and personalization in data management (22254)
- Fix: Cannot logout on SSO-enabled environment with Azure IdP (22060)
- Fix: Wrong API_TOKEN causes failure to authenticate using ENV (22586)
- Fix: User mismatch on workspace restart (22531)
# Version v5.0 (July 22nd 2024)
New Slim Operator - Enhanced Security, Operational Efficiency, Scalability, and Infrastructure Independence
- Enhanced Security: Integration of job protection through SSO and transition to JWT token-based authentication across all components, providing a more secure and standardized authentication mechanism.
- Operational Efficiency and Stability: Centralized SSO solution and reduced load on logging & metrics databases (ElasticSearch, Prometheus) decrease operational overhead and improve system stability.
- Scalability and Flexibility: Support for multiple platform installations within the same cluster and optional high-level (admin) cluster permissions, offering greater flexibility in deployment and adaptability to complex scenarios.
- Infrastructure Independence: Agnostic to virtualization infrastructure (OpenShift, Kubernetes), networking (Istio), and monitoring (Prometheus), empowering users to utilize their existing infrastructure, simplifying deployment.
- Support Kubernetes versions 1.25-1.30.
- Generic support for every Oauth2 (OIDC) SSO vendor.
- For more information, please contact support.
# Version v4.8.8 (May 23th 2024)
- New Feature: Experiments - End sync: A default behavior that syncs the experiment when it finishes running without exceptions
- Improvement: Datasets - DATA_COMMIT_LIMIT - Positive integer (e.g., 2): Limits the number of latest commits returned per dataset to the specified value. (Enabled by a feature flag)
- Improvement: Dashboard - Adding an indication for Cordoned Kubernetes nodes in the cnvrg Dashboard
- Fix: Datasets - Datasetwarmup creation loop when an auto-clear cached commit is enabled
- Fix: Dashboard - Wrong allocated GPU counter when specific node selected
- Fix: Flows - Invalid Hyperparameter Format, the format “{ { Params.route | hyper } }” is invalid due to the spaces
- Fix: Experiments - Experiments stuck on Finalizing status
- Fix: Experiments - During an experiment with numerous logs to be parsed into tags, the ExtractAutoTagsJob stops extracting randomly after a while
- Fix: Experiments - Jobs randomly miss data points and insert values out of order
- Fix: Logging in with one or more capital letters in the email address results in an error
# Version v4.8.7 (May 12th 2024)
- Fix: Workspace - Users can view a list of only the 20 most recent volumes when creating a new job
- Fix: Flows - Decimal value display issue in objective goal
- Fix: FluentBit "failed to parse" errors
- Fix: Removing un-used Postgres records to save space (airgap)
# Version v4.8.6 (May 9th 2024)
- New Feature: Serving - Support file input with stream output for web service endpoints
# Version v4.8.5 (April 30th 2024)
- Improvement: Workspace - Enable dataset removal from workspace using the UI
- Fix: The cnvrgv2 query returns duplicate results and fails to return all results if there are more than 20
# Version v4.8.4 (April 18th 2024)
- Fix: Volumes - Filter for claim name on PVC table is not working
# Version v4.8.3 (April 14th 2024)
- Improvement: Cli V2 release
# Version v4.8.2 (April 1st 2024)
- Fixed: Projects - Alignment issue with Create Project pop-up
- Fixed: Workspace - Blank page displayed on UI when the workspace is 'Offline'
# Version v4.8.1 (March 24th 2024)
- Improvement: Volumes - Display capacity and PVC claim name in volumes table
- Fix: SDK v2 - Dataset put 'force' option doesn't work as expected (should create new commit containing only the files being uploaded in the current upload)
- Fix: Example Projects - Use first storage always, even when it is 'Offline'
# Version v4.8.0 (March 19th 2024)
New Feature: Agent - Improve jobs scalability, stability, speed and visibility
- Increased stability of jobs when running at scale (Long-running commands, end sync, SDK commands)
- Increased speed in uploading logs & executing multiple experiments, up to 80% faster
- Transition to cnvrgv2 CLI to perform job commands behind the scenes (replacing ruby cli)
- Better visualization of job lifecycle & improved visibility to logs and errors. Recovery from command failure, including logging of stack trace
- Smart disk management for cloning datasets using SDK/CLI - Download will not execute if there is insufficient storage space
- Fix known issue: Image build feature from the ‘Containers’ tab (and removal of build image from workspace)
For more information, please contact support.
# Version v4.7.116 (March 3rd 2024)
- Fix: Example Projects - Endpoint in 'lending club' example project is failing
# SDK Version 1.0.48 (Feb 20th 2024)
- Improvement: Add option to check version via CLI command (cnvrgv2 -v / cnvrgv2 --version)
- Improvement: Allow the option to adjust the threads and chunks while uploading files
# Version v4.7.115 (Feb 20th 2024)
- Fix: Workspaces - iframe doesn't load for certain browsers
- Fix: Compute - allow for selecting master specifications for 'PyTorch Distributed' templates
# Operator Version 4.3.36
- Improvement: Support Kubernetes versions 1.25-1.28 (using cnvrg-operator version 4.3.36)
# Version v4.7.114
- Improvement: Allow for dataset indexing to save the latest commit only to reduce load (enabled upon request)
# Version v4.7.113
- Fix: VScode Workspace - attempts to connect to external server times out when firewall is enabled
- Fix: CVAT Workspace - not supported when proxy is enabled
All subsequent versions apply exclusively to cnvrg Classic
# Version v4.7.112
- Improvement: Send email notification on experiment failure upon entering debug mode instead of when it ends with an error
# Version v4.7.111
- Improvement: Workspaces - Upgrade CVAT version to 2.2.0
- Fix: Endpoints - version rollout doesn't go to the current version
# Version v4.7.110
- Fix: CLI v2 - Unhandled exception for
experiment run --volume
when the volume is already in use - Fix: Endpoints - R endpoints raise error when file input is enabled
# Version v4.7.109
- Improvement: Support Gaudi2 resources
# Version v4.7.108
- Improvement: Notify user when Chrome browser is unable to load workspace iframe (and direct to open in new tab)
# Version v4.7.107
- Improvement: SDK v2 - Show names of folders that failed naming validation on upload
- Improvement: Support Office365 exchange server mail relay as SMTP server
- Improvement: Support custom internal cluster domain
- Fix: SDK v2 - "start ssh" command returns an error when using in git connected projects
- Fix: Endpoints - Updating endpoint with custom git branch via flow deploy task changes git branch to be master
- Fix: Sign up - Allow creation of user with above 30 characters in their email address (SSO/LDAP/MetaCloud environments)
# Version v4.7.106
- New Feature: Workspace Services - Enable 3rd party services and access their UI within jupyter workspaces (such as Matlab, Nsight and more) Note: Enabled upon request, offline documentation is available
# Version v4.7.102
- Fix: Git integrations - project's bitbucket integration fails when set using the connect git wizard
# Version v4.7.101
- Fix: Organization settings - User can't change organization settings despite being an admin
- Fix: Exporting csv through experiment index page and dashboard showing 500 error
# Version v4.7.99
- Fix: Experiments- The experiments do not appear in the experiments tab after git repository is connected to the project
- Fix: Operator/App - Missing TLS section for ingress objects
- Fix: Experiments table - Hyperlinks for start and end commit are removed after the deploy task is ran
# Version v4.7.97
- Fix: Custom PyPi - create a single unified pip.conf
- Fix: Experiment - system tab does not resolve and returns 500 error
# Version v4.7.95
- Improvement: SDK v2 - Add file's url to the "list_file" command, that can be used to download specific file from project/dataset
# Version v4.7.89
- Fix: Compute template - Users list doesn't load for "Private" template
- Fix: Experiment - git commit url does not resolve correctly
# SDK Version 1.0.39
- Fix: Datasets - cached commit does not contain all files
# Version v4.7.87
- Fix: Experiments - page UI - Trim long experiment title to fit table view
# Version v4.7.85
- Improvement: Support Jupyter workspace natively for NVIDIA images above v22.04
# Version v4.7.83
- Improvement: Release of cnvrg:v6 image to be used in jobs (lightweight image based on python3.10 with cnvrgv2 cli)
# Version v4.7.82
- Fix: Billing - Page coming blank after billing plan update
# Version v4.7.81
- Fix: Experiments table - On running Hyper Experiments experiments table displayes wrong filter (undefined grid id).
# Version v4.7.79
- Fix: Slow load in some pages for air-gapped environments
# Version v4.7.78
- Fix: Dataset Caching - uncache commit fails
# Version v4.7.75
- Fix: Experiment - logging millions of chart metrics can cause experiment page not to load
# Version v4.7.73
- Fix: Containers - can't edit an image that was built in the environment
# Version v4.7.72
- Improvement: New view for organization settings with access to "billing" and "system" pages
# Version v4.7.69
- New Feature: CVAT - add jupyterlab UI to the CVAT workspace
- Fix: nginx ingress migration from v1beta1 to v1
# Version v4.7.67
- New Feature - Spark Configurations - allow for customizable spark execution by passing different spark configurations to the compute template
# Version v4.7.66
- Fix: Flows - some libreries are not able to load in the "New Task" selector
# Version v4.7.65
- Fix: Team - Can't invite team member through the onboarding process
- Fix: Endpoints - can't provide file input in "try it live"
- Fix: Dataset caching - auto-clear feature runs according to "creation time" instead of "last used"
- Fix: Experiment page - "Linked to git" button leads to the main branch instead of designated branch
# Version v4.7.64
- New Feature: Connect to 'Seagate' storage in Metacloud
# Version v4.7.62
- Fix: Compute Resource - 'Created by' is displaying as false
# Version v4.7.61
- Fix: Dashboard - value in "Node" column disappears when job ends
- Fix: Papers - Experiment Compare does not show parameters that have an underscore in their name
# Version v4.7.60
- Fix: Flow: can't search for blueprint library in 'New Task' drop down
- Fix: Don't allow deleting a cluster when volumes are being used
- Fix: Can't run AI Libraries using SDK
- Fix: Dashboard - Experiments deleted while running show up as "active" in the dashboard
# Version v4.7.58
- Fix: Flow - next run shows in UTC instead of the timezone it was scheduled on (based on user's account settings)
- Fix: Project - improve search performence on projects page
- Improvement: Papers - allow for comparing up to 100 experiments at a time
# Version v4.7.55
- Improvement: add numbering differentiation on hyperparam experimens created by flow tasks
- Fix: Endpoints - can't display pdf files in try live section
- Fix: Dashboard Export - CSV link produces JSON data
# Version v4.7.53
- New Feature: SDK - add the option to manage organization's members
- Improvement: new view for "Teams" page
# Version v4.7.49
- Fix: Papers - allow viewing more then 5 experiments at a time in the "add experiments" tab
- Fix: Papers - Charts are getting automatically downloaded after downloading once
# Version v4.7.48
- Fix: Dataset Caching - Can't install cnvrg sdk from private PYPI repository
# Version v4.7.47
- Fix: Containers - Can't pull image when default compute template is missing
# Version v4.7.46
- Fix: Dataset - Can't clear cached commit
- Fix: Workspace - Metrics aren't displayed in Modin/Ray workspace
- Fix: Resource - labels displayed wrong in the status page
# Version v4.7.45
- Fix: Workspace - Tensorboard process doesn't re-launch in case of workspace restarts
# Version v4.7.44
- Fix: Dataset - Filepath query includes every file that ends with the provided text(regex: "*Text"), not only the specified file
# Version v4.7.43
- Improvement: AWS resource - Show availability zones by instance type and region
- Fix: SDK - can't pull artifacts from aborted tasks using SDK
- Fix: Experiments - End sync gets aborted before it finishes
- Fix: Workspace - Resume Jupyter workspaces when they get killed due to oom
- Fix: Bluprints - Input example is not working
- Fix: MetaCloud organizations can't use AI Libraries
- Fix: Dashboard - Filter by Active Status from top bar of dashboard doesn't return correct results
# Version v4.7.41
- Fix: Experiments - ray images are not displayed for jobs
# Version v4.7.40
- Improvement: SDK - Add the option to create registries
# Version v4.7.39
- Improvement: Define custome prometheus URL in worker cluster (built-in monitoring stack)
- Fix: SDK - Cached commit can be deleted using SDK when it is in use
- Fix: Experiment - Objective_reached tag is missing when running Hyper Optimization
# Version v4.7.36
- New Feature: Compute Resource - Change the text to "bring your own kubernetes" on on premise resources
- Fix: AI Libraries - yaml files are corrupted
# Version v4.7.32
- New Feature: SDK - add the option to create images
- Fix: SDK - Can't update default image in project's settings
- Fix: Containers - Disable the option to build image until a registry is added
- Fix: Endpoint - Endpoint keeps exporteing data after disabling Feedback loop
- Fix: Project - Loading project page stalls when it has a lot of activities
- Fix: Endpoint - Model upate isn't working for Endpoint which are created through SDK
- Fix: Experiment Table - Dictionary values in exported CSV files is against CSV format
- Fix: Endpoint - Stop and rollback isn't working in batch endpoint in case of canary rollout
# Version v4.7.31
- New Feature: Compute Resource - Support eastwest integration for clusters
- Fix: Experiments - Experiments remains in debug status after the debug time is completed when running many jobs
- Fix: Workspace - Workspace isn't shut off on idle time (gpu metrics were not transmitted)
# Version v4.7.29
- Fix: Experiment - "system" tab doesn't show HPU utilization metric in distributed jobs
- Fix: Experiment - "system" tab charts show unix time instead of human time on X-axis
# Version v4.7.27
- Fix: Blueprints - Inference blueprints displays errors when chosen compute is not on the default cluster
# Version v4.7.26
- Fix: General - User settings are not functioning in air-gapped environments
- Fix: General - Fix cnvrg logo
- Fix: Experiments - Allow managers to approve Human in the loop tasks
# Version v4.7.23
- New Feature: SDK - Add caching functionality
# Version v4.7.22
- New Feature: Git wizard - A new walkthrough connecting Git repositories to cnvrg
- New Feature: Project - Support Bitbucket token on git integration
- Fix: Flow - Custom tasks get the same default title ("Task 1")
- Fix: Experiments - Stop button is disable for experiments that are in 'pending approval' status
# Version v4.7.20
- Improvement: Compute Resource - Add terms and usage for partners compute
# Version v4.7.19
- Improvement: Projects index page - Improve load time
- Fix: Blueprints - Typo in Training flow welcome popup
- Fix: all "max duration reached" messages refer to endpoint
- Fix: Experiment table - Filtering by flow or flow version shows unrelated experiments
- Fix: SDK - Error when logging a tag with float or integer value
- Fix: Workspace - creating image from workspace text is white over white background
- Fix: Papers - Papers section can not be accessed after manipulating the chart
- Fix: Workspace - Tap on commit sha1 from workspace configuration tab redirects to same workspace in new tab
- Fix: SDK - Can't update max_restarts value on project's settings
# Version v4.7.18
- Improvement: SDK - Add the option to create user and membership
# Version v4.7.17
- Fix: Compute Resource - Delete resource button is diabled while there are running jobs on. instead it should pop-up message to confirm deletion
- Fix: Papers - Text is disappearing after opening the same paper url in another tab
# Version v4.7.15
- Fix: Flow - User list is not returning on human validation condition
# Version v4.7.14
- New Feature: SDK - Support e.pull_artifacts(wait_until_success=True)
- Improvement: Compute templates - Add the option to increase pod shared memory
- Fix: Experiment - Objective_reached tag is missing when running Hyper Optimization
- Fix: Compute Resource - Text typo on Min Workers field when Autoscaling is on
# Version v4.7.11
- New Feature: SDK - Create cache link functionality
- New Feature: SDK - add the option to "merge commit to master" via sdk2
- Fix: SDK - dataset clone(),download() ignores empty folders
# Version v4.7.10
- Fix: Dashboard - Can't filter by 'Online' status
# Version v4.7.8
- Improvement: Restore cnvrg logo when pride month is finished (but will continue to be in our hearts)
# Version v4.7.7
- Improvement: Endpoints - Add a warning when deleting a deploy task from the experiments table
- Improvement: Support organization names that are case-sensitive
- Fix: Compute - Fix nodegroup tags/taints/labels key value display in AWS resources
- Fix: Compute Dashboard - Pending mode jobs are not filtered as active jobs
- Fix: Endpoints - Endpoint type does not change after updating version
# Version v4.7.6
- Improvement: Compute - Support adding shared memory (/dev/shm) to jobs
# Version v4.7.5
- Improvement: SDK - Add an option to add users to the organization
# Version v4.7.4
- New Feature: Support Partners compute integration to cnvrg Metacloud
- Improvement: Endpoint - Add an option to send files in "Try it Live" section
# Version v4.7.3
- Fix: Compute - Compute type is displayed as "other" for default resource
- Fix: Workspaces - Artifacts font size is very small
- Fix: Compute Dashboard - Table is missing padding
# Version v4.7.2
- New Feature: Support Partners compute integration to cnvrg Metacloud
# Version v4.7.1
- New Feature: Support Partners compute integration to cnvrg Metacloud
- Fix: Experiments - "Stop Flow" button is missing in the flow diagram
# Version v4.7.0
- New Feature: Bring Your Own Storage: GCS (Available in cnvrg Metacloud)
- Fix: In case of uppercase letters Flows creates duplicate endpoints with the same name, url doesn't work
- Fix: SDK - Can't upload files to public datasets
# Version v4.6.0
- New Feature: Bring Your Own Storage: Azure Containers (Available in cnvrg Metacloud)
- New Feature: MetaGPU release: Allow sharing gpu between gpus in the same job
# Version v4.5.0
- New Feature: Papers Release: A new space to analyze and share your work
# Version v4.4.0
- New Feature: Blueprints Release: Open-source pre-configured ML pipelines that can be quickly integrated with any application (Available in cnvrg Metacloud)
- New Feature: Endpoints - Add an option to send request to endpoint from the UI
# Version v4.3.0
New Feature: Billing and plans integration (Available in cnvrg Metacloud)
# Version v4.2.0
- New Feature: Compute - Support Partners compute integration to cnvrg metacloud
# Version v4.1.0
- New Feature: Metacloud Release
- New Feature: Bring Your Own Compute: AWS EKS (Available in cnvrg Metacloud)
- New Feature: Bring Your Own Storage: AWS S3 (Available in cnvrg Metacloud)
- Improvement: Compute - Add Hugepages option to compute templates
- Fix: User Settings - Not able to add user profile image from account settings
# Version v4.0.0
- New Feature: Metacloud Release
- New Feature: Bring Your Own Compute: AWS EKS
- New Feature: Bring Your Own Storage: AWS S3
# Version v3.12.26
- Improvement: Compute - Change default Hugepages in Gaudi templates to 1000Mi
- Fix: Endpoints - Endpoint should have "Online" status when its running
# Version v3.12.25
- Fix: Experiments - Export experiments results with an error
- Fix: Experiments - Debug mode buttons are not working
# Version v3.12.24
- Fix: Experiments - When running pytorch distributed jobs end sync gets aborted
# Version v3.12.23
- Fix: Experiments - "Periodic sync" is not working
# Version v3.12.22
- Fix: Experiments - When using "Last 7 days" filter it doesn't show results
- Fix: Flows - Max duration doesn't work
- Fix: Experiments - filter for: "lt,gt,=" doesn't work
- Fix: Dataset - "active N hours/weeks/months ago" doesn't change
- Fix: Endpoints - Wrong compute value is showing during stop and rollback in endpoint
# Version v3.12.21
- Fix: Experiments - Can't run pytorch distributed jobs
# Version v3.12.20
- Fix: Compute Dashboard - Node name is missing for jobs running on on-premise resources
- Fix: Workspaces - "Restart when idle" is not working when using gpu
# Version v3.12.19
- Fix: Experiments - Can't filter experiments by tags that have a dot in the tag name
- Fix: Experiments - Running multiple experiments is slow when you have a project with 10k+ experiments
- Fix: Flows - Deploy task is stuck on "Initializing" state when trying to update an existing endpoint
# Version v3.12.18
- Fix: Experiments - Downloading artifacts as a zip file ignores file's paths
- Fix: Workspaces - Fix permissions issue for users when running CVAT workspaces
# Version v3.12.17
- Improvement: Endpoints - Include port in the request example for python
- Fix: SDK - Uploading files from the sdk while having an empty idx file causes files to not show up in the UI
- Fix: Project Settings - Update environment variables sometimes don't work
- Fix: Compute - Taints aren't passed to the pods when running pytorch distributed templates
# Version v3.12.16
- Fix: Projects - Activity feed doesn't show latest feed
# Version v3.12.15
- Improvement: Endpoints - Increase mx duration time to be 60 minutes
- Improvement: Projects - Improve projects home page loading time
- Fix: Experiments table - tag values for custom columns not displayed in UI
# Version v3.12.14
- Fix: Compute - Can't build image on newly added compute resources
- Fix: Experiments - Custom columns are not displayed in the UI
# Version v3.12.13
- Fix: SDK - Restart webapps command doesn't work
- Fix: Experiments - Chart doesn't load when using an uppercase in the title of the chart
- Fix: Compute Dashboard - Metrics are not populated when using internal prometheus
# Version v3.12.12
- Improvement: Flows - Continual learning trigger for updated datasets takes latest commit by default
- Fix: Flows - Flow which includes only deploy tasks transitions to an empty experiment's page
- Fix: Workspaces - Search for datasets in a running workspace doesn't show all relevant results
# Version v3.12.11
- Fix: SDK - Can't clone specific project's commit
# Version v3.12.10
- Improvement: Images - Support dynamic spark workers images
# Version v3.12.9
- Fix: Flows - Running flow with a single deploy task transitions to an empty experiments page
- Fix: Experiments - Endpoint task is displayed before the rest of the tasks, even though it was the last task to run
- Fix: Workspaces - When adding datasets to a workspace, not all datasets are searchable or visible in the drop down
- Fix: Endpoints - Continual learning alerts aren't triggered when condition is met
# Version v3.12.8
- Improvement: System - Support parameter based prefixes for cnvrg subdomains when installing cnvrg
# Version v3.12.7
- Fix: CLI - Running experiments with the --wait flag doesn't output logs to the terminal
# Version v3.12.6
- Improvement: SDK - Add or delete secrets using the sdk
- Improvement: Experiments - Support integer and float params when running Bayesian Optimization
- Fix: SDK: Date is incorrect when updating secrets through the SDK
- Fix: SDK - e.log_param() doesn't support boolean values, and could cause a flow to be stuck
- Fix: Workspaces - Deleting a workspace will delete commits related to that workspace
# Version v3.12.5
- Fix: Flows - Time is missing from UI when scheduling a flow
# Version v3.12.4
- Improvement: SDK - Add an option to clone a specific project's commit
- Fix: Webapps - When SSO is enabled, webapp UI is not visible
- Fix: User Setting - changing data in user page has no validators
- Fix: Flows - Support clock changes for scheduled flows (Daylight Saving Time)
# Version v3.12.3
- Improvement: System - Allow changing user address for emails sent by cnvrg system
# Version v3.12.2
- Fix: Workspaces - Allow workspaces to start from experiments commit and sync changes to the project
# Version v3.12.1
- Fix: Workspaces - "max duration exceeded" status remains after starting and stopping workspaces
# Version v3.12.0
- New Feature: Endpoints - Added integration for Openvino model server
# Version v3.11.14
- New Feature: Project Settings - Enable "local folder" to be visible only to admin ( Enabled by a feature flag)
- Improvement: Organization Settings - Improve creation of private pypi conf for all jobs
- Fix: Experiments - When syncing from sdk with local experiments, artifacts tab shows files from previous sync command
# Version v3.11.13
- Fix: Project Home page - User's icons are not align when one of the users has image
# Version v3.11.12
- Fix: Experiments - "copy command" does not include notify on error/success with emails
- Fix:Workspaces - Workspaces are stuck in a restarting loop when using environment variables that include space in its key
# Version v3.11.11
- Improvement: Add option to add preconfigured registry and images when installing cnvrg
- Fix: Users - Forget password page is missing the button to reset password
- Fix: Experiments - "cnvrg_auth" command gets stuck in initializing stage
# Version v3.11.10
- Fix: Workspaces - cvat tab is not clickable
- Fix: Experiments - Experiment sometime is stuck on "terminating" after end commit
# Version v3.11.9
- Improvement: Dashboard - Improve metrics for organization's dashboard
# Version v3.11.8
- Improvement: Workspaces - Update install commands for Vscode to support version v4.0.1+
# Version v3.11.7
- Fix: Workspaces - webapps can't run from inside the workspace
- Fix: SDK - Allow updating image from project settings
- Fix: Experiments - Memory metrics can exceed 100%
# Version v3.11.6
- Fix: Experiments - Load config table with empty parameters
# Version v3.11.5
- Improvement: Experiments - Faster loading of hyper parameters experiments in the experiments' table
# Version v3.11.4
- Improvement: Endpoints - Support endpoint with python 2.7
- Fix: Experiments - Wrong values when running grid search with scale log2
- Fix: Experiments - Deleting of experiment popup has wrong description
- Fix: Experiments - Wrong values when running grid search with scale log10 and floats
- Fix: Workspaces - Workspace is sometime stuck on "stopping"
- Fix: Datasets - Fix creating file from UI in a folder with + sign creates 2 folders
# Version v3.11.3
- Fix: Existing volumes are not visible in the Volume drop down
- Fix: Drop down is not available in Human Validation in flows
# Version v3.11.2
- Improvement: Flow - Allow task parameters to contain other task's variable and string to transfer dynamic parameters between tasks
- Fix: Endpoint - Continual learning alerts aren't triggered when condition is met
- Fix: Experiment - Can't view last few charts (out of 25) in "visual" tab
- Fix: Flow - Creating task in a flow with title containing dot can break the flow
- Fix: Dashboard - "no data" when super-admin from 2nd org views dashboard in 1st org
# Version v3.11.1
- Improvement: Workspace - Sync automatically master and workers files when using OpenMPI
- Improvement: Add log_param functionality to experiment object
- Fix: Endpoint - Add Rabbitmq option to to the UI
- Fix: Deprecated jobs cab get stuck on GPU
- Fix: Project - Can't add administrator to project's collaborators
- Fix: Remove commits request from files page
- Fix: Changing the 'Working Directory' in project settings doesn't affect flow tasks
- Fix: Containers - "Build Image" job ignores template selected and takes default system-template instead
# Version v3.11.0
- New Feature: Refactor Experiments table page - Improve loading time and usability
# Version v3.10.15
- Improvement: Internal - Update release tags in cnvrg releases
# Version v3.10.14
- Fix: Double v in cnvrg version's icon
# Version v3.10.13
- Improvement: Change pip.conf creation from "append" to "overwrite"
# Version v3.10.12
- Improvement: VScode workspace - install jupyter extension manually
# Version v3.10.11
- Fix: WebApps - voila app strip-sources isn't visible in UI
# Version v3.10.10
- Fix: Workspace - commits with large amount of files can cause slow load times for workspace page
# Version v3.10.9
- New Feature: Experiment - Support HTML format (line breaks,bold,etc) for alerts messages
# Version v3.10.8
- Fix: Verify experiments exit status is updated correctly in load
# Version v3.10.7
- Fix: Delete flow url is shown wrong
# Version v3.10.6
- Fix: hpu is not shown in the templates list as a description ( also for mpi )
# Version v3.10.5
- Improvement: SDK: Deleting flow having ongoing task using flows.delete() not deleting the task
- Fix: Flow - timezone displayed in the 'schedule' tab is utc
- Fix: SDK - e.alert not working, 404 error
- Fix: Experiment - Printing massive amount of logs can cause idle time
# Version v3.10.4
- Fix: Add liveness checks to webapps
# Version v3.10.3
- Fix: "Tensorflow" endpoints can not be accessed from Serving Tab.
# Version v3.10.2
- Fix: Add support for region when setting up an on premise bucket
# Version v3.10.1
- New Feature: Add support for triton endpoints
# Version v3.10.0
- New Feature: Add Queues and jobs priority - Add the ability to prioritize jobs and preempt others
- New Feature: Add option to hard delete artifacts, projects and datasets
- Improvement: Add auto update to all tables
# Version v3.9.75
- New Feature: Refactor Dataset selector
- Improvement: Experiment Show - Add timestamp to Artifacts/Commits list
- Improvement: Add filter to volumes list
- Fix: Allocated cpu and memory are showing wrong in Dashboard when cpu and memory was edited in the template
- Fix: Table: Sorting for project field in dashboard and volumes table is working in reverse order
- Fix: Add loader in all tables to when switching states / filter is loading / refreshing page
- Fix: Type is showing empty for default cluster on list page
# Version v3.9.74
- Fix: Auth - cant login with sso, blank page.
# Version v3.9.73
- Fix: Remote SSH - global cluster-scope environment variables not available in remote ssh terminal
# Version v3.9.72
- Fix: Template/Caching - Cant cache: System Template not created for a newly/old Cluster.
# Version v3.9.71
- Fix: Experiment - using print("cnvrg_tag_:value) only extract small amount of the metrics
# Version v3.9.70
- Fix: sdk - Experiment.list() returns error
# Version v3.9.69
- Fix: Flow - duplicate tasks when running webhook/flow_version_runner
# Version v3.9.68
- Fix: Remote SSH - global cluster-scope environment variables not available in remote ssh terminal
# Version v3.9.67
- Fix: Endpoint - 2 Emails are sent when updating to new model in kafka endpoints
# Version v3.9.66
- New Feature: organization-wide settings option to auto attach an nfs
- Improvement: Add commits table to experiment artifacts
- Improvement: Dataset selector - search for commits and queries
- Improvement: Workspace makeover of top bar
- Fix: "Batch predict" library goes to DEBUG mode, when we try to make a Batch endpoint live
- Fix: "End commit" is not showing for Experiment
- Fix: Search Issues in Experiment's table
- Fix: Flow- deploy task of rabbitmq UI BUG image is not resized on select
- Fix: Experiment - Categorical value separated with whitespaces accept only last char
- Fix: Sorting not working for fields "Architecture" and "Word embedding" in the experiment's table
- Fix: Experiment visuals - can't zoom in when trying to capture first data point
- Fix: Flow - 'emails_to_notify' field exist in the exec_task but doesn't reach the experiment
# Version v3.9.65
- Fix: Experiment - Dataset link leads to 404 if there is a Hyphen in the dataset name
- Fix: Endpoint - is_deployment_running return false even if there is model up
- Fix: User that isn't a collaborator can't start jobs in a public project
# Version v3.9.64
- New Feature: Show workspace names in delete confirmation message
- Improvement: Add support to "openssl_verify_mode" param in SMTP settings
- Fix: SSO: trying to add a user that has a dash in its username fails
- Fix: Workspace - MPI Template slotsPerWorker not effected by the configuration
- Fix: Endpoints - Logs are not pulled if a Prometheus authentication on the cluster not defined
- Fix: SDK- Update model will fail if there is uppercase letter in the organization name
# Version v3.9.63
- New Feature: SDK - support new python SDK functionality
- Improvement: Add activity feed for deletion and uploading of files
- Fix: Workspace - sync indication in progress, not presented to user
- Fix: Unable to Rename workspace from workspace overview page
- Fix: Experiments are not available in tensorboard iframe
- Fix: Schedule Flow: In Every hr, Next run is not showing correct over DAG, also the time changes in flow
- Fix: Single_bar_chart and multi_bar_chart are not loading in visual tab for experiment
- Fix: On using deleted hyper experiment's volume in a new experiment, experiment overview page is not showing
- Fix: Webapps are not getting public when using SSO
- Fix: Flow: Pending approval is requested, even if flow version chosen does not have any pending approval in it
- Fix: Flow: Status of flow remains "running" on DAG forever, even the Deploy Task is success
- Fix: Favorites are not getting generated in the "Visual-quick view" of experiment overview page, when creating a new experiment.
- Fix: Experiments running from flow are not presented in experiment table
- Fix: Toggle webhook is not working in flows
- Fix: Charts of the same task name doesn't show names in compare
# Version v3.9.62
- Fix: SDK- Update model will fail if there is uppercase letter in the organization name
# Version v3.9.61
- Fix: SDK - endpoint.is_deployment_running() failing with streaming endpoint
# Version v3.9.60
- Improvement: Spark- Share /data and files across workers and master
# Version v3.9.59
- Fix: Dataset - "cnvrg data put" to a public dataset gets "unauthorized" error
- Fix: NEW SDK - User gets "server error" on cnvrgp dataset clone, when a second user is pending registration
- Fix: File previewer in 'Files' doesn't render non-english text properly
- Fix: Workspace - "copy clone command" from commit copies "undefined" to clipboard
- Fix: Endpoints fail to update version on nginx deployments
# Version v3.9.58
- Improvement: Dashboard export optimization
- Improvement: Add claim name to external disk api
- Fix: Private pypi doesn't affect all the dependencies in a job
- Fix: Experiments Compare doesn't work when started from the SDK using Experiment.init()
# Version v3.9.57
- Fix: Rabbitmq endpoint - 'Prefetch Count' params is getting reset to nil with flow/deploy task
# Version v3.9.56
- Improvement: support for random namespace in remote cluster
- Improvement: Change user logo from initials to full username
- Fix: Not able to cache the dataset
- Fix: Endpoint - Rabbit endpoint stuck on "updating in progress" when Updating model from flow
- Fix: System charts fail to load (UI race condition)
- Fix: Experiment - Accuracy value and tag not pulled from logs
- Fix: Grid Search - objective name doesn't accept underscores
- Fix: Dataset - Caching commit when user doesn't have permissions to the template
# Version v3.9.55
- Fix: Flow - parsing error when connecting 2 tasks
- Fix: Flow - task got terminated by the monitor, following task hangs on "initializing" with no logs
- Fix: System charts fail to load in some cases
# Version v3.9.54
- Fix: Experiment - debug mode isn't working in spark experiments
# Version v3.9.53
- FIx: Docker container id was missing from grid experiments
- Improvement: Dataset - change get started with cnvrg CLI example
# Version v3.9.52
- Fix: Flow - Emails on error sent only the user who lunched the flow
# Version v3.9.51
- Improvement: Parse logs field in elasticsearch as JSON
# Version v3.9.50
- Fix: Experiment - Parallel jobs are not working in bayesian optimization
# Version v3.9.49
- Improvement: improve load times when updating experiments fields
# Version v3.9.48
- Fix: User that isn't an admin can't cache a public dataset
# Version v3.9.47
- Improvement: Support R endpoints with latest rocker/rstudio image
# Version v3.9.46
- Fix: Containers- Pulling image without default compute template results with 401 error
# Version v3.9.45
- Fix: Public projects are not visible to all users
# Version v3.9.44
- Fix: Load on the DB caused experiment that ended with status code 0 to enter debug mode
# Version v3.9.43
- Improvement: Private PyPi repo configure automatically when connecting to a workspace
# Version v3.9.42
- Fix: CLI - Running a flow with deploy task fails
- Fix: Experiment jupyter debug dot is still shown in success
- Improvement: Compute Dashboard export optimization
# Version v3.9.41
- Fix: Install R dependencies for endpoints
# Version v3.9.40
- Improvement: Export dashboard CSV using stream
# Version v3.9.39
- Improvement: GridSearch default values for parallel jobs should be unlimited
# Version v3.9.38
- Fix: Hyper view should show scheduled time with regards to the user's time zone
# Version v3.9.37
- Improvement: GridSearch default values for parallel jobs should be unlimited
# Version v3.9.36
- Fix: Remove validations from experiment hyper search form
# Version v3.9.35
- Fix: Emails notifications from cli are not saved
# Version v3.9.34
- Fix: Fix dynamic routes in tasks
# Version v3.9.33
- Fix: Experiment running rscript doesn't install jupyter for debug mode
# Version v3.9.32
- Fix: SDK - No email validation applies when registering a new user
# Version v3.9.31
- Improvement: Add reindex button to dataset commit
- Fix: Compare different versions of a files doesn't work properly
- Fix: Experiment - Can't filter by tags that have dot in their name
- Fix: Experiment - Changing experiment's title (via stdout) doesn't support whitespaces
- Fix: Endpoint - New Alert pop up is missing flow options
- Fix: Experiment - Debug icon is still on when experiment is in error
- Fix: Endpoint - "Invalid Date" on time column when starting an endpoint
- Fix: Experiment - Can't right-click save-as to download file in the artifacts tab
- Fix: Ray compute can't start with volume/pv
- Fix: Endpoint - Flow name is missing in continual learning tab when we create new trigger for flow
- Fix: "Default Homepage" drop down does not show all the organizations name.
- Fix: Experiment - "stop & sync" should not be available for local experiments
- Fix: SDK - The function
e.finish(exit_status=-1)
exits with wrong status - Fix: Experiment - Charts take long time to load when there are a lot of logs
- Fix: Experiment - Search in config and metrics tab of experiment's overview page is case sensitive
- Fix: Compute Templates - amount of HPU in worker spec disappears from OpenMPI template
- Fix: Experiment - Clone artifacts fails on "Not Authorize, Are you logged in?"
- Fix: After clicking "Link to git repository", it goes to "General" tab instead of "Git Integration".
- Fix: Volumes- Filter by size removes the volumes from the UI
- Fix: Dashboard: Endpoints with status "stopped" due to "max duration" are not shown in the list of jobs
- Fix: Build image from workspace - agent container ID doesn't update when re-running workspace
- Fix: Missing validation when deleting a compute template that is used as the default compute for the organization
# Version v3.9.30
- Improvement: Support multi tenant terms of use
# Version v3.9.29
- Improvement: SDK - Support remove files and directories commands
# Version v3.9.28
- Improvement: Add support to multi tenant terms of use
- Fix: User setting doesn't update automatically after saving Git Access Token
# Version v3.9.27
- Fix: Can't pull agent container from default docker hub registry
# Version v3.9.26
- Fix: Training Flow is missing in continual learning Project
# Version v3.9.25
- Fix: SDK - When running experiments with grid search parameters, parameters are not showing in the UI
- Fix: Dashboard - exported csv file broken when a compute has multiple node-selectors
# Version v3.9.24
- Improvement: Add 2 containers to the default intel registry
# Version v3.9.23
- Fix: Folder names with special characters aborts sync
- Fix: Experiment - wrong parsing when printing tags with parenthesis in the key name
- Fix: Compute template that has a dot in it's name fails to launch flow using cli/trigger
- Fix: Volumes - volumes aren't deleted when a project is deleted
- Fix: Color of recurring status is wrong when creating hyper experiment with recurring option
- Fix: Tensorboard icon showing in enable mode when workspace is offline
- Fix: Workspace - Can't start a workspace with a volume that was deleted
- Fix: Periodic sync is not working in flows
# Version v3.9.22
- Fix: Endpoint scale horizontally functionality doesn't work in RabbitMQ endpoints
# Version v3.9.21
- Fix: Flow - Show an error if a flow is executed with parameters that don't have a key
- Fix: Flow - Failed validation closes the YAML editor
- Fix: SDK - flow.pull_artifacts returns "internal server error"
# Version v3.9.20
- Improvement: Add support to global secrets
- Improvement: Change the acknowledgment of the receiving messages in RabbitMQ
- Fix: Users API should return access_level as a string instead of a number
- Fix: Data scientist can't clone public datasets that were created by admin
# Version v3.9.19
- Fix: Support CVAT with https
# Version v3.9.18
- Fix: Fix compare commits
# Version v3.9.17
- Improvement: Support MIG configuration of 10GB
- Improvement: Support filter by node in the dashboard
- Improvement: Filter in experiments is not saved when going back and forth in the web
- Fix: Title of experiment is not link to experiment show page
- Fix: Can't see all workspace logs after workspace restart
- Fix: Experiments charts from sdk aren't parsed correctly
- Fix: Endpoint's predictions are not showing after 10k of predictions
- Fix: Endpoints log is in endless loop in case elasticsearch returns an error code
# Version v3.9.16
- Improvement: Support labels from config-map
- Fix: Timestamp od last workspace sync is not shown
- Fix: Restarting endpoint with many logs loads a blank page
- Fix: Experiments list is taking too long to load
- Fix: Endpoint metrics are being polled in an asc order and not desc
- Fix: Experiment's duration is squeezed to multiple lines when it's too long
# Version v3.9.15
- Fix: CVAT server doesn't run from within a script
# Version v3.9.14
- Improvement: Wrong error when inviting a user and the user doesnt exist
- Fix: Can't create tag with the name: "Accuracy"
- Fix: Missing validation on environment variables keys ( allows empty values)
- Fix: Metric with underscore is not shown in the GridSearch view
- Fix: Running experiment with parameter that includes a comma runs as Grid Search
- Fix: SDK method
fv.info()
doesn't returns a response - Fix: Experiment that reached idle time while initializing gets stuck trying to restart
# Version v3.9.13
- Fix: Experiment - "zoom out" or "reset zoom" button is missing in charts
# Version v3.9.12
- Improvement: Disable git clone in Tensorboard compare session
# Version v3.9.11
- Fix: Experiment - Charts refresh is too fast
# Version v3.9.10
- Fix: Support existing users in cvat
# Version v3.9.9
- Fix: Flow - Environment variables of artifacts input folders are not passed to all the connected flows
# Version v3.9.8
- Improvement: Run updates of cli asynchronously
# Version v3.9.7
- Fix: Clone projects is not necessary in CVAT workspaces
# Version v3.9.6
- Improvement: Support ssh to job in an openshift environment
# Version v3.9.5
- Improvement: Support Prometheus and Elasticsearch secrets
# Version v3.9.4
- Fix: Export Dashboard is very slow
# Version v3.9.3
- Improvement: Support specific annotations to cnvrg jobs
# Version v3.9.2
- Improvement: Automatically recognize when Kafka is offline without affecting the running endpoint
- Improvement: Support SDK v2 workflow version
# Version v3.9.1
- Improvement: Run with the same docker user in the agent container
# Version v3.8.5
- Improvement: Remove 'source' column from artifacts table
# Version v3.8.2
- Feature: CVAT integration in workspace
- Fix: mpi/pytorch templates with gpu doesnt show gpu stats
# Version v3.8.1
- Improvement: Improve organization dashboard stats load time
- Fix: File preview is not shown in Input and Output artifacts
- Fix: Fix pytorch distributed gpu count in command
- Fix: Remove auto-complete from dataset selector
- Fix: Workspace - collaborators can't run someone else's workspace
# Version v3.8
- Feature: Support multi tenancy integration
- Feature: Support OpenShift integration
- Feature: Support ML Infra Dashboard
- Improvement: Support secrets with prometheus
- Improvement: Support vhost in rabbitmq
- Fix: Example project yaml files are invalid
- Fix: Utilized gpu/hpu is showing "NAN%" instead of 0%.
- Fix: Experiments charts are hidden when playing with the order & size
# Version v3.7
- Feature: Support Rabbitmq in endpoints
- Feature: ADD permissions to compute templates
- Fix: Experiment notifications not sent to slack webhook
- Fix: Export experiments table to a csv generates too many column of grid search IDs
- Fix: Fix compare of commit without parent
- Fix: Flow - no validation of data types when directly pasting flow YAML
- Fix: support download specific commit
- Fix: sdk/not able to stop webapps, workspace, and experiments
- Fix: support all types of mig devices
# Version v3.6.13
- Feature: Support TFServing in endpoints
- Improvement: Add files sizes to get commit files response
- Improvement: Library show page fails on missing compute
- Fix: Access token doesn't update when re-logging in to a different user
- Fix: Webapps GET requests sent from within the webapp fails on timeout
- Fix: Datasets page appear blank in safari
- Fix: SDK: flow.pull_artifacts returns an error
- Fix: SDK: e.pull_artifacts() returns an error
# Version v3.6.12
- Feature: Replace terminal section on sidebar to jupyter in experiments
- Improvement: Add compute name in volumes selector
- : Webapp iframe url stays consistent throughout deployments
- Fix: Webhook in flow changes parameter type from categorical to discrete
- Fix: Experiment with parenthesis in key name gets stuck in a restarting loop
- Fix: Organization Settings - colon in pypi repo URL doesnt pass validation
- Fix: Editing a file through the UI works only the first time
# Version v3.6.10
- New Feature: Experiment New Dashboard
- New Feature: Add flow dag to experiment page
- New Feature: Change mount path when attaching PVC as local folder
- New Feature: Support params to flows from webhook
- Improvement: Allow custom pypi to be used for installing requirements
- Improvement: Support log images in experiments
- Improvement: add ability to drag charts in quick view
- Improvement: add search and sort to visuals tab
- Improvement: Support dot in chart title
- Fix: Validation in tasks is not working if parallel job count field is greater than max job field
- Fix: Creating template using specific node label raises an error
- Fix: Saving a task as a library raises an error
- Fix: Flow - notify on success/error set to true in yaml but email is not sent
- Fix: cant push new images if repo exist in ECR
- Fix: Flow- 2 compute names from different clusters causes the flow to not run
- Fix: SDK: running an experiment with arguments ignores the compute
- Fix: SDK: running an experiment with non-existent compute templates defaults to organization settings instead of project settings
- Fix: Compute is showing as "unassigned" for scheduled experiment until manual refresh
- Fix: Datasets is not downloaded in endpoint when endpoint is run from flow
# Version v3.6.9
- Feature: Support Oauth Proxy
- Feature: ADD mig (Multi instance gpu) Support
- Feature: ADD Ray integration
- Feature: ADD permissions to compute templates
- Improvement: Dataset - JSON files displayed without indentation
- Improvement: Retrieve organization name from URL In the websocket
- Improvement: Support cluster restart and restart pods lifecycle
- Improvement: SDK: Support Experiment.alert from sdk
- Improvement: CLi: improve mechanism of removing many folders in dataset
- Improvement: Update workspace title to be Workspace
- Improvement: support global gateway
- Improvement: Add Custom Scheduler option
- Fix: Can't restart jobs with PCV
- Fix: Dashboard graph is not presented
- Fix: mpi workers don't pull secrets
# Version v3.6.8
- Feature: Support HyperParameter tuning for experiments and flows: Grid Search, Random Search, Bayesian
- Fix: CLI: Experiments are always taking folder as output even if we are running exp on custom folder
- Improvement: Change error message when attempting to delete a predefined image
- Fix: Fix View of commit in the browser
- Fix: flow is starting experiments in "waiting mode"
- Fix: Fix default timezone for users
- Fix: SDK: Can't authenticate in windows
- Fix: SDK: Fetch all files in dataset returns error