Describe the bug Fail to create any workspace and workflow. Unschedulable: 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.

Could it be the problem of the yaml?

#################################################################
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Component: Database
# Description: Database connection information
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
database:
  # Name of database
  # If using an external production database, use the name of that database.
  # For in-cluster test database, use any name you like.
  databaseName: onepanel
  # Do not change, only `postgres` driver is supported at this time.
  driverName: postgres
  # Database host - use `postgres` for in-cluster test database
  host: postgres
  # Database password
  # If using an external production database, use the password for that database.
  # For in-cluster test database, use any password you like.
  password: onepanel
  # Database port
  port: 5432
  # Database username
  # If using an external production database, use the username for that database.
  # For in-cluster test database, use any username you like.
  username: onepaneldemo
##################################################################

opctl version

$ opctl version

CLI version: v0.15.0-rc.0
Manifest version: v0.15.0-rc.0
API version: v0.15.0-rc.0
Web UI version: v0.15.0-rc.0

opctl init command opctl init --provider aks --artifact-repository-provider s3 --gpu-device-plugins nvidia

Kubernetes information

• Cloud provider: AKS
• Kubernetes version: 1.18.10

Machine information

• OS: Windows 10
• Browser: Chrome, Firefox

Screenshots If applicable, add screenshots to help explain your problem.

What this PR does: Adds a design document of the implementation.

• Particularly, the background and explanation of the solution.

Adds migrations to add lifecycle hooks into the workspace templates.

Which issue(s) this PR fixes:

Fixes onepanelio/core#623

Special notes for your reviewer:

Checklist

Please check if applies

• [ ] I have added/updated relevant unit tests
• [X] I have added/updated relevant documentation

Required

• [X] I accept to release these changes under the Apache 2.0 License

• Should be run as root
• Should have one docker image for latest TensorFlow GPU
• Another docker image for latest PyTorch GPU
• The following extensions:
  • jupyter-widgets/jupyterlab-manager
  • Git with GitHub
  • TensorBoard, see this notebook for validation
  • jupyter-matplotlib
  • nbdime-jupyterlab
  • jupyterlab/debugger

Issue

We want to be able to run TensorBoard as a side car container for workflows, particularly the workflow container that is doing the training.

This can be added as a sidecar container in the manifest. It needs to make sure that it also has the same volume mounted as the training container so the data is shared and it can read it and draw the data.

The UI should show a tensorboard button under artifacts (sidebar on the right) when this node is selected.

Eventually this should launch on demand, but that might be tough because of time and volume mount sharing.

When I installed onepanel, the final deployment was stuck：

[root@localhost home]# KUBECONFIG=./kubeconfig opctl apply Starting deployment...

2020/11/30 16:23:13 namespace/application-system unchanged customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io configured clusterrole.rbac.authorization.k8s.io/application-manager-role configured clusterrolebinding.rbac.authorization.k8s.io/application-manager-rolebinding configured service/application-controller-manager-service unchanged statefulset.apps/application-controller-manager unchanged

Excuse me, what is this question?

What this PR does:

Updates the workspace templates so that runtime variables like the DOMAIN are injected when the workspace is is created instead of when the workspace template is created.

This also updates the server so that it reloads data upon a detected configuration change, so variables like DOMAIN are always up to date.

Which issue(s) this PR fixes:

Fixes onepanelio/core#348

Special notes for your reviewer:

For the runtime variables, there are 3 main places of interest.

• When we create a template for viewing, as when we generate a dynamic template while creating/editing a workspace template. In this situation, we need to inject runtime variables for things like the Machine Type dropdown.
• Whenever we get a Workspace Template, we need to modify the manifest to include the variables.
• When we create the Workflow Execution for a Workspace. The Argo WorkflowTemplate needs to have the appropriate variables injected into it.

NOTE: If you attempt to view the YAML in the workflow execution, it will not have the injected variables as it gets the Workflow Execution, and we do not know to inject the runtime variables there. We might want to eventually add a method to get the workflow execution yaml for a Workspace to get around this.

Currently, we're using environment variables, instead we should use the Python SDK's list_workflow_templates API to get a list of templates and only display the ones in CVAT with the label used-by: cvat

This will replace the need for ONEPANEL_MASKRCNN_TEMPLATE_ID and ONEPANEL_OD_TEMPLATE_ID environment variables.

If I have understood correctly, to used OpenPanelio, the data has to be saved on one of the following cloud servers, Microsoft Azure, Amazon EKS, Google GKE.

I have my own data server and I have no plans to put my data on cloud. Is it still possible to use OpenPanelio? If yes, in the docs, I could not find how to do so.

Related to this line in the CLI https://github.com/onepanelio/cli/blob/bea78bcd4d1d33318e5603e4ffcae3f1170d5710/cmd/build.go#L238

Does it make sense for Minio to require a region?

I tried to deploy onepanel in minikube (single node cluster) Steps which we followed:

1. Install VirtualBox Hypervisor, Minikube and Kubectl in VM (Standard_B4ms --> CPU: 4, RAM: 16 GB) Ubuntu(18.04) linux.
2. Install opctl, refered link: https://github.com/onepanelio/core/releases/tag/v0.17.0
3. opctl init --provider minikube --artifact-repository-provider s3
4. After running the step3 command got 2 files i.e. config.yaml and param.yaml then I edit the param.yaml accordingly (sharing the edited param.yaml file without credential).

After executing all the above steps I run optctl apply command it gives a error, sharing the error screenshot in attachment. If possible kindly share the doc to deploy the onepanel in minikube.

Thankyou

Screenshots

Param.yaml File

param-file.txt

These environment variables will mount to every container to enable integrations from within the containers:

• ONEPANEL_API_URL - Format: <http-scheme>://<host>[:<apiHTTPPort>]/[apiPath]
• PROVIDER_TYPE - local or cloud

When using: microk89s enable gpu and deploying + opening CVAT 1.6, new window apperas and there is login box with notifications (with cvat 1 there is no such problem).

The problem is: when using CPU and not enablin microk8s enable gpu everything works but when enabling gpu cvat 1.6 gives error (additional info: also there are some anomalies running other workspaces - it seems like more than 1-2 together doesnt work):

Could not check authorization on the server

Error: Request failed with status code 503. "upstream connect error or disconnect/reset before headers. reset reason: connection failure".

Could not get user agreements from the server Error: Request failed with status code 503. "upstream connect error or disconnect/reset before headers. reset reason: connection failure".

• $ opctl version CLI version: v1.0.2 Manifest version: v1.0.2 API version: v1.0.2 Web UI version: v1.0.2

• opctl init --provider microk8s --enable-metallb --artifact-repository-provider abs

• Cloud provider: AKS

• Using on local machine, microk8s

• sudo snap install microk8s --channel=1.19/stable --classic

• Ubuntu 20.04.5 LTS

• nvidia-smi_ Mon Dec 19 14:08:52 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:65:00.0 On | N/A | | 0% 36C P8 8W / 180W | 1MiB / 8192MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

Any ideas whats the problem or how to diagnose/find the problem?

Discussed in https://github.com/onepanelio/onepanel/discussions/980

opctl apply
Starting deployment...

clusterrole.rbac.authorization.k8s.io/application-manager-role configured
clusterrolebinding.rbac.authorization.k8s.io/application-manager-rolebinding configured
namespace/application-system unchanged
service/application-controller-manager-service unchanged
statefulset.apps/application-controller-manager unchanged

Failed: unable to recognize ".onepanel/application.kubernetes.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

Describe the bug /kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] Following the docs tutorial with multipass on a Windows machine.

After deploying Onepanel with microk8s config > kubeconfig KUBECONFIG=./kubeconfig opctl apply the kfserving-controller:v0.6.0 image fails to pull with an 401 Unauthorized error.

In the Onepanel UI creating a new model server like in here results in the following error:

[500] Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.v1beta1.defaulter": Post "https://kfserving-webhook-server-service.kfserving-system.svc:443/mutate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s": dial tcp 10.152.183.188:443: connect: connection refused http://serving.onepanel.pvaintern/api/namespaces/pvaonepanel/inferenceservices

I am guessing that those two error are connected.

What did you expect to happen: The gcr.io/kfserving/kfserving-controller:v0.6.0 should be accessible. A new model server should be created.

Anything else you would like to add: Output of microk8s.kubectl get pods/kfserving-controller-manager-0 -n kfserving-system kfserving-system kfserving-controller-manager-0 1/2 ImagePullBackOff 1 23h

kubectl describe pod: Events: Type Reason Age From Message

Warning Failed 26m (x267 over 23h) kubelet Failed to pull image "gcr.io/kfserving/kfserving-controller:v0.6.0": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/kfserving/kfserving-controller:v0.6.0": failed to resolve reference "gcr.io/kfserving/kfserving-controller:v0.6.0": pulling from host gcr.io failed with status code [manifests v0.6.0]: 401 Unauthorized Normal Pulling 21m (x269 over 23h) kubelet Pulling image "gcr.io/kfserving/kfserving-controller:v0.6.0" Normal BackOff 55s (x6038 over 23h) kubelet Back-off pulling image "gcr.io/kfserving/kfserving-controller:v0.6.0"

Output of microk8s.kubectl logs pod/kfserving-controller-manager-0 -n kfserving-system -c manager Error from server (BadRequest): container "manager" in pod "kfserving-controller-manager-0" is waiting to start: trying and failing to pull image

Output of microk8s.kubectl logs pod/kfserving-controller-manager-0 -n kfserving-system -c kube-rbac-proxy I1004 07:52:03.495440 1 main.go:209] Generating self signed cert as no cert is provided I1004 07:52:03.666661 1 main.go:242] Listening securely on 0.0.0.0:8443

Anything else you would like to add:

Importing the docker image via microk8s ctr image import kfserving.kfserving-controller.tar manually did not solve the problem.

According to the issues below changing the pull location from gcr.io to docker.io should help. (This where I was able to pull the image manually.) https://github.com/kserve/kserve/issues/1781 https://github.com/kserve/kserve/issues/1976#issuecomment-1007453347 https://hub.docker.com/u/kfserving

I also tried changing line 32121 (below) in .onepanel/kubernetes.yaml from gcr.io to the docker.io and applying the changes with KUBECONFIG=./kubeconfig opctl apply but the file was reset to its original state.

containers: - args: - --metrics-addr=127.0.0.1:8080 command: - /manager env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: SECRET_NAME value: kfserving-webhook-server-cert image: gcr.io/kfserving/kfserving-controller:v0.6.0 # line 32121 imagePullPolicy: Always name: manager

opctl version CLI version: v1.0.2 Manifest version: v1.0.2 API version: v1.0.2 Web UI version: v1.0.2

opctl init command opctl init --provider microk8s --enable-metallb --artifact-repository-provider s3

Kubernetes information

• Cloud provider: Microk8s
• Kubernetes version: 1.21

Machine information

• OS: Windows 10 Pro 19043.1889
• Browser: Firefox

Any help would be appreciated! Thanks :)

Hi.. im trying to install on a local PC ( using minio) but im stopped just from the start. im following the guide.

1. install mikcrok8.. i have tried with 1.19 ( which is in the page) and latest
2. add user to group and restart
3. wait for the status. At first some warnings.. after applying the recommendations.. it works ok.
4. so HERE STATUS is STARTED
5. Apply some changes to sudo nano /var/snap/microk8s/current/args/kube-apiserver
6. restart the service
7. check status. -> its waits forever OR says not running check with inspect
8. i do the inspect and it has no warning

im stuck at this stage.. is there something missing? thanks

CVAT-1.0.6 version

Describe the bug A clear and concise description of what the bug is.

CLI version: v1.0.2 Manifest version: v1.0.2 API version: v1.0.2 Web UI version: v1.0.2

opctl init --provider microk8s
--enable-metallb
--artifact-repository-provider s3

Kubernetes information

• Cloud provider: [ Microk8s]
• Kubernetes version: 1.19.15

Machine information

• OS: [Ubuntu 21.04]
• Browser: [Chrome]

CVAT: onepanel/cvat:v1.0.2_cvat.1.6.0 `2022-05-08 13:17:16,675 DEBG 'runserver' stderr output: Traceback (most recent call last): File "/home/django/manage.py", line 21, in execute_from_command_line(sys.argv) File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line utility.execute() File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv self.execute(*args, **cmd_options) File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute output = self.handle(*args, **options) File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped res = handle_func(*args, **kwargs) File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 75, in handle self.check(databases=[database]) File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 392, in check all_issues = checks.run_checks( File "/opt/venv/lib/python3.8/site-packages/django/core/checks/registry.py", line 70, in run_checks new_errors = check(app_configs=app_configs, databases=databases) File "/opt/venv/lib/python3.8/site-packages/django/core/checks/urls.py", line 13, in check_url_config return check_resolver(resolver) File "/opt/venv/lib/python3.8/site-packages/django/core/checks/urls.py", line 23, in check_resolver return check_method() File "/opt/venv/lib/python3.8/site-packages/django/urls/resolvers.py", line 408, in check for pattern in self.url_patterns: File "/opt/venv/lib/python3.8/site-packages/django/utils/functional.py", line 48, in get res = instance.dict[self.name] = self.func(instance) File "/opt/venv/lib/python3.8/site-packages/django/urls/resolvers.py", line 589, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/opt/venv/lib/python3.8/site-packages/django/utils/functional.py", line 48, in get res = instance.dict[self.name] = self.func(instance) File "/opt/venv/lib/python3.8/site-packages/django/urls/resolvers.py", line 582, in urlconf_module return import_module(self.urlconf_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/home/django/cvat/urls.py", line 27, in path('', include('cvat.apps.engine.urls')), File "/opt/venv/lib/python3.8/site-packages/django/urls/conf.py", line 34, in include urlconf_module = import_module(urlconf_module) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/home/django/cvat/apps/engine/urls.py", line 7, in from . import views File "/home/django/cvat/apps/engine/views.py", line 42, in import cvat.apps.dataset_manager.views # pylint: disable=unused-import File "/home/django/cvat/apps/dataset_manager/views.py", line 15, in import cvat.apps.dataset_manager.task as task File "/home/django/cvat/apps/dataset_manager/task.py", line 17, in from .annotation import AnnotationIR, AnnotationManager File "/home/django/cvat/apps/dataset_manager/annotation.py", line 10, in from shapely import geometry File "/opt/venv/lib/python3.8/site-packages/shapely/geometry/init.py", line 4, in from .base import CAP_STYLE, JOIN_STYLE File "/opt/venv/lib/python3.8/site-packages/shapely/geometry/base.py", line 19, in from shapely.coords import CoordinateSequence File "/opt/venv/lib/python3.8/site-packages/shapely/coords.py", line 8, in from shapely.geos import lgeos File "/opt/venv/lib/python3.8/site-packages/shapely/geos.py", line 92, in free = load_dll('c', fallbacks=c_alt_paths).free File "/opt/venv/lib/python3.8/site-packages/shapely/geos.py", line 60, in load_dll raise OSError( OSError: Could not find lib c or load any of its variants ['libc.musl-x86_64.so.1'].

2022-05-08 13:17:16,675 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-08 13:17:16,675 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-08 13:17:16,675 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-08 13:17:16,675 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-08 13:17:16,675 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-08 13:17:16,715 DEBG 'git_status_updater' stderr output: Traceback (most recent call last): File "/home/django/manage.py", line 21, in execute_from_command_line(sys.argv) File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line utility.execute() File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 244, in fetch_command klass = load_command_class(app_name, subcommand) File "/opt/venv/lib/python3.8/site-packages/django/core/management/init.py", line 37, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/home/django/cvat/apps/dataset_repo/management/commands/update_git_states.py", line 6, in from cvat.apps.dataset_repo.dataset_repo import update_states File "/home/django/cvat/apps/dataset_repo/dataset_repo.py", line 20, in from cvat.apps.dataset_manager.task import export_task File "/home/django/cvat/apps/dataset_manager/task.py", line 17, in from .annotation import AnnotationIR, AnnotationManager File "/home/django/cvat/apps/dataset_manager/annotation.py", line 10, in from shapely import geometry File "/opt/venv/lib/python3.8/site-packages/shapely/geometry/init.py", line 4, in from .base import CAP_STYLE, JOIN_STYLE File "/opt/venv/lib/python3.8/site-packages/shapely/geometry/base.py", line 19, in from shapely.coords import CoordinateSequence File "/opt/venv/lib/python3.8/site-packages/shapely/coords.py", line 8, in from shapely.geos import lgeos File "/opt/venv/lib/python3.8/site-packages/shapely/geos.py", line 92, in free = load_dll('c', fallbacks=c_alt_paths).free File "/opt/venv/lib/python3.8/site-packages/shapely/geos.py", line 60, in load_dll raise OSError( OSError: Could not find lib c or load any of its variants ['libc.musl-x86_64.so.1'].`

I installed the libc.musl-x86_64.so.1 and make noting effect.

Any suggestions?