Thank you for the reply and the post. We're running JupyterLab for some students, and when several of them are training models, they consume GPU memory.
Now, we thought of several ways of solving this. We have a branch that uses Kubernetes but this is not the one that is deployed.
We know about SLURM and this is something we will support for a coarse granularity. i.e: notebook level jobs. This is a common scenario and we'll need it for our AppBook (we allow users to turn a notebook into an application with one click and we automatically generate the form fields for the features, and the API endpoint for the model).
However, I'm interested in finer granularity computation; as in: I want a job for the cell. This involves looking into Jupyter kernels to circumvent the front-end<-->kernel disconnection (we'll have to do that for another use case in low internet bandwidth scenario).
>Sorry, I could not understand this question. Do you mean metadata about A job?
How do your users upload, and manage data access and is the data mutable and versioned. What version of the code ran on which version of the data, and who changed that data, etc.
How do users access the data from a Jupyter notebook? Is the data in different object storages? Do you proxy the requests and handle access, things like that.