Open OnDemand automated configuration and deployment @ DCSR-UNIL

Introduction

Open OnDemand (OOD or OpenOD) is a browser-based GUI for interacting with high-performance computing environments.

This repository encompasses the Ansible configuration for the deployment of a dedicated host on the DCSR infrastructure, together with the Podman/Docker definition files for the Open OnDemand services, as well as the Apptainer definition files for running the interactive desktops on the compute nodes.

Requirements

Unprivileged user access to a controller machine located within the UNIL network (e.g. your laptop); your username on the controller will later be referred as CONTROLLER_USER
Ansible installed on the controller machine (see the installation instructions with pip)
A dedicated host (e.g. a virtual machine) for hosting Open OnDemand, also located within the UNIL network, with:
- A minimal RHEL 8/9 installation
- Root access through SSH, with the public key of the user on the controller machine installed on the /root/.ssh/authorized_keys file
- Optional: the home directories of the cluster users locally mounted (typically on /users; homedir paths must always match those on the cluster); on a development deployment this will be achieved later with sshfs for selected users
The public keys of the cluster frontend are stored in files/curnagl_ood/pubkeys; if the dedicated host is not on the same subnet as the cluster, check that those keys are still valid, or alternatively enable the automatic download (and acceptance) of the new keys directly from the frontend by setting the variable sshuttle_keyscan to true, for example on the group_vars/all/user_config.yml configuration file (that you might have to create, see further below)
The inventory file hosts must be adjusted so that the ood machine properly refers to your dedicated host (and port, if non-standard), you can use one of the templates (hosts-local for a development deployment on a local VM, or hosts-dev or hosts-prod for a development (respectively production) deployment on a host in the curnagl subnet, on which the /users filesystem is mounted)
Basic host-related variables need to be defined in the corresponding file inside host_vars folder, in a file matching the hostname (as defined in the [ood_hosts] section of the hosts file). The files ood-localhost.yml, ood-dev.yml and ood-prod.yml can be used as templates.
Unprivileged user access to the cluster frontend through SSH, with the public key of the user on the controller machine installed on the ~/.ssh/authorized_keys file; your username on the cluster will later be referred as CLUSTER_USER
The MUNGE key of the cluster must be copied to the right location (e.g. for the curnagl cluster, it must be copied to files/curnagl_ood/munge.key)

Configuration

Create the file group_vars/all/user_config.yml with the following contents (replacing the uppercase entities with the appropriate values):

---
# Frontend remote user
frontend_remote_user: CLUSTER_USER

# LDAP credentials
ldap_bindDN: LDAP_BIND_DN
ldap_authtok: LDAP_AUTHTOK

# Dex secrets
ood_session_secret: OOD_SECRET
hedgedoc_session_secret: HEDGEDOC_SECRET

# HedgeDoc
hedgedoc_db_password: HEDGEDOC_DB_PASSWORD

Tip

If CONTROLLER_USER coincides with CLUSTER_USER, you don't need to explicitly provide the variable frontend_remote_user above.

The OOD_SECRET and HEDGEDOC_SECRET are arbitrary; they're the shared secret between the Dex OpenID Connect authentication provider and Open OnDemand and HedgeDoc, respectively. Ansible will deploy them automatically. HEDGEDOC_DB_PASSWORD is the password for the HedgeDoc database (set in the first deployment).

Tip

Random safe secrets can be generated with:

openssl rand -base64 33

Deployment

Most of the Open OnDemand deployment is made within a podman container. The host will however need a minimal configuration and very few packages installed. You can achieve this with:

ansible-playbook -i hosts ood-deploy.yml

If your host is running in a different subnet from the curnagl subnet, you will need to tunnel all of its traffic to curnagl's subnet. You can achieve this with:

ansible-playbook -i hosts ood-deploy.yml --tags sshuttle

Important

You will need to repeat this command whenever the internet connection or the tunnel is interrupted

The next step is to configure the SSSD daemon to connect to the UNIL LDAP:

ansible-playbook -i hosts ood-sssd.yml

You will then need to fetch the software stack and Jupyter kernel from the curnagl frontend node and produce a corresponding cache on your Open OnDemand host:

ansible-playbook -i hosts ood-dcsrsoft.yml

This step needs to be repeated whenever you want to update the cache.

Tip

You can globally change the stack that Open OnDemand apps will use in order to load their required modules; this is defined in group_vars/curnagl_ood/main.yml, and then you will have to rebuild the cache.

Now if you are in a development context, in which the /users directory is not mounted, you'll have to mount homes individually for each user who will be accessing the cluster. You then need to take a detour to the next section for performing these mounts using sshfs before proceeding.

The very last step is to build and start the Open OnDemand container on the host machine:

ansible-playbook -i hosts ood-container.yml

The HedgeDoc pad, running in a dedicated pod, can then be deployed similarly:

ansible-playbook -i hosts hedgedoc-pod.yml

Tip

Then you can stop, start or restart the Open OnDemand container and the HedgeDoc pod by providing the tags stop, start or restart respectively, in the above commands (just appending the --tags argument). Note that the states of the containers are preserved upon reboots.

User management in a development environment

Important

All user management commands that involve mounting and/or unmounting home directories must be performed with the Open OnDemand container stopped.

Stop the Open OnDemand container with:

ansible-playbook -i hosts-local ood-container.yml --tags stop

Providing access to a given user involves the following steps:

Register the user for having their home directory bind-mounted on the Open OnDemand container (as a volume)
Generate a pair of cryptographic keys and upload the public key to the user's ~/.ssh/authorized_keys on the cluster
Mount the user's home directory on the Open OnDemand host (on the location that will then be bind-mounted on the container) with sshfs

The `CLUSTER_USER`:

These steps can be automatically performed for the CLUSTER_USER with the following command:

ansible-playbook -i hosts-local ood-sshfs.yml --tags add,mount

Unmounting their home is performed with:

ansible-playbook -i hosts-local ood-sshfs.yml --tags umount

The user can also be completely removed (reverting the two first steps) with:

ansible-playbook -i hosts-local ood-sshfs.yml --tags del

Tip

Unmounting is automatically performed (if necessary) before removing users.

Other users:

Other users can also be simply provisioned:

ansible-playbook -i hosts-local ood-sshfs.yml -e "sshfs_users={{['dupond','dupont']}}" --tags add

The new public keys for the new users will then be printed so that you can copy them to the corresponding ~/.ssh/authorized_keys files on the cluster. Once done, mounting is performed with:

ansible-playbook -i hosts-local ood-sshfs.yml -e "sshfs_users={{['dupond','dupont']}}" --tags mount

Tip

For working with a single user, you can use the syntax -e "sshfs_user=dupont".

The del tag is similarly available.

Removing all users:

ansible-playbook -i hosts-local ood-sshfs.yml --tags reset

Staying up to date

System packages upgrade on the host:

ansible-playbook -i hosts-local ood-deploy.yml --tags upgrade

System packages upgrade on the Open OnDemand container:

Force a rebuild of all containers by removing the existing images:

ansible-playbook -i hosts-local -e "podman_rmi=true" ood-container.yml

Safe removal of the MUNGE key from the host machine

The MUNGE key that is copied to the host machine during deployment can also be removed from it, in a safe way:

ansible-playbook -i hosts-local ood-deploy.yml --tags rmkey

The MUNGE key is hence shredded on the host machine before deletion.

Important

This command removes the MUNGE key from the host, it does not remove it from the controller machine.

License

Scientific-Computing/ood-config

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages