diff --git a/doc/.custom_wordlist.txt b/doc/.custom_wordlist.txt index 27910882a..a44d59478 100644 --- a/doc/.custom_wordlist.txt +++ b/doc/.custom_wordlist.txt @@ -73,6 +73,7 @@ subfolders subnet subnets SVG +systemd Terraform TinyPNG TLS diff --git a/doc/explanation/security.md b/doc/explanation/security.md index 8ab18e742..cd28aeabd 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -14,8 +14,12 @@ MicroCloud {ref}`further enforces security ` through th MicroCloud runs on Ubuntu and benefits from all [Ubuntu platform security measures](https://ubuntu.com/security), including kernel hardening, signed packages, and continuous security maintenance. For production environments, we recommend using a recent Ubuntu LTS release to ensure long-term support and predictable security updates. +Ubuntu LTS releases subscribed to Ubuntu Pro can use the [Ubuntu Security Guide (USG)](https://documentation.ubuntu.com/security/compliance/usg/) for CIS hardening. Refer to the LXD documentation on {ref}`lxd:howto-security-harden-cis` for related details about auditing LXD hosts with USG. + (exp-security-snaps)= -## Snaps +## Snaps and supported versions + +The MicroCloud team maintains both Long Term Support (LTS) and feature releases. See {ref}`ref-releases-snaps` and our {ref}`ref-release-notes` for details about the currently supported releases. MicroCloud and its components are distributed as [snaps](https://snapcraft.io/docs), which enhances security by providing a confined environment with a streamlined update mechanism. Both LTS and feature channels receive regular security updates through Canonical’s official infrastructure. @@ -24,15 +28,21 @@ All snaps are digitally signed using {ref}`assertions `. Also see the {doc}`MicroOVN security process documentation `. +MicroOVN secures its network endpoints using the TLS protocol (version 1.2 or higher), along with P-384 elliptic curve keys. For details, refer to the MicroOVN documentation on {doc}`cryptography `, {doc}`working with TLS `, and the {doc}`MicroOVN security process `. diff --git a/doc/how-to/decommission.md b/doc/how-to/decommission.md new file mode 100644 index 000000000..e7566d155 --- /dev/null +++ b/doc/how-to/decommission.md @@ -0,0 +1,276 @@ +--- +myst: + html_meta: + description: Follow these steps to securely decommission a MicroCloud cluster member or cluster. +--- + +(howto-decommission)= +# How to securely decommission a MicroCloud deployment + +```{important} +This process will erase all data associated with your MicroCloud deployment. +Make copies of any data that you need to preserve before proceeding. +Refer to {ref}`lxd:instances-backup` and {ref}`lxd:howto-storage-backup-volume` for relevant details. +``` + +This guide walks you through the steps to decommission an entire MicroCloud cluster. + +If you only need to decommission a single cluster member, first {ref}`remove the member from the cluster `. +After removing the member, {ref}`update the certificate ` on the cluster remaining in production. +Then, return to this guide, skip ahead to {ref}`howto-decommission-remove-microcloud`, and follow the instructions in the sections that follow. + +Some commands used to decommission MicroCloud are LXD or MicroCeph commands. +Refer to the {ref}`LXD decommissioning guide ` and the {doc}`MicroCeph guide to removing disks ` for related information. + +(howto-decommission-remove-offline-member)= +## Remove offline cluster members + +Use the `--force` flag to {ref}`remove any offline cluster members ` (you will remove online cluster members later in the process): + +```bash +sudo microcloud remove --force +``` + +(howto-decommission-revoke-remote)= +## Revoke remote access + +List all identities that have access to LXD, then delete each identity: + +```bash +lxc auth identity list +lxc auth identity delete / +``` + +(howto-decommission-list-projects)= +## List projects + +Instances, profiles, and custom volumes are scoped by {ref}`project `. +For deployments with more than one project, you must repeat some steps for **each** project, each time using the `--project` flag. +You do not need to use the `--project` flag to decommission deployments with only one project. + +Run this command to get a list of all projects: + +```bash +lxc project list +``` + +````{note} +You can also delete a project (except the `default` project) and all of its project-level entities with: + +```bash +lxc project delete --force +``` +```` + +(howto-decommission-delete-data)= +## Delete data + +You can run the commands in this section on any online cluster member to delete data. + +```{important} +Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks. +To prevent unauthorized data recovery, you must {ref}`destroy and sanitize your data `. +``` + +(howto-decommission-delete-instances)= +### Stop and delete instances + +For each project, stop all instances: + +```bash +lxc stop --all --project +``` + +Next, for each project, list all instances, then delete each instance: + +```bash +lxc list --project +lxc delete --project +``` + +If you are unable to stop or delete an instance, use the `--force` flag: + +```bash +lxc stop --force --project +lxc delete --force --project +``` + +(howto-decommission-delete-profiles)= +### Delete profiles + +For each project, list all profiles: + +```bash +lxc profile list --project +``` + +Each project has a `default` profile that cannot be deleted. +Delete all other profiles: + +```bash +lxc profile delete --project +``` + +(howto-decommission-remove-disk-devices)= +### Remove disk devices from `default` profiles + +You cannot delete a storage pool used by an instance, profile, or custom volume. +You must, therefore, remove any disk devices used by the `default` profiles in order to delete any storage pools or custom volumes referenced by those devices. + +At a minimum, the `default` profile of the `default` project has a disk device named `root` that references a storage pool. +Remove this device with: + +```bash +lxc profile device remove default root --project default +``` + +To check for additional disk devices, view information about the `default` profile of each project: + +```bash +lxc profile show default --project +``` + +Remove any remaining disk devices that reference storage pools: + +```bash +lxc profile device remove default --project +``` + +(howto-decommission-delete-volumes)= +### Delete custom volumes + +To delete {ref}`custom volumes `, you must specify the storage pools used by the volumes. +First, list all storage pools across projects: + +```bash +lxc storage list +``` + +Next, for each storage pool, list the custom volumes. +Use the `--all-projects` flag to view all custom volumes across projects: + +```bash +lxc storage volume list type=custom --all-projects +``` + +Use the `PROJECT` column in the output to identify the project associated with each custom volume. +Then delete each custom volume, specifying both the storage pool and the project: + +```bash +lxc storage volume delete --project +``` + +(howto-decommission-delete-pools)= +### Delete storage pools + +```{note} +Storage pools are not scoped by project, so you do not need to use the `--project` flag with `lxc storage` commands. +``` + +List all storage pools, then delete each one: + +```bash +lxc storage list +lxc storage delete +``` + +(howto-decommission-delete-monitoring)= +### Delete monitoring data + +Delete data from any external systems that you used to monitor {ref}`LXD events `, {ref}`LXD metrics `, or {doc}`Ceph logging `, such as [Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), or [Grafana](https://grafana.com/). +Refer to the documentation for those systems for details. + + +(howto-decommission-remove-microceph-osds)= +## Remove MicroCeph OSDs + +To {doc}`remove MicroCeph OSDs `, list all disks, then remove each one: + +```bash +microceph disk list +sudo microceph disk remove +``` + +Finally, verify that the OSDs have been removed: + +``` +microceph disk list +``` + +````{note} +If you are unable to remove an OSD, use the `--bypass-safety-checks` flag: + +```bash +sudo microceph disk remove --bypass-safety-checks +``` +```` + + +(howto-decommission-remove-remaining-members)= +## Remove remaining cluster members + +After deleting data, you can remove the online cluster members from the cluster. +First, list all cluster members: + +```bash +microcloud cluster list +``` + +You can then {ref}`remove most cluster members ` with: + +```bash +sudo microcloud remove +``` + +However, before reducing the cluster from two members to one member, you must {ref}`clean up the Ceph monitor map `. + +```{note} +As you remove each member, you can run `microcloud status` on the remaining cluster members to verify the removal. +``` + +(howto-decommission-remove-microcloud)= +## Remove snaps + +```{important} +Run these commands on **every** machine that you decommission. + +Removing MicroCloud **does not** erase {ref}`ZFS pools (zpools) ` or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs). +To securely decommission MicroCloud, you must {ref}`destroy and sanitize your data `. +``` + +Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps. +Use the `--purge` flag, or a snapshot of your data will be preserved: + +```bash +sudo snap remove microcloud --purge +sudo snap remove lxd --purge +sudo snap remove microceph --purge +sudo snap remove microovn --purge +``` + +```{note} +The MicroCeph and MicroOVN snaps may not be installed if you deployed a MicroCloud without those components. +``` + +Verify that the snaps and associated data were removed. +The following commands should report that none of these snaps are installed and that the `/var/snap/microcloud/`, `/var/snap/lxd/`, `/var/snap/microceph/`, and `/var/snap/microovn` directories do not exist: + +```bash +snap list microcloud lxd microceph microovn +ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/ +``` + +(howto-decommission-destroy-data)= +## Destroy and sanitize data + +Data deleted with MicroCloud, LXD, or MicroCeph commands remains readable and can be recovered by users with access to disks used in your deployment. +To prevent unauthorized recovery, you must physically overwrite the data. +Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning. + +If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics. +For clusters {ref}`configured with OIDC `, consult your OIDC identity provider for the steps to remove any data associated with your profile. +Likewise, if you used {ref}`ACME services to issue server certificates `, refer to the service provider for the steps to remove any associated data. + +```{important} +Sanitized data is irreversibly destroyed and cannot be recovered. +``` diff --git a/doc/how-to/index.md b/doc/how-to/index.md index e4b4c9589..543b53a4d 100644 --- a/doc/how-to/index.md +++ b/doc/how-to/index.md @@ -58,6 +58,7 @@ Manage cluster members Recover MicroCloud Update and upgrade Manage the snaps +Decommission a MicroCloud ``` ## Engage with us diff --git a/doc/how-to/member_remove.md b/doc/how-to/member_remove.md index 72128ea4f..1ec39b639 100644 --- a/doc/how-to/member_remove.md +++ b/doc/how-to/member_remove.md @@ -7,6 +7,8 @@ You can remove a cluster member from a MicroCloud at any time, as long as at lea sudo microcloud remove ``` +Run {command}`microcloud status` on any of the remaining cluster members to verify the removal. + Before removing the cluster member, ensure that there are no LXD instances, storage volumes, or MicroCeph OSDs located on it. See {ref}`how to remove instances ` in the LXD documentation. @@ -31,6 +33,7 @@ If the machine is no longer reachable over the network, you can also add the `-- Removing a cluster member with `--force` will not attempt to perform any clean-up of the removed machine. All services will need to be fully re-installed before they can be re-initialized. Resources allocated to the MicroCloud like disks and network interfaces may need to be re-initialized as well. ``` +(howto-member-remove-reduce-cluster)= ## Reducing the cluster to one member When shrinking the cluster down to one member, you must also clean up the Ceph monitor map (`monmap`) before proceeding, even when using the `--force` flag.