From 7884cad658349cf33ce1bcfd78fabf860f1d70f1 Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Thu, 25 Jun 2026 10:57:52 -0700 Subject: [PATCH 1/8] doc: add note about CIS hardening with USG Signed-off-by: Elijah Greenstein (cherry picked from commit a6a50cf5963b9644b868f7ad141766b86cdf28e9) --- doc/explanation/security.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/explanation/security.md b/doc/explanation/security.md index 8ab18e742..94e550333 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -14,6 +14,8 @@ MicroCloud {ref}`further enforces security ` through th MicroCloud runs on Ubuntu and benefits from all [Ubuntu platform security measures](https://ubuntu.com/security), including kernel hardening, signed packages, and continuous security maintenance. For production environments, we recommend using a recent Ubuntu LTS release to ensure long-term support and predictable security updates. +Ubuntu LTS releases subscribed to Ubuntu Pro can use the [Ubuntu Security Guide (USG)](https://documentation.ubuntu.com/security/compliance/usg/) for CIS hardening. Refer to the LXD documentation on {ref}`lxd:howto-security-harden-cis` for related details about auditing LXD hosts with USG. + (exp-security-snaps)= ## Snaps From c1758824327e5654a441e6f003eed6d712ae725c Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Fri, 26 Jun 2026 11:40:11 -0700 Subject: [PATCH 2/8] doc: add MicroOVN security cross references Signed-off-by: Elijah Greenstein (cherry picked from commit 6722b40e2196739b568678fd9ada932a691cfe7e) --- doc/explanation/security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/explanation/security.md b/doc/explanation/security.md index 94e550333..450c0f7d0 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -48,4 +48,4 @@ The {doc}`MicroCeph security documentation `. Also see the {doc}`MicroOVN security process documentation `. +MicroOVN secures its network endpoints using the TLS protocol (version 1.2 or higher), along with P-384 elliptic curve keys. For details, refer to the MicroOVN documentation on {doc}`cryptography `, {doc}`working with TLS `, and the {doc}`MicroOVN security process `. From 54537d7b783b52976fb616811ca48c474bd8fbe9 Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Tue, 7 Jul 2026 13:24:11 -0700 Subject: [PATCH 3/8] doc: add logging section to security explanation Signed-off-by: Elijah Greenstein (cherry picked from commit ba20f65bfac1132ec2b349d64e0ea880516e58b2) --- doc/explanation/security.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/explanation/security.md b/doc/explanation/security.md index 450c0f7d0..222fc991b 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -31,10 +31,16 @@ Report potential security issues privately through GitHub by [filing a security (exp-security-microcloud)= ## MicroCloud +### Cryptography + MicroCloud manages cluster membership and encrypted communication through mTLS and certificate-based identities. When a machine joins a cluster, it verifies the cluster’s certificate fingerprint and receives the complete set of member certificates, establishing a consistent trust store. During the join process, MicroCloud uses an **explicit trust establishment mechanism** designed to prevent secret leakage and mitigate {spellexception}`man-in-the-middle` attacks. This mechanism uses a Hash-Based Message Authentication Code (HMAC) to sign the messages exchanged between the machine that initiates the join process and the joining peers. The shared secret used for joining is never transmitted over the network. The join process also enforces rate limits and session timeouts to reduce the risk of replay and brute-force attacks. For further information, refer to the [public specification](https://discourse.ubuntu.com/t/explicit-trust-establishment-mechanism-for-microcloud/44261). +### Logging + +MicroCloud creates logs through systemd. These logs can be accessed with `sudo snap logs microcloud`. + (exp-security-lxd)= ## LXD From 0e1698aab41d7b85325fa83e616797de8637007e Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Tue, 7 Jul 2026 13:48:03 -0700 Subject: [PATCH 4/8] doc: add links to security reporting policies Signed-off-by: Elijah Greenstein (cherry picked from commit 27bdc62a3e273273094930ba8605e9f585ea7902) --- doc/explanation/security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/explanation/security.md b/doc/explanation/security.md index 222fc991b..e28ead268 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -26,7 +26,7 @@ All snaps are digitally signed using {ref}`assertions Date: Tue, 7 Jul 2026 13:58:51 -0700 Subject: [PATCH 5/8] doc: add release notes links to security page Signed-off-by: Elijah Greenstein (cherry picked from commit 9c040642b8d5a5ef8a0e8fbeff2e6d2914dd07a4) --- doc/explanation/security.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/explanation/security.md b/doc/explanation/security.md index e28ead268..cd28aeabd 100644 --- a/doc/explanation/security.md +++ b/doc/explanation/security.md @@ -17,7 +17,9 @@ MicroCloud runs on Ubuntu and benefits from all [Ubuntu platform security measur Ubuntu LTS releases subscribed to Ubuntu Pro can use the [Ubuntu Security Guide (USG)](https://documentation.ubuntu.com/security/compliance/usg/) for CIS hardening. Refer to the LXD documentation on {ref}`lxd:howto-security-harden-cis` for related details about auditing LXD hosts with USG. (exp-security-snaps)= -## Snaps +## Snaps and supported versions + +The MicroCloud team maintains both Long Term Support (LTS) and feature releases. See {ref}`ref-releases-snaps` and our {ref}`ref-release-notes` for details about the currently supported releases. MicroCloud and its components are distributed as [snaps](https://snapcraft.io/docs), which enhances security by providing a confined environment with a streamlined update mechanism. Both LTS and feature channels receive regular security updates through Canonical’s official infrastructure. From 6cb268717d1ecb27c687f04239b4690b3f7a695c Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Thu, 9 Jul 2026 12:26:50 -0700 Subject: [PATCH 6/8] doc/.custom_wordlist.txt: add systemd Signed-off-by: Elijah Greenstein (cherry picked from commit 900fde89ae51fdccc3727145bf732d0cc91a1bfd) --- doc/.custom_wordlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/.custom_wordlist.txt b/doc/.custom_wordlist.txt index 27910882a..a44d59478 100644 --- a/doc/.custom_wordlist.txt +++ b/doc/.custom_wordlist.txt @@ -73,6 +73,7 @@ subfolders subnet subnets SVG +systemd Terraform TinyPNG TLS From 1977e64b35c38b29e809c14cb3d0ca6f063fb0b8 Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Tue, 7 Jul 2026 15:06:27 -0700 Subject: [PATCH 7/8] doc: add decommissioning guide Signed-off-by: Elijah Greenstein (cherry picked from commit e8265244b83eec781e174746b9be8813c5e5e72b) Amended: - Removed references to replicators and cluster links, which are only relevant for LXD 6 and MicroCloud 3. - Updated cross references as needed for labels and/or pages that are missing or in a different location in the MicroCloud 2 integrated docs. Signed-off-by: Elijah Greenstein --- doc/how-to/decommission.md | 276 ++++++++++++++++++++++++++++++++++++ doc/how-to/index.md | 1 + doc/how-to/member_remove.md | 1 + 3 files changed, 278 insertions(+) create mode 100644 doc/how-to/decommission.md diff --git a/doc/how-to/decommission.md b/doc/how-to/decommission.md new file mode 100644 index 000000000..e7566d155 --- /dev/null +++ b/doc/how-to/decommission.md @@ -0,0 +1,276 @@ +--- +myst: + html_meta: + description: Follow these steps to securely decommission a MicroCloud cluster member or cluster. +--- + +(howto-decommission)= +# How to securely decommission a MicroCloud deployment + +```{important} +This process will erase all data associated with your MicroCloud deployment. +Make copies of any data that you need to preserve before proceeding. +Refer to {ref}`lxd:instances-backup` and {ref}`lxd:howto-storage-backup-volume` for relevant details. +``` + +This guide walks you through the steps to decommission an entire MicroCloud cluster. + +If you only need to decommission a single cluster member, first {ref}`remove the member from the cluster `. +After removing the member, {ref}`update the certificate ` on the cluster remaining in production. +Then, return to this guide, skip ahead to {ref}`howto-decommission-remove-microcloud`, and follow the instructions in the sections that follow. + +Some commands used to decommission MicroCloud are LXD or MicroCeph commands. +Refer to the {ref}`LXD decommissioning guide ` and the {doc}`MicroCeph guide to removing disks ` for related information. + +(howto-decommission-remove-offline-member)= +## Remove offline cluster members + +Use the `--force` flag to {ref}`remove any offline cluster members ` (you will remove online cluster members later in the process): + +```bash +sudo microcloud remove --force +``` + +(howto-decommission-revoke-remote)= +## Revoke remote access + +List all identities that have access to LXD, then delete each identity: + +```bash +lxc auth identity list +lxc auth identity delete / +``` + +(howto-decommission-list-projects)= +## List projects + +Instances, profiles, and custom volumes are scoped by {ref}`project `. +For deployments with more than one project, you must repeat some steps for **each** project, each time using the `--project` flag. +You do not need to use the `--project` flag to decommission deployments with only one project. + +Run this command to get a list of all projects: + +```bash +lxc project list +``` + +````{note} +You can also delete a project (except the `default` project) and all of its project-level entities with: + +```bash +lxc project delete --force +``` +```` + +(howto-decommission-delete-data)= +## Delete data + +You can run the commands in this section on any online cluster member to delete data. + +```{important} +Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks. +To prevent unauthorized data recovery, you must {ref}`destroy and sanitize your data `. +``` + +(howto-decommission-delete-instances)= +### Stop and delete instances + +For each project, stop all instances: + +```bash +lxc stop --all --project +``` + +Next, for each project, list all instances, then delete each instance: + +```bash +lxc list --project +lxc delete --project +``` + +If you are unable to stop or delete an instance, use the `--force` flag: + +```bash +lxc stop --force --project +lxc delete --force --project +``` + +(howto-decommission-delete-profiles)= +### Delete profiles + +For each project, list all profiles: + +```bash +lxc profile list --project +``` + +Each project has a `default` profile that cannot be deleted. +Delete all other profiles: + +```bash +lxc profile delete --project +``` + +(howto-decommission-remove-disk-devices)= +### Remove disk devices from `default` profiles + +You cannot delete a storage pool used by an instance, profile, or custom volume. +You must, therefore, remove any disk devices used by the `default` profiles in order to delete any storage pools or custom volumes referenced by those devices. + +At a minimum, the `default` profile of the `default` project has a disk device named `root` that references a storage pool. +Remove this device with: + +```bash +lxc profile device remove default root --project default +``` + +To check for additional disk devices, view information about the `default` profile of each project: + +```bash +lxc profile show default --project +``` + +Remove any remaining disk devices that reference storage pools: + +```bash +lxc profile device remove default --project +``` + +(howto-decommission-delete-volumes)= +### Delete custom volumes + +To delete {ref}`custom volumes `, you must specify the storage pools used by the volumes. +First, list all storage pools across projects: + +```bash +lxc storage list +``` + +Next, for each storage pool, list the custom volumes. +Use the `--all-projects` flag to view all custom volumes across projects: + +```bash +lxc storage volume list type=custom --all-projects +``` + +Use the `PROJECT` column in the output to identify the project associated with each custom volume. +Then delete each custom volume, specifying both the storage pool and the project: + +```bash +lxc storage volume delete --project +``` + +(howto-decommission-delete-pools)= +### Delete storage pools + +```{note} +Storage pools are not scoped by project, so you do not need to use the `--project` flag with `lxc storage` commands. +``` + +List all storage pools, then delete each one: + +```bash +lxc storage list +lxc storage delete +``` + +(howto-decommission-delete-monitoring)= +### Delete monitoring data + +Delete data from any external systems that you used to monitor {ref}`LXD events `, {ref}`LXD metrics `, or {doc}`Ceph logging `, such as [Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), or [Grafana](https://grafana.com/). +Refer to the documentation for those systems for details. + + +(howto-decommission-remove-microceph-osds)= +## Remove MicroCeph OSDs + +To {doc}`remove MicroCeph OSDs `, list all disks, then remove each one: + +```bash +microceph disk list +sudo microceph disk remove +``` + +Finally, verify that the OSDs have been removed: + +``` +microceph disk list +``` + +````{note} +If you are unable to remove an OSD, use the `--bypass-safety-checks` flag: + +```bash +sudo microceph disk remove --bypass-safety-checks +``` +```` + + +(howto-decommission-remove-remaining-members)= +## Remove remaining cluster members + +After deleting data, you can remove the online cluster members from the cluster. +First, list all cluster members: + +```bash +microcloud cluster list +``` + +You can then {ref}`remove most cluster members ` with: + +```bash +sudo microcloud remove +``` + +However, before reducing the cluster from two members to one member, you must {ref}`clean up the Ceph monitor map `. + +```{note} +As you remove each member, you can run `microcloud status` on the remaining cluster members to verify the removal. +``` + +(howto-decommission-remove-microcloud)= +## Remove snaps + +```{important} +Run these commands on **every** machine that you decommission. + +Removing MicroCloud **does not** erase {ref}`ZFS pools (zpools) ` or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs). +To securely decommission MicroCloud, you must {ref}`destroy and sanitize your data `. +``` + +Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps. +Use the `--purge` flag, or a snapshot of your data will be preserved: + +```bash +sudo snap remove microcloud --purge +sudo snap remove lxd --purge +sudo snap remove microceph --purge +sudo snap remove microovn --purge +``` + +```{note} +The MicroCeph and MicroOVN snaps may not be installed if you deployed a MicroCloud without those components. +``` + +Verify that the snaps and associated data were removed. +The following commands should report that none of these snaps are installed and that the `/var/snap/microcloud/`, `/var/snap/lxd/`, `/var/snap/microceph/`, and `/var/snap/microovn` directories do not exist: + +```bash +snap list microcloud lxd microceph microovn +ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/ +``` + +(howto-decommission-destroy-data)= +## Destroy and sanitize data + +Data deleted with MicroCloud, LXD, or MicroCeph commands remains readable and can be recovered by users with access to disks used in your deployment. +To prevent unauthorized recovery, you must physically overwrite the data. +Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning. + +If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics. +For clusters {ref}`configured with OIDC `, consult your OIDC identity provider for the steps to remove any data associated with your profile. +Likewise, if you used {ref}`ACME services to issue server certificates `, refer to the service provider for the steps to remove any associated data. + +```{important} +Sanitized data is irreversibly destroyed and cannot be recovered. +``` diff --git a/doc/how-to/index.md b/doc/how-to/index.md index e4b4c9589..543b53a4d 100644 --- a/doc/how-to/index.md +++ b/doc/how-to/index.md @@ -58,6 +58,7 @@ Manage cluster members Recover MicroCloud Update and upgrade Manage the snaps +Decommission a MicroCloud ``` ## Engage with us diff --git a/doc/how-to/member_remove.md b/doc/how-to/member_remove.md index 72128ea4f..4d8d34c93 100644 --- a/doc/how-to/member_remove.md +++ b/doc/how-to/member_remove.md @@ -31,6 +31,7 @@ If the machine is no longer reachable over the network, you can also add the `-- Removing a cluster member with `--force` will not attempt to perform any clean-up of the removed machine. All services will need to be fully re-installed before they can be re-initialized. Resources allocated to the MicroCloud like disks and network interfaces may need to be re-initialized as well. ``` +(howto-member-remove-reduce-cluster)= ## Reducing the cluster to one member When shrinking the cluster down to one member, you must also clean up the Ceph monitor map (`monmap`) before proceeding, even when using the `--force` flag. From eb158cf398cc1a7f71f049f70971e6079dd1a238 Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Fri, 10 Jul 2026 13:16:40 -0700 Subject: [PATCH 8/8] doc: add sentence on how to verify member removal Signed-off-by: Elijah Greenstein (cherry picked from commit 40aa374934b8d948778f7428676f0c6b500bc67e) --- doc/how-to/member_remove.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/how-to/member_remove.md b/doc/how-to/member_remove.md index 4d8d34c93..1ec39b639 100644 --- a/doc/how-to/member_remove.md +++ b/doc/how-to/member_remove.md @@ -7,6 +7,8 @@ You can remove a cluster member from a MicroCloud at any time, as long as at lea sudo microcloud remove ``` +Run {command}`microcloud status` on any of the remaining cluster members to verify the removal. + Before removing the cluster member, ensure that there are no LXD instances, storage volumes, or MicroCeph OSDs located on it. See {ref}`how to remove instances ` in the LXD documentation.