diff --git a/mkdocs.yml b/mkdocs.yml index 971fd5e..396267c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -13,7 +13,8 @@ nav: - How to plan an installation: how-to/install/planning.md - Version requirements: how-to/install/version-requirements.md - Dependencies on operator’s machine: how-to/install/dependencies.md - - How to install Wire in a box (Demo): how-to/install/demo-wiab.md + - How to install Wire in a box (Dev): how-to/install/wiab-dev.md + - How to install Wire in a box (Staging): how-to/install/wiab-staging.md - Introduction: how-to/install/prod-intro.md - How to install kubernetes and databases: how-to/install/ansible-VMs.md - How to configure AWS services: how-to/install/aws-prod.md diff --git a/src/how-to/install/README.md b/src/how-to/install/README.md index c0b0c0b..b93b6d7 100644 --- a/src/how-to/install/README.md +++ b/src/how-to/install/README.md @@ -2,8 +2,13 @@ ## 1. Planning an Installation - [How to plan an installation](planning.md) - - [Demo installation (trying functionality out)](planning.md#demo-installation-trying-functionality-out) + - [Deployment options at a glance](planning.md#deployment-options-at-a-glance) + - [WIAB Dev (single-VM Wire-in-a-Box)](planning.md#wiab-dev-single-vm-wire-in-a-box) + - [WIAB Staging (multi-VM KVM-based test cluster)](planning.md#wiab-staging-multi-vm-kvm-based-test-cluster) - [Production installation (persistent data, high-availability)](planning.md#production-installation-persistent-data-high-availability) + - [Capacity and scalability](planning.md#capacity-and-scalability) + - [Artifact bundle and offline deployment](planning.md#artifact-bundle-and-offline-deployment) + - [Next steps for high-available production installation](prod-intro.md#next-steps-for-high-available-production-installation) ## 2. Version Requirements - [Version requirements](version-requirements.md) @@ -13,21 +18,35 @@ - [Dependencies on operator’s machine](dependencies.md) - [(Alternative) Installing dependencies using Direnv and Nix](dependencies.md#alternative-installing-dependencies-using-direnv-and-nix) -## 4. How to install Wire in a box (Demo) -- [Demo Wire-in-a-Box Deployment Guide](demo-wiab.md) - - [Introduction](demo-wiab.md#introduction) - - [What will be installed?](demo-wiab.md#what-will-be-installed) - - [Wire Demo installation diagram](demo-wiab.md#diagram) - - [Installation Guide](demo-wiab.md#installation-guide) - - [Deployment requirements](demo-wiab.md#deployment-requirements) - - [Getting Started](demo-wiab.md#getting-started) - - [Deployment Flow](demo-wiab.md#deployment-flow) - - [General Tips](demo-wiab.md#general-tips) - - [Trying Things Out](demo-wiab.md#trying-things-out) - - [Troubleshooting](demo-wiab.md#troubleshooting) - - [Cleaning/Uninstalling Wire-in-a-Box](demo-wiab.md#cleaninguninstalling-wire-in-a-box) - -## 5. Introduction +## 4. How to install Wire in a box (Dev) +- [WIAB Dev (Wire-in-a-Box development) Deployment Guide](wiab-dev.md) + - [Introduction](wiab-dev.md#introduction) + - [What will be installed?](wiab-dev.md#what-will-be-installed) + - [Relation to WIAB Staging and Production](wiab-dev.md#relation-to-wiab-staging-and-production) + - [Architecture overview](wiab-dev.md#architecture-overview) + - [Deployment requirements](wiab-dev.md#deployment-requirements) + - [Getting Started](wiab-dev.md#getting-started) + - [Deployment Flow](wiab-dev.md#deployment-flow) + - [Ansible notes](wiab-dev.md#ansible-notes) + - [Troubleshooting](wiab-dev.md#troubleshooting) + - [Cleaning/Uninstalling Wire-in-a-Box](wiab-dev.md#cleaninguninstalling-wire-in-a-box) + +## 5. How to install Wire in a box (Staging) +- [WIAB Staging Deployment Guide](wiab-staging.md) + - [Introduction](wiab-staging.md#introduction) + - [Architecture overview](wiab-staging.md#architecture-overview) + - [Relation to production](wiab-staging.md#relation-to-production) + - [Requirements](wiab-staging.md#requirements) + - [VM Provisioning](wiab-staging.md#vm-provisioning) + - [Getting the Ansible playbooks](wiab-staging.md#getting-the-ansible-playbooks) + - [When VMs are ready](wiab-staging.md#when-vms-are-ready) + - [Kubernetes & Data Services Deployment](wiab-staging.md#kubernetes-and-data-services-deployment) + - [Helm Operations to install Wire services and supporting Helm charts](wiab-staging.md#helm-operations-to-install-wire-services-and-supporting-helm-charts) + - [Network Traffic Configuration](wiab-staging.md#network-traffic-configuration) + - [cert-manager behaviour in NAT / bridge environments](wiab-staging.md#cert-manager-behaviour-in-nat-or-bridge-environments) + - [Next steps and troubleshooting](wiab-staging.md#next-steps-and-troubleshooting) + +## 6. Introduction - [Introduction](prod-intro.md) - [What will be installed by following these parts?](prod-intro.md#what-will-be-installed-by-following-these-parts) - [What will not be installed?](prod-intro.md#what-will-not-be-installed) @@ -35,21 +54,26 @@ - [Getting support](prod-intro.md#getting-support) - [Next steps for high-available production installation](prod-intro.md#next-steps-for-high-available-production-installation) -## 6. Installing Kubernetes and Databases -- [How to install kubernetes and databases](ansible-VMs.md) +## 7. Installing Kubernetes and Databases +- [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md) - [Introduction](ansible-VMs.md#introduction) - - [Assumptions](ansible-VMs.md#assumptions) - - [Preparing to run ansible](ansible-VMs.md#preparing-to-run-ansible) - - [Running ansible to install software on your machines](ansible-VMs.md#running-ansible-to-install-software-on-your-machines) + - [Downloading and extracting the artifact](ansible-VMs.md#downloading-and-extracting-the-artifact) + - [Editing the inventory](ansible-VMs.md#editing-the-inventory) + - [Generating secrets for the services](ansible-VMs.md#generating-secrets-for-the-services) + - [Deploying Kubernetes and stateful services](ansible-VMs.md#deploying-kubernetes-and-stateful-services) + - [Deploying secondary k8s cluster for calling services](ansible-VMs.md#deploying-secondary-k8s-cluster-for-calling-services) + - [Post Installation checks](ansible-VMs.md#post-installation-checks) + - [Next steps for high-available production installation](ansible-VMs.md#next-steps-for-high-available-production-installation) -## 7. Configuring AWS Services + +## 8. Configuring AWS Services - [How to configure AWS services](aws-prod.md) - [Introduction](aws-prod.md#introduction) - [Using real AWS services for SNS](aws-prod.md#using-real-aws-services-for-sns) - [Using real AWS services for SES / SQS](aws-prod.md#using-real-aws-services-for-ses-sqs) - [Using real AWS services for S3](aws-prod.md#using-real-aws-services-for-s3) -## 8. Installing Wire-Server using Helm +## 9. Installing Wire-Server using Helm - [How to install wire-server using Helm](helm-prod.md) - [Introduction](helm-prod.md#introduction) - [Prerequisites](helm-prod.md#prerequisites) @@ -57,7 +81,7 @@ - [Watching changes as they happen](helm-prod.md#watching-changes-as-they-happen) - [General installation notes](helm-prod.md#general-installation-notes) - [How to install charts that provide access to external databases](helm-prod.md#how-to-install-charts-that-provide-access-to-external-databases) - - [Fake AWS (SNS/SQS) for websocket-only notifications (no FCM/APNS)](helm-prod.md##fake-aws-snssqs-for-websocket-only-notifications-no-fcmapns) + - [Fake AWS (SNS/SQS) for websocket-only notifications (no FCM/APNS)](helm-prod.md#fake-aws-snssqs-for-websocket-only-notifications-no-fcmapns-on-android-devices) - [Preparing to install wire-server](helm-prod.md#preparing-to-install-wire-server) - [How to install RabbitMQ](helm-prod.md#how-to-install-rabbitmq) - [How to configure real SMTP (email) services](helm-prod.md#how-to-configure-real-smtp-email-services) @@ -65,7 +89,7 @@ - [How to install wire-server itself](helm-prod.md#how-to-install-wire-server-itself) - [DNS records](helm-prod.md#dns-records) -## 9. Multi-Ingress Configuration +## 10. Multi-Ingress Configuration - [How to configure a Wire deployment for Multi-Ingress](multi-ingress.md) - [What is a Multi-ingress setup?](multi-ingress.md#what-is-a-multi-ingress-setup) - [How is multi-ingress useful?](multi-ingress.md#how-is-multi-ingress-useful) @@ -77,7 +101,7 @@ - [Calling](multi-ingress.md#calling) - [How to setup VPS for each domain between a client and wire-backend](multi-ingress.md#how-to-setup-vps-for-each-domain-between-a-client-and-wire-backend) -## 10. Infrastructure Configuration +## 11. Infrastructure Configuration - [Infrastructure configuration](infrastructure-configuration.md) - [Redirect some traffic through a http(s) proxy](infrastructure-configuration.md#redirect-some-traffic-through-a-http-s-proxy) - [Enable push notifications using the public appstore / playstore mobile Wire clients](infrastructure-configuration.md#enable-push-notifications-using-the-public-appstore-playstore-mobile-wire-clients) @@ -99,11 +123,11 @@ - [S3 Addressing Style](infrastructure-configuration.md#s3-addressing-style) - [I have a team larger than 500 users](infrastructure-configuration.md#i-have-a-team-larger-than-500-users) -## 11. Monitoring Wire-Server +## 12. Monitoring Wire-Server - [How to monitor wire-server](monitoring.md) - [Dashboards](monitoring.md#dashboards) -## 12. Centralized Logs for Wire-Server +## 13. Centralized Logs for Wire-Server - [How to see centralized logs for wire-server](logging.md) - [Introduction](logging.md#introduction) - [Status](logging.md#status) @@ -115,28 +139,28 @@ - [Nuking it all.](logging.md#nuking-it-all) - [Debugging](logging.md#debugging) -## 13. Ingress-Controller (Getting Traffic In) +## 14. Ingress-Controller (Getting Traffic In) - [Ingress-controller (getting traffic in)](ingress.md) - [Installing in a cloud-like environment](ingress.md#installing-in-a-cloud-like-environment) - [Installing on bare-metal without dynamic load balancer support](ingress.md#installing-on-bare-metal-without-dynamic-load-balancer-support) -## 14. Web App Settings +## 15. Web App Settings - [Web app settings](web-app-settings.md) - [Enforce desktop application only](web-app-settings.md#enforce-desktop-application-only) - [Enforce constant bit rate](web-app-settings.md#enforce-constant-bit-rate) - [Disable media plugins](web-app-settings.md#disable-media-plugins) - [Enable extra entropy (only on Windows)](web-app-settings.md#enable-extra-entropy-only-on-windows) -## 15. Installing Conference Calling 2.0 (aka SFT) +## 16. Installing Conference Calling 2.0 (aka SFT) - [Installing Conference Calling 2.0 (aka SFT)](sft.md) - [Background](sft.md#background) -## 16. Installing Restund +## 17. Installing Restund - [Installing Restund](restund.md) - [Background](restund.md#background) - [Installation instructions](restund.md#installation-instructions) -## 17. Configure TLS Ciphers +## 18. Configure TLS Ciphers - [Configure TLS ciphers](tls.md) - [Ingress Traffic (wire-server)](tls.md#ingress-traffic-wire-server) - [Egress Traffic (wire-server/federation)](tls.md#egress-traffic-wire-server-federation) @@ -146,16 +170,16 @@ - [Restund (ansible)](tls.md#restund-ansible) - [Restund (kubernetes)](tls.md#restund-kubernetes) -## 18. Managing Authentication with Ansible +## 19. Managing Authentication with Ansible - [Managing authentication with ansible](ansible-authentication.md) - [How to use password authentication when you ssh to a machine with ansible](ansible-authentication.md#how-to-use-password-authentication-when-you-ssh-to-a-machine-with-ansible) - [Configuring SSH keys](ansible-authentication.md#configuring-ssh-keys) - [Sudo without password](ansible-authentication.md#sudo-without-password) -## 19. Using Tinc +## 20. Using Tinc - [Using tinc](ansible-tinc.md) -## 20. Troubleshooting During Installation +## 21. Troubleshooting During Installation - [Troubleshooting during installation](troubleshooting.md) - [Problems with CSP on the web based applications (webapp, team-settings, account-pages)](troubleshooting.md#problems-with-csp-on-the-web-based-applications-webapp-team-settings-account-pages) - [Problems with ansible and python versions](troubleshooting.md#problems-with-ansible-and-python-versions) @@ -167,10 +191,7 @@ - [Diagnosing issues with installation steps.](troubleshooting.md#diagnosing-issues-with-installation-steps) - [Verifying correct deployment of DNS / DNS troubleshooting.](troubleshooting.md#verifying-correct-deployment-of-dns-dns-troubleshooting) -## 21. Verifying Your Installation +## 22. Verifying Your Installation - [Verifying your installation](post-install.md) - [NTP Checks](post-install.md#ntp-checks) - [Logs and Data Protection checks](post-install.md#logs-and-data-protection-checks) - - [Demo installation (trying functionality out)](planning.md#demo-installation-trying-functionality-out) - - [Production installation (persistent data, high-availability)](planning.md#production-installation-persistent-data-high-availability) - diff --git a/src/how-to/install/ansible-VMs.md b/src/how-to/install/ansible-VMs.md index 1807361..24c74a2 100644 --- a/src/how-to/install/ansible-VMs.md +++ b/src/how-to/install/ansible-VMs.md @@ -1,340 +1,425 @@ - - # Installing kubernetes and databases on VMs with ansible ## Introduction -In a production environment, some parts of the wire-server -infrastructure (such as e.g. cassandra databases) are best configured -outside kubernetes. Additionally, kubernetes can be rapidly set up with -kubespray, via ansible. This section covers installing VMs with ansible. - -## Assumptions +In a production environment, some parts of the wire-server infrastructure (such as e.g. Cassandra, PostgresSQL, RabbitMQ, Minio etc databases) are best configured outside kubernetes. Additionally, kubernetes can be rapidly set up with kubespray, via ansible. This section covers installing k8s services and databases with ansible. -- A bare-metal setup (no cloud provider) -- All machines run ubuntu 18.04 -- All machines have static IP addresses -- Time on all machines is being kept in sync -- You have the following virtual machines: +Please ensure that we have created VMs as per the [production architecture](planning.md#production-installation-persistent-data-high-availability), we would be using this information in the next step of creating inventory. -| Name | Amount | CPU Cores | Memory (GB) | Disk Space (GB) | -|------------------------------------------------------|----------|--------------|---------------|-------------------| -| Cassandra | 3 | 2 | 4 | 80 | -| MinIO | 3 | 1 | 2 | 400 | -| ElasticSearch | 3 | 1 | 2 | 60 | -| postgresql | 3 | 1 | 2 | 50 | -| Kubernetes³ | 3 | 6¹ | 8 | 40 | -| Restund⁴ | 2 | 1 | 2 | 10 | -| **Per-Server Totals** | — | 12 CPU Cores | 20 GB Memory | 640 GB Disk Space | -| Admin Host² | 1 | 1 | 4 | 40 | -| Asset Host² | 1 | 1 | 4 | 100 | -| **Per-Server Totals with
Admin and Asset Hosts** | — | 14 CPU Cores | 28 GB Memory | 780 GB Disk Space | -- ¹ Kubernetes hosts may need more resources to support SFT (Conference Calling). See “Conference Calling Hardware Requirements” below. -- ² Admin and Asset Hosts can run on any one of the 3 servers, but that server must not allocate additional resources as indicated in the table above. -- ³ Etcd is run inside of Kubernetes, hence no specific resource allocation -- ⁴ Restund may be hosted on only 2 of the 3 servers, or all 3. Two nodes are enough to ensure high availability of Restund services +![Wire Server Architecture HA](img/architecture-prod-ha.png) -General Hardware Requirements +## Downloading and extracting the artifact -- Minimum 3 physical servers required -- Wire has a minimum requirement for a total of 16 Ubuntu 18.04 virtual machines across the 3 servers (in accordance with the table above) +Create a fresh workspace to download the artifacts: -Conference Calling Hardware Requirements +```bash +$ cd ... # you pick a good location! +``` +Obtain the latest airgap artifact for wire-server-deploy. Please [contact us](https://support.wire.com/hc/en-us) to get it. -- Kubernetes Hosts may need additional resources for SFT services. For concurrent SFT users (SFT = Selective Forwarding Turn-server, ie. Conference calling), we recommend an extra 3% of CPU allocation, evenly distributed across the nodes (i.e. 1% more CPU per kubernetes server). So for every 100 users plan on adding one CPU core on each Kubernetes node. The SFT component runs inside of Kubernetes, and does not require a separate virtual machine for operation. +Extract the above listed artifacts into your workspace: -(It’s up to you how you create these machines - kvm on a bare metal -machine, VM on a cloud provider, real physical machines, etc.) +```bash +$ wget https://s3-eu-west-1.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-.tgz +$ tar xvzf wire-server-deploy-static-.tgz +``` +Where `` above is the hash of your deployment artifact, given to you by Wire, or acquired by looking at the above build job. +Extract this tarball. -## Preparing to run ansible +Make sure that the admin host can `ssh` into all the machines that you want to provision. Our [Making tooling available in your environment](https://github.com/wireapp/wire-server-deploy/blob/master/offline/docs_ubuntu_22.04.md#making-tooling-available-in-your-environment) will use the `.ssh` folder and the `ssh-agent` of the user running the scripts. - +There's also a docker image containing the tooling inside this repo. -### Adding IPs to hosts.ini +## Editing the inventory -Go to your checked-out wire-server-deploy/ansible folder: +Copy [ansible/inventory/offline/99-static](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/offline/99-static) to `ansible/inventory/offline/hosts.ini`, and backup the original. -```default -cd wire-server-deploy/ansible +```bash +cp ansible/inventory/offline/99-static ansible/inventory/offline/hosts.ini ``` -Copy the example hosts file: +Edit `ansible/inventory/offline/hosts.ini`. Here, you will describe the topology of your offline deploy as explained in the next section. -```default -cp hosts.example.ini hosts.ini -``` +Add one entry in the `all` section of this file for each machine you are managing via ansible. This will be all of the machines in your Wire cluster. -- Edit the hosts.ini, setting the permanent IPs of the hosts you are - setting up wire on. -- On each of the lines declaring a database service node ( - lines in the `[all]` section beginning with cassandra, elasticsearch, - or minio) replace the `ansible_host` values (`X.X.X.X`) with the - IPs of the nodes that you can connect to via SSH. these are the - ‘internal’ addresses of the machines, not what a client will be - connecting to. -- On each of the lines declaring a kubernetes node (lines in the `[all]` - section starting with ‘kubenode’) replace the `ip` values - (`Y.Y.Y.Y`) with the IPs which you wish kubernetes to provide - services to clients on, and replace the `ansible_host` values - (`X.X.X.X`) with the IPs of the nodes that you can connect to via - SSH. If the IP you want to provide services on is the same IP that - you use to connect, remove the `ip=Y.Y.Y.Y` completely. -- On each of the lines declaring an `etcd` node (lines in the `[all]` - section starting with etcd), use the same values as you used on the - coresponding kubenode lines in the prior step. -- If you are deploying Restund for voice/video services then on each of the - lines declaring a `restund` node (lines in the `[all]` section - beginning with restund), replace the `ansible_host` values (`X.X.X.X`) - with the IPs of the nodes that you can connect to via SSH. -- Edit the minio variables in `[minio:vars]` (`prefix`, `domain` and `deeplink_title`) - by replacing `example.com` with your own domain. +If you are using username/password to log into and sudo up, in the `all:vars` section, add: +```ini +ansible_user= +ansible_password= +ansible_become_pass= +``` -There are more settings in this file that we will set in later steps. +> Note: Make sure that `assethost` is present in the inventory file with the correct `ansible_host` (and `ip` values if required) - +**SSH authentication options:** -#### WARNING -Some of these playbooks mess with the hostnames of their targets. You -MUST pick different hosts for playbooks that rename the host. If you -e.g. attempt to run Cassandra and k8s on the same 3 machines, the -hostnames will be overwritten by the second installation playbook, -breaking the first. +- If the VMs are reachable with a private key, set `ansible_ssh_private_key_file` in the inventory and run Ansible normally. +- If you rely on an SSH agent, keep `ansible_ssh_private_key_file` commented out and ensure the agent on the `adminhost` can reach all VMs. +- If you do not use a private key entry in the inventory and password authentication is enabled on the VMs, add `--ask-pass` when running ansible-playbooks manually and `--ask-become-pass` for sudo access. +- Our installation scripts are non-interactive, define `ansible_password` and `ansible_become_pass` in the inventory instead of relying on interactive password prompts. -At the least, we know that the cassandra, kubernetes and restund playbooks are -guilty of hostname manipulation. +Before running the offline deployment scripts, verify that the inventory resolves to the expected machines. The commands below assume you are running them from `/home/ansible_user/wire-server-deploy` on the `adminhost`. -### Authentication +### Updating Database Group Memberships +It's recommended to update the lists of what nodes belong to which group, so ansible knows what to install on these nodes. -#### NOTE -If you use ssh *keys*, and the user you login with is either root or can elevate to root without a password, you don’t need to do anything further to use ansible. If, however, you use password authentication for ssh access, and/or your login user needs a password to become root, see [Manage ansible authentication settings](ansible-authentication.md#ansible-authentication). +These sections are divided into individual host groups, reflecting the architecture of the target infrastructure. Examples with individual nodes for Elastic, MinIO, PostgreSQL, RabbitMQ and Cassandra are commented out below. +```ini +[elasticsearch] +elasticsearch1 +elasticsearch2 +elasticsearch3 -## Running ansible to install software on your machines +[minio] +minio1 +minio2 +minio3 -You can install kubernetes, cassandra, restund, etc in any order. +[cassandra] +cassandra1 +cassandra2 +cassandra3 -#### NOTE -In case you only have a single network interface with public IPs but wish to protect inter-database communication, you may use the `tinc.yml` playbook to create a private network interface. In this case, ensure tinc is setup BEFORE running any other playbook. See [tinc](ansible-tinc.md#tinc) +[cassandra_seed] +cassandraseed1 -### Installing kubernetes +[postgresql] +postgresql1 +postgresql2 +postgresql3 -Kubernetes is installed via ansible. +[postgresql_rw] +postgresql1 -To install kubernetes: +[postgresql_ro] +postgresql2 +postgresql3 -From `wire-server-deploy/ansible`: +[rmq-cluster] +rabbitmq1 +rabbitmq2 +rabbitmq3 -```default -ansible-playbook -i hosts.ini kubernetes.yml -vv ``` -When the playbook finishes correctly (which can take up to 20 minutes), you should have a folder `artifacts` containing a file `admin.conf`. Copy this file: - -```default -mkdir -p ~/.kube -cp artifacts/admin.conf ~/.kube/config -``` +### Configuring kubernetes and etcd -Make sure you can reach the server: +To run Kubernetes, at least three nodes are required, which need to be added to the `[kube-master]`, `[etcd]` and `[kube-node]` groups of the inventory file. Any additional nodes should only be added to the `[kube-node]` group, for example: -```default -kubectl version +```ini +[kube-master] +kubemaster1 +kubemaster2 +kubemaster3 + +[etcd] +etcd1 etcd_member_name=etcd1 +etcd2 etcd_member_name=etcd2 +etcd3 etcd_member_name=etcd3 + +[kube-node] +prodnode1 +prodnode2 +prodnode3 +prodnode4 ``` -should give output similar to this: +### Setting up databases and kubernetes to talk over the correct (private) interface +If you are deploying wire on servers that are expected to use one interface to talk to the public, and a separate interface to talk amongst themselves, you will need to add "ip=" declarations for the private interface of each node. for instance, if the first kubenode was expected to talk to the world on 192.168.122.21, but speak to other wire services (kubernetes, databases, etc) on 192.168.0.2, you should edit its entry like the following: +``` +kubenode1 ansible_host=192.168.122.21 ip=192.168.0.2 +``` +Repeat this for all of the instances. -```default -Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} -Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} +### Setting up Database network interfaces and service specific variables: +- Make sure that `cassandra_network_interface` is set to the name of the network interface on which the kubenodes should talk to cassandra and on which the cassandra nodes should communicate among each other. Run `ip addr` on one of the cassandra nodes to determine the network interface names, and which networks they correspond to. In Ubuntu 22.04 for example, interface names are predictable and individualized, eg. `enp41s0`. +- Similarly `elasticsearch_network_interface`, `rabbitmq_network_interface`, `postgresql_network_interface` and `minio_network_interface` should be set to the network interface names to the service specific groups to ensure communicatation with kubernetes and among each other. +- RabbitMQ requires the variable `rabbitmq_cluster_master` to configure one of the `rmq-cluster` nodes as master, the value should be the `hostname` for master node. +- PostgreSQL requires following variables to define the database topology and database to create. These has been pre-filled with default values - modify them as per your environment. +```ini +wire_dbname= wire-server +repmgr_node_config = {"postgresql1": {"node_id": 1, "priority": 150, "role": "primary"}, "postgresql2": {"node_id": 2, "priority": 100, "role": "standby"}, "postgresql3": {"node_id": 3, "priority": 50, "role": "standby"}} ``` +- In an INI inventory, the `repmgr_node_config` keys must match the PostgreSQL inventory hostnames. +- To read more about configurations related to each data service read at: + - [PostgreSQL](../administrate/postgresql-cluster.md) + - [Cassandra](../administrate/cassandra.md) + - [Elasticsearch](../administrate/elasticsearch.md) + - [Minio](../administrate/minio.md) + - [RabbitMQ](https://github.com/wireapp/wire-server-deploy/blob/master/offline/rabbitmq_setup.md) + - [Kubernetes](../administrate/kubernetes/README.md) -### Cassandra +### Example hosts.ini -- If you would like to change the name of the cluster, in your - ‘hosts.ini’ file, in the `[cassandra:vars]` section, uncomment - the line that changes ‘cassandra_clustername’, and change default - to be the name you want the cluster to have. -- If you want cassandra nodes to talk to each other on a specific - network interface, rather than the one you use to connect via SSH, - In your ‘hosts.ini’ file, in the `[all:vars]` section, - uncomment, and set ‘cassandra_network_interface’ to the name of - the ethernet interface you want cassandra nodes to talk to each - other on. For example: +Here is an example hosts.ini file for the primary k8s cluster and database services. ```ini +[all:vars] +ansible_user= +ansible_password= +ansible_become_pass= + +[assethost] +assethost ansible_host=10.1.1.1 + +[cassandra] +cassandra1 ansible_host=10.1.1.17 +cassandra2 ansible_host=10.1.1.2 +cassandra3 ansible_host=10.1.1.16 + [cassandra:vars] -# cassandra_clustername: default +cassandra_network_interface=enp7s0 -[all:vars] -## set to True if using AWS -is_aws_environment = False -## Set the network interface name for cassandra to bind to if you have more than one network interface -cassandra_network_interface = eth0 -``` +[cassandra_seed] +cassandra3 -(see -[defaults/main.yml](https://github.com/wireapp/ansible-cassandra/blob/master/defaults/main.yml) -for a full list of variables to change if necessary) +[elasticsearch] +elasticsearch1 ansible_host=10.1.1.9 +elasticsearch2 ansible_host=10.1.1.15 +elasticsearch3 ansible_host=10.1.1.19 -- Use ansible to deploy Cassandra: +[elasticsearch:vars] +elasticsearch_network_interface=enp7s0 -```default -ansible-playbook -i hosts.ini cassandra.yml -vv -``` +[elasticsearch_master:children] +elasticsearch -### ElasticSearch +[kube-node] +kubenode1 ansible_host=10.1.1.3 etcd_member_name=kubenode1 ip=10.1.1.3 +kubenode2 ansible_host=10.1.1.4 etcd_member_name=kubenode2 ip=10.1.1.4 +kubenode3 ansible_host=10.1.1.8 etcd_member_name=kubenode3 ip=10.1.1.8 -- In your ‘hosts.ini’ file, in the `[all:vars]` section, uncomment - and set ‘elasticsearch_network_interface’ to the name of the - interface you want elasticsearch nodes to talk to each other on. -- If you are performing an offline install, or for some other reason - are using an APT mirror other than the default to retrieve - elasticsearch-oss packages from, you need to specify that mirror - by setting ‘es_apt_key’ and ‘es_apt_url’ in the `[all:vars]` - section of your hosts.ini file. +[kube-master:children] +kube-node -```ini -[all:vars] -# default first interface on ubuntu on kvm: -elasticsearch_network_interface=ens3 +[etcd:children] +kube-master + +[k8s-cluster:children] +kube-master +kube-node + +[k8s-cluster:vars] +calico_mtu=1450 +calico_veth_mtu=1430 + +[minio] +minio1 ansible_host=10.1.1.6 +minio2 ansible_host=10.1.1.7 +minio3 ansible_host=10.1.1.20 + +[minio:vars] +minio_network_interface=enp7s0 -## Set these in order to use an APT mirror other than the default. -# es_apt_key = "https:///linux/ubuntu/gpg" -# es_apt_url = "deb [trusted=yes] https:///apt bionic stable" +[postgresql] +postgresql1 ansible_host=10.1.1.11 +postgresql2 ansible_host=10.1.1.5 +postgresql3 ansible_host=10.1.1.12 + +[postgresql:vars] +postgresql_network_interface=enp7s0 +wire_dbname=wire-server +repmgr_node_config={"postgresql1":{"node_id":1,"priority":150,"role":"primary"},"postgresql2":{"node_id":2,"priority":100,"role":"standby"},"postgresql3":{"node_id":3,"priority":50,"role":"standby"}} + +[postgresql_ro] +postgresql2 +postgresql3 + +[postgresql_rw] +postgresql1 + +[rmq-cluster] +rabbitmq1 ansible_host=10.1.1.18 +rabbitmq2 ansible_host=10.1.1.13 +rabbitmq3 ansible_host=10.1.1.14 + +[rmq-cluster:vars] +rabbitmq_cluster_master=rabbitmq3 +rabbitmq_network_interface=enp7s0 ``` -- Use ansible and deploy ElasticSearch: +### More checks on Inventory + +> **Note:** If your environment uses a non-standard MTU (e.g. cloud providers, VPNs, or overlay networks), you must [configure the MTU](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/calico.md#configuring-interface-mtu) for Calico in `k8s-cluster.vars`. Ensure all VMs have the same MTU on their primary interface: +> ```bash +> ip link show +> ``` +> Then set: +> ```ini +> [k8s-cluster:vars] +> calico_mtu= +> calico_veth_mtu= +> ``` +> As a rule of thumb: +> - `calico_mtu = underlying network MTU - encapsulation overhead` +> - `calico_veth_mtu` ≤ `calico_mtu` + +Using following commands we can verify if our inventory is functional: -```default -ansible-playbook -i hosts.ini elasticsearch.yml -vv +```bash +# confirm the inventory hostnames match the actual VM hostnames +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'hostname' + +# verify the default IPv4 interface and address reported by Ansible +d ansible all -i ansible/inventory/offline/inventory.yml -m setup -a 'filter=ansible_default_ipv4' + +# verify time and timezone consistency across the machines +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'date' + +# verify if the MTU is consistent across all the VMs +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a "ip link show | grep mtu" ``` -### Minio +If any hostname, IP address, SSH setting, or interface name is wrong at this stage, correct `ansible/inventory/offline/inventory.yml` before continuing. The next deployment steps assume this inventory is accurate. -Minio is used for asset storage, in the case that you are not -running on AWS infrastructure, or feel uncomfortable storing assets -in S3 in encrypted form. If you are using S3 instead of Minio, skip -this step. +## Generating secrets for the services -- In your ‘hosts.ini’ file, in the `[all:vars]` section, make sure - you set the ‘minio_network_interface’ to the name of the interface - you want minio nodes to talk to each other on. The default from the - playbook is not going to be correct for your machine. +Minio and coturn services have shared secrets with the `wire-server` helm chart. Run the folllowing script that generates a fresh set of secrets for these components: -```ini -[all:vars] -# Default first interface on ubuntu on kvm: -minio_network_interface=ens3 +```bash +./bin/offline-secrets.sh ``` -#### Configure Access Key and Secret Key for MinIO and Cargohold Service +This should generate 3 secret files as: +- `ansible/inventory/group_vars/all/secrets.yaml` - This file will be used by ansible minio playbooks to configure the service secrets. +- `values/wire-server/prod-secrets.example.yaml` - This contains the secrets for Wire services and share some secrets from coturn and database services. +- `values/coturn/prod-secrets.example.yaml` - This contains a secret for the coturn service. +- `values/kube-prometheus-stack/prod-secrets.example.yaml` - This contains secret for Prometheus in kube-prometheus-stack. Only applicable when deploying the `kube-prometheus-stack` helm chart with `auth` enabled for prometheus. -**Purpose**: Configure a secure, least-privilege access method for the Cargohold service to utilize the MinIO object storage. +## Deploying Kubernetes and stateful services -**Security Model**: -- **MinIO root credentials**: Used only for administrative purposes -- **Cargohold IAM user**: A least privileged user with a policy that only gives access to the `assets` bucket -- **Service account**: Separate access/secret key pair for Cargohold service operations +In order to deploy all mentioned services, run: +``` +d ./bin/offline-cluster.sh +``` -## Setup Process +This wrapper runs the following Ansible playbooks in order: -1. **Generate credentials**: Run `./bin/offline-secrets.sh` from the wire-server-deploy directory. +1. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/setup-offline-sources.yml`: prepares the `assethost`, copies offline artifacts, and configures the other hosts to fetch packages and images from it. +2. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/kubernetes.yml --tags bastion,bootstrap-os,preinstall,container-engine`: runs the first Kubernetes bootstrap phase so the container runtime is ready. +3. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/seed-offline-containerd.yml`: loads the offline container images onto the nodes after the runtime is available. +4. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/sync_time.yml -v`: installs and configures time synchronization before the rest of the cluster comes up. +5. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/kubernetes.yml --skip-tags bootstrap-os,preinstall,container-engine,multus`: finishes the remaining Kubernetes deployment after the prerequisites are in place. +6. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/cassandra.yml`: deploys the Cassandra nodes. +7. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/elasticsearch.yml`: deploys Elasticsearch. +8. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/minio.yml`: deploys MinIO. +9. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/postgresql-deploy.yml`: deploys the PostgreSQL cluster. +10. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/roles/rabbitmq-cluster/tasks/configure_dns.yml`: prepares DNS entries required by the RabbitMQ cluster. +11. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/rabbitmq.yml`: deploys RabbitMQ. +12. `d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/helm_external.yml`: writes the external service IPs into the Helm values files so the charts can target those services. This step is a [pre-requiste](#before-installing-helm-charts) before continuing with helm operations. - This generates a `secrets.yaml` file in `ansible/inventory/offline/group_vars/all/` with: - ```yaml - minio_access_key: "" - minio_secret_key: "" - minio_cargohold_access_key: "" - minio_cargohold_secret_key: "" - ``` +The order matters: offline package sources and container runtime must be ready before image seeding, time sync should happen before the cluster stabilizes, Kubernetes must exist before the rest of the platform is wired around it, and `helm_external.yml` comes last because it depends on the database and messaging nodes already being deployed. -2. **For existing Wire systems** - Backup and regenerate secrets: - ```bash - # Backup current secrets file - cp ansible/inventory/offline/group_vars/all/secrets.yaml \ - ansible/inventory/offline/group_vars/all/secrets.yaml.backup +If one step fails and you want to run the playbooks manually, use the `d` alias shown above and execute the specific command directly in the same order. - # Remove current secrets and generate new ones - rm ansible/inventory/offline/group_vars/all/secrets.yaml - ./bin/offline-secrets.sh - ``` +### Ensuring Kubernetes is healthy. -3. **Migration step**: Replace the newly generated `minio_access_key` and `minio_secret_key` with the values from `secrets.yaml.backup` to maintain compatibility. +Ensure the k8s cluster comes up healthy. The container also contains `kubectl`, so check the node status: -4. **Deploy MinIO configuration**: - ```bash - ansible-playbook -i hosts.ini minio.yml -vv - ``` +```bash +d kubectl get nodes -owide +``` +They should all report ready. + +### Troubleshooting external services +Cassandra, Minio, PostgresSQL, RabbitMQ and Elasticsearch are running outside Kubernets cluster, make sure those machines have necessary ports open - + +On each of the machines running Cassandra, Minio, PostgresSQL, RabbitMQ and Elasticsearch, run the following commands to open the necessary ports, if necessary (Verify if ports are open by default): +```bash +sudo bash -c ' +set -eo pipefail; + +# cassandra +ufw allow 9042/tcp; +ufw allow 9160/tcp; +ufw allow 7000/tcp; +ufw allow 7199/tcp; + +# elasticsearch +ufw allow 9300/tcp; +ufw allow 9200/tcp; + +# minio +ufw allow 9000/tcp; +ufw allow 9092/tcp; + +#rabbitmq +ufw allow 5671/tcp; +ufw allow 5672/tcp; +ufw allow 4369/tcp; +ufw allow 25672/tcp; + +#postgresql +ufw allow 5432/tcp; +' +``` -5. **Update Cargohold service configuration** in `values/wire-server/secrets.yaml`: - ```yaml - cargohold: - secrets: - # Replace with values from ansible/inventory/offline/group_vars/all/secrets.yaml - awsKeyId: dummykey # replace with minio_cargohold_access_key - awsSecretKey: dummysecret # replace with minio_cargohold_secret_key - ``` +## Deploying secondary k8s cluster for calling services -6. **Deploy updated Wire Server**: - ```bash - helm upgrade --install wire-server ./charts/wire-server \ - --timeout=15m0s \ - --values ./values/wire-server/values.yaml \ - --values ./values/wire-server/secrets.yaml - ``` +Now we are done with configuring primary k8s cluster and databases. Now, we would be installing a secondary k8s cluster. -This configures the Cargohold service with its IAM user credentials to securely manage the `assets` bucket. +![Calling Architecture HA](img/architecture-calling-ha.png) -### Postgresql cluster +### Marking kubenodes for calling servers (SFT/Coturn) -To set up a high-availability PostgreSQL cluster for Wire Server, refer to the [PostgreSQL High Availability Cluster - Quick Setup](../administrate/postgresql-cluster.md) guide. This will walk you through deploying a three-node PostgreSQL cluster with automatic failover capabilities using repmgr. +The SFT & Coturn Calling server should be running on a kubernetes nodes that are connected to the network where they are reachable to your clients. If client network is undefined and clients connect from all over public Internet, then these nodes should be connected to public network. If not all kubernetes nodes match the network criteria, you should specifically labels on nodes that match these criteria and similary nodeSelector criteria for SFT helm chart, so that you're sure SFT is deployed correctly. -The setup includes: -- Three PostgreSQL nodes (one primary, two replicas) -- Automatic failover and split-brain protection -- Integration with Kubernetes via the postgres-endpoint-manager +By using a `node_label` you can make sure SFT & Coturn are only deployed on certain nodes like `call_kubenode1` & `call_kubenode2`: -After deploying the PostgreSQL cluster, make sure to install the `postgresql-external` helm chart as described in the guide to integrate it with your Wire Server deployment. +```ini +[all:vars] +ansible_user= +ansible_password= +ansible_become_pass= -### Restund +[assethost] +assethost ansible_host=10.1.1.1 -For instructions on how to install Restund, see [this page](restund.md#install-restund). +[kube-node] +call_kubenode1 ansible_host=10.1.1.33 etcd_member_name=call_kubenode1 ip=10.1.1.33 node_labels="{'wire.com/role': 'sftd'}" node_annotations="{'wire.com/external-ip': 'a.b.c.d'}" +call_kubenode2 ansible_host=10.1.1.34 etcd_member_name=call_kubenode2 ip=10.1.1.34 node_labels="{'wire.com/role': 'coturn'}" +call_kubenode3 ansible_host=10.1.1.36 etcd_member_name=call_kubenode3 ip=10.1.1.36 -### IMPORTANT checks +[kube-master:children] +kube-node + +[etcd:children] +kube-master + +[k8s-cluster:children] +kube-master +kube-node +``` + +If the node is not bound to the public IP the users will see(e.g. becuase it's behind NAT) then you should also set the `wire.com/external-ip` annotation to the public IP of the node. + +## Post Installation checks > After running the above playbooks, it is important to ensure that everything is setup correctly. Please have a look at the post install checks in the section [Verifying your installation](post-install.md#checks) -```default +```bash ansible-playbook -i hosts.ini cassandra-verify-ntp.yml -vv ``` -### Installing helm charts - prerequisites +### Before installing helm charts -The `helm_external.yml` playbook is used to write or update the IPs of the -databases servers in the `values/-external/values.yaml` files, and -thus make them available for helm and the `-external` charts (e.g. -`cassandra-external`, `elasticsearch-external`, etc). +The `helm_external.yml` playbook is used to write or update the IPs of the databases servers in the `values/-external/values.yaml` files, and thus make them available for helm and the `-external` charts (e.g. `cassandra-external`, `elasticsearch-external`, `minio-external`, `postgresql-external` etc). -Due to limitations in the playbook, make sure that you have defined the -network interfaces for each of the database services in your hosts.ini, -even if they are running on the same interface that you connect to via SSH. -In your hosts.ini under `[all:vars]`: +Due to limitations in the playbook, make sure that you have defined the network interfaces for each of the database services in your hosts.ini, even if they are running on the same interface that you connect to via SSH. -```ini -[all:vars] -minio_network_interface = ... -cassandra_network_interface = ... -elasticsearch_network_interface = ... -# if you're using redis external... -redis_network_interface = ... -``` +> **Note:** If you have already ran the script [/bin/offline-cluster.sh](#deploying-kubernetes-and-stateful-services) then this playbook might have already been ran for you. You can confirm this by looking into the database specific helm values, if they have entries for each database service: +> - `values/cassandra-external/values.yaml` +> - `values/elasticsearch-external/values.yaml` +> - `values/minio-external/values.yaml` +> - `values/postgresql-external/values.yaml` +> - `values/rabbitmq-external/values.yaml` Now run the helm_external.yml playbook, to populate network values for helm: -```default -ansible-playbook -i hosts.ini -vv --diff helm_external.yml +```bash +d ansible-playbook -i ansible/inventory/offline/hosts.ini ansible/helm_external.yml ``` +You can now can continue with the installation of helm charts. -You can now can install the helm charts. - -#### Next steps for high-available production installation +## Next steps for high-available production installation -Your next step will be [Installing wire-server (production) components using Helm](helm-prod.md#helm-prod) +Your next step will be [Installing wire-server (production) components using Helm](helm-prod.md) diff --git a/src/how-to/install/dependencies.md b/src/how-to/install/dependencies.md index 217bb6d..26bfb6d 100644 --- a/src/how-to/install/dependencies.md +++ b/src/how-to/install/dependencies.md @@ -1,5 +1,3 @@ - - # Dependencies on operator’s machine In order to operate a wire-server installation, you’ll need a bunch of software @@ -46,7 +44,7 @@ bash-4.4# ansible --version ansible 2.9.12 ``` -Once you’re in there, you can move on to [installing kubernetes](ansible-VMs.md#installing-kubernetes). +Once you’re in there, you can move on to [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md). ## (Alternative) Installing dependencies using Direnv and Nix diff --git a/src/how-to/install/helm-prod.md b/src/how-to/install/helm-prod.md index 729772e..bbce66c 100644 --- a/src/how-to/install/helm-prod.md +++ b/src/how-to/install/helm-prod.md @@ -34,7 +34,7 @@ You need to have docker on the machine you are using to perform this installatio - FIXME - If you don’t have a Kubernetes cluster, you have two options: - You can get access to a managed Kubernetes cluster with the cloud provider of your choice. - - You can install one if you have ssh access to a set of sufficiently large virtual machines, see [installing kubernetes](ansible-VMs.md#installing-kubernetes). + - You can install one if you have ssh access to a set of sufficiently large virtual machines, see [Production installation (persistent data, high-availability)](planning.md#production-installation-persistent-data-high-availability). - If you don’t have `helm` yet, see [Installing helm](https://helm.sh/docs/using_helm/#installing-helm). If you followed the instructions in [Dependencies on operator’s machine](dependencies.md#dependencies) should have helm installed already. Type `helm version`, you should, if everything is configured correctly, see a result similar this: diff --git a/src/how-to/install/img/architecture-calling-ha.drawio b/src/how-to/install/img/architecture-calling-ha.drawio new file mode 100644 index 0000000..d6c9ae8 --- /dev/null +++ b/src/how-to/install/img/architecture-calling-ha.drawio @@ -0,0 +1,148 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/how-to/install/img/architecture-calling-ha.png b/src/how-to/install/img/architecture-calling-ha.png new file mode 100644 index 0000000..48dd665 Binary files /dev/null and b/src/how-to/install/img/architecture-calling-ha.png differ diff --git a/src/how-to/install/img/architecture-prod-ha.drawio b/src/how-to/install/img/architecture-prod-ha.drawio new file mode 100644 index 0000000..7ce1a9f --- /dev/null +++ b/src/how-to/install/img/architecture-prod-ha.drawio @@ -0,0 +1,372 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/how-to/install/img/architecture-prod-ha.png b/src/how-to/install/img/architecture-prod-ha.png new file mode 100644 index 0000000..27f684f Binary files /dev/null and b/src/how-to/install/img/architecture-prod-ha.png differ diff --git a/src/how-to/install/img/architecture-server-ha.drawio b/src/how-to/install/img/architecture-server-ha.drawio deleted file mode 120000 index dec25c4..0000000 --- a/src/how-to/install/img/architecture-server-ha.drawio +++ /dev/null @@ -1 +0,0 @@ -../../../../wire-server/docs/src/how-to/install/img/architecture-server-ha.drawio \ No newline at end of file diff --git a/src/how-to/install/img/architecture-server-ha.png b/src/how-to/install/img/architecture-server-ha.png deleted file mode 120000 index 8320d7d..0000000 --- a/src/how-to/install/img/architecture-server-ha.png +++ /dev/null @@ -1 +0,0 @@ -../../../../wire-server/docs/src/how-to/install/img/architecture-server-ha.png \ No newline at end of file diff --git a/src/how-to/install/img/architecture-wiab-dev.drawio b/src/how-to/install/img/architecture-wiab-dev.drawio new file mode 100644 index 0000000..f975719 --- /dev/null +++ b/src/how-to/install/img/architecture-wiab-dev.drawio @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/src/how-to/install/img/architecture-wiab-dev.png b/src/how-to/install/img/architecture-wiab-dev.png new file mode 100644 index 0000000..0bfd9a2 Binary files /dev/null and b/src/how-to/install/img/architecture-wiab-dev.png differ diff --git a/src/how-to/install/img/architecture-wiab-stag.drawio b/src/how-to/install/img/architecture-wiab-stag.drawio new file mode 100644 index 0000000..07180e7 --- /dev/null +++ b/src/how-to/install/img/architecture-wiab-stag.drawio @@ -0,0 +1,247 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/how-to/install/img/architecture-wiab-stag.png b/src/how-to/install/img/architecture-wiab-stag.png new file mode 100644 index 0000000..9c5c5fd Binary files /dev/null and b/src/how-to/install/img/architecture-wiab-stag.png differ diff --git a/src/how-to/install/logging.md b/src/how-to/install/logging.md index a7cd88c..4ebf2d7 100644 --- a/src/how-to/install/logging.md +++ b/src/how-to/install/logging.md @@ -14,8 +14,8 @@ Logging support is in active development as of September 2019, some logs may not You need to have wire-server installed, see either of -- [How to install Wire in a box (Demo)](demo-wiab.md) -- [Installing wire-server (production) components using Helm](helm-prod.md#helm-prod). +- [How to install WIAB Dev (Wire-in-a-Box development)](wiab-dev.md) +- [Installing wire-server (production) components using Helm](helm-prod.md). ## Installing required helm charts diff --git a/src/how-to/install/multi-ingress.md b/src/how-to/install/multi-ingress.md index 65d00ad..920ab4c 100644 --- a/src/how-to/install/multi-ingress.md +++ b/src/how-to/install/multi-ingress.md @@ -51,7 +51,7 @@ flowchart LR ## Requirements - A Wire backend installation (> [5.14](https://github.com/wireapp/wire-server/releases/tag/v2025-04-07) to enable domain specific deeplinks) -- DNS records for domains to be used with all the required [sub-domains](../install/demo-wiab.md#dns-requirements). In the examples below, we demonstrate with **3 domains** (`green.example.org`, `red.example.com`, `blue.example.net`), but this setup can be **extended to as many domains as needed**: +- DNS records for domains to be used with all the required [sub-domains](includes/helm_dns-ingress-troubleshooting.inc.md#how-to-set-up-dns-records). In the examples below, we demonstrate with **3 domains** (`green.example.org`, `red.example.com`, `blue.example.net`), but this setup can be **extended to as many domains as needed**: - `green.example.org` (default domain) - `red.example.com` (additional domain) - `blue.example.net` (additional domain) diff --git a/src/how-to/install/planning.md b/src/how-to/install/planning.md index 108ecbc..10d025c 100644 --- a/src/how-to/install/planning.md +++ b/src/how-to/install/planning.md @@ -1,58 +1,71 @@ # Implementation plan -There are two types of implementation: demo and production. -## Demo installation (trying functionality out) +There are three types of installation targets for Wire: **WIAB Dev (formerly WIAB Demo)**, **WIAB Staging**, and **Production**. This page helps you choose the right one and points you to the corresponding guides. -Please note that there is no way to migrate data from a demo -installation to a production installation - it is really meant as a way -to try things out. +## Deployment options at a glance -Please note your data will be in-memory only and may disappear at any given moment! +| Option | Best for | Scale & Topology | Persistent Data | Typical Audience | +|-----------------|-------------------------------------------|-------------------------------------------|------------------------|--------------------------------------| +| WIAB (Wire-in-a-Box) Dev | Quick functional demo on a single VM | 1 Ubuntu 24.04 VM with Minikube (without hypervisor) | No (ephemeral data) | Developers, PoC | +| WIAB (Wire-in-a-Box) Staging | Realistic multi-VM, non-production setup | ~7 Ubuntu 22.04 VMs on 1 physical host | Yes (multi-service) | Infra/ops testing, training | +| Production | High-availability, long-term deployment | Separate DB VMs + 2 Kubernetes clusters distributed in 3 Availability zones | Yes (HA,failure domains etc)| Production operations | -### What you need: +- **WIAB Dev** is a self-contained, single-node demo environment (previously called *WIAB Demo*). All core services run on one VM, using in-cluster, ephemeral datastores. It is ideal for trying out functionality but **not suitable for production**. +- **WIAB Staging** reproduces the production architecture using a fixed set of virtual machines hosted on a single physical box. It is persistent but still a single failure domain and therefore **not production-grade**. +- **Production** uses dedicated virtual machines for databases and a Kubernetes cluster for Wire services. It is designed for **durability, scale, and high availability**. -- **System**: Ubuntu 24.04 (Focal) on amd64 architecture with following requirements: - - CPU cores >= 16 - - Memory > 16 GiB - - Disk > 200 GiB -- **DNS Records**: - - a way to create DNS records for your domain name (e.g. wire.example.com) - - Find a detailed explanation at [How to set up DNS records](demo-wiab.md#dns-requirements) -- **SSL/TLS certificates**: - - a way to create SSL/TLS certificates for your domain name (to allow connecting via https://) - - To ease out the process of managing certs, we recommend using [Let\'s Encrypt](https://letsencrypt.org/getting-started/) & -[cert-manager](https://cert-manager.io/docs/tutorials/acme/http-validation/) -- **Network**: No interference from UFW or other system specific firewalls, and IP forwarding enabled between network cards. Public internet access to download Wire artifacts and Ubuntu packages. -- **Packages**: Ansible and unzip (or git) on the localhost (any machine you have access to) - - Ansible version: [core 2.16.3] or compatible - - Note: The deployment will automatically install additional required packages on the deploy_node (see [Package Installation](#4-package-installation) section). You can skip this step using `--skip-tags install_pkgs` if these packages are already installed -- **Permissions**: Sudo access required for installation on remote_node -- **Ansible Playbooks**: - - The `ansible` directory from [wire-server-deploy repository](https://github.com/wireapp/wire-server-deploy) - - Obtain it using **either** method: - - **Download as ZIP:** [wire-server-deploy/archive/master.zip](https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip) (requires unzip) - - **Clone with Git:** `git clone https://github.com/wireapp/wire-server-deploy.git` (requires git) -- **Network Access Requirements**: +> **Important:** There is no supported migration path from WIAB Dev or WIAB Staging to Production. Treat them as disposable environments. -| Protocol | Port(s) | Purpose | -|----------|-------------|--------------------------------------------| -| TCP | 22 | SSH access (for remote management) | -| TCP | 80 | HTTP (certificate renewal) | -| TCP | 443 | HTTPS (primary Wire access) | -| TCP | 3478 | Alternative STUN/TURN traffic | -| UDP | 3478 | STUN/TURN for voice/video calls | -| UDP | 32768-65535 | Voice/video calling traffic (Coturn/SFTD) | +## WIAB Dev (single-VM Wire-in-a-Box) -- Note: If outbound traffic is restricted, [Note on port ranges](https://docs.wire.com/latest/understand/notes/port-ranges.html) should be followed. +**Use this if:** +- You want to see Wire running quickly. +- You are evaluating features or doing development integration. +- You are happy to lose the data when you tear down the VM or during restart. -### Next steps for demo installation +**Key characteristics:** -When the above [requirements](#what-you-need) are achieved, continue with the ansible playbook instructions for the [demo wire in a box](demo-wiab.md) installation. +- Single Ubuntu 24.04 VM (No hypervisor support required) +- Minikube-based Kubernetes cluster runs on the same node. +- Ephemeral datastores (no external DBs or storage appliances required). +- Includes webapp, account pages, team settings page, demo SMTP, coturn, and a cert-manager with Let’s Encrypt (unless you choose bring-your-own certificates). +- Intended for roughly **15-50 users** in small test teams; this is guidance only, not a performance benchmark. +![Wire in a Box Dev Architecture](img/architecture-wiab-dev.png) - +For full demo architecture, requirements, deployment steps, and cleanup instructions, see: + +- **Guide:** [`WIAB Dev (Demo Wire-in-a-Box) Deployment`](wiab-dev.md) + +> **Internet requirement for WIAB Dev:** The WIAB Dev playbook assumes that the deploy_node has outbound internet connectivity during installation. Tooling such as Minikube, Docker, kubectl, Python packages, and system packages are downloaded from public repositories and are **not shipped inside the artifact bundle**. After installation, you can lock down outbound access more aggressively, but Let’s Encrypt certificate renewal (if used) will still require periodic internet access. + +## WIAB Staging (multi-VM KVM-based test cluster) + +**Use this if:** + +- You want to rehearse a production-style deployment or upgrade process. +- You want realistic separation of roles (kubernetes vs data services vs assets) on multiple VMs. +- You want a staging Wire server with stateful services. + +**Key characteristics:** + +- One physical machine with hardware virtualization support (KVM). +- Typically **7 VMs** (Ubuntu 22.04) to simulate a production layout: + - `assethost` – asset and artifact host. + - `kubenode1–3` – Kubernetes control-plane and worker nodes running Wire services and calling. + - `datanode1–3` – data services such as Cassandra, RabbitMQ, PostgreSQL, MinIO and Elasticsearch. +- Wire is deployed using an offline artifact bundle (`wire-server-deploy`), Ansible, and Helm. +- Suitable for up to **100 users** with teams smaller than **100 users per team**; capacity here is for staging realism, not for production sizing. + +![Wire in a Box Staging Architecture](img/architecture-wiab-stag.png) + +> **Internet requirement for WIAB Staging provisioning:** If you use the WIAB Staging provisioning Ansible playbook to create the VMs on your physical host, that provisioning step will require internet access to download the supporting tools and images it needs (for example VM cloud images and KVM/libvirt tooling). Once the VMs and the Wire artifact are in place, subsequent deployment stages can run with limited or no internet, depending on your TLS/certificate setup. + +For detailed requirements, VM sizing, playbook usage, and follow-up cluster deployment steps, see: + +- **Guide:** [`WIAB Staging Deployment`](wiab-staging.md) ## Production installation (persistent data, high-availability) @@ -60,54 +73,92 @@ What you need: - a way to create **DNS records** for your domain name (e.g. `wire.example.com`) - a way to create **SSL/TLS certificates** for your domain name (to allow connecting via `https://wire.example.com`) -- A **kubernetes cluster with at least 3 worker nodes and at least 3 etcd nodes** (some cloud providers offer a managed kubernetes cluster these days) -- minimum **17 virtual machines** for components outside kubernetes (cassandra, minio, elasticsearch, redis, restund) +- two **kubernetes clusters with at least 3 worker nodes each** (some cloud providers offer a managed kubernetes cluster these days) + - First (primary) kubernetes cluster will be used by wire-server and supporting services. Second kubernetes cluster aka secondary would be used for calling workload. Calling infrastructure is designed to operate in a Zero trust / DMZ environment, so that the networking requirements of calling can be separated from the sensitive data stored in your Wire backend (primary) k8s cluster. Read more about Wire architecture [here](../../understand/overview.md#backend-routing) +- minimum **15 virtual machines** for components outside kubernetes (Cassandra, MinIO, Elasticsearch, RabbitMQ, PostgreSQL) -A recommended installation of Wire-server in any regular data centre, -configured with high-availability will require the following virtual -servers: +A recommended installation of Wire-server in any regular data centre, configured with high-availability will require the following virtual servers: -| Name | Amount | CPU Cores | Memory (GB) | Disk Space (GB) | +| Name | VM Count | CPU Cores per VM | Memory (GB) per VM | Disk Space (GB) per VM| |------------------------------------------------------|----------|--------------|---------------|-------------------| -| Cassandra | 3 | 2 | 4 | 80 | +| Cassandra | 3 | 1 | 2 | 80 | +| postgreSQL | 3 | 1 | 2 | 80 | | MinIO | 3 | 1 | 2 | 400 | | ElasticSearch | 3 | 1 | 2 | 60 | -| Kubernetes³ | 3 | 6¹ | 8 | 40 | -| Restund⁴ | 2 | 1 | 2 | 10 | -| **Per-Server Totals** | — | 11 CPU Cores | 18 GB Memory | 590 GB Disk Space | -| Admin Host² | 1 | 1 | 4 | 40 | -| Asset Host² | 1 | 1 | 4 | 100 | -| **Per-Server Totals with
Admin and Asset Hosts** | — | 13 CPU Cores | 26 GB Memory | 730 GB Disk Space | -- ¹ Kubernetes hosts may need more ressources to support SFT (Conference Calling). See “Conference Calling Hardware Requirements” below. -- ² Admin and Asset Hosts can run on any one of the 3 servers, but that server must not allocate additional resources as indicated in the table above. -- ³ Etcd is run inside of Kubernetes, hence no specific resource allocation -- ⁴ Restund may be hosted on only 2 of the 3 servers, or all 3. Two nodes are enough to ensure high availability of Restund services +| RabbitMQ | 3 | 2 | 4 | 80 | +| Primary Kubernetes (k8s control/worker node) | 3 | 6 | 8 | 80 | +| Admin Host | 1 | 1 | 4 | 90 | +| Asset Host | 1 | 1 | 4 | 100 | +| Stateful Services Totals per physical node | - | 14 CPU Cores | 28 GB Memory | 970 GB Disk Space | +| Secondary Kubernetes (Calling Services) | 3 | 4 | 10 | 70 | +| Single Server Totals | - | 18 CPU Cores | 38 GB Memory | 1040 GB Disk Space | + +> Notes: +> - Secondary kubernetes hosts may need more resources to support heavy conference calling. For concurrent SFT users (SFT = «Selective Forwarding Turn» server, ie. Conference calling), we recommend an extra 3% of CPU allocation, evenly distributed across the nodes (i.e. 1% more CPU per kubernetes server). So for every 100 users plan on adding one CPU core on each Kubernetes node. The SFT component runs inside of Kubernetes, and does not require a separate virtual machine for operation. +> - Admin Host and Asset Host can run on any one of the 3 servers, but the respective server must allocate additional resources as indicated in the table above. +> - Wire requires the usage of CPUs built on the AMD64 architecture, and assumes these are running at the equivalent CPU/Memory performance of a recent Intel/AMD server system clocked at at least 2.5 Ghz. For reference, please refer to https://cpubenchmark.net/high_end_cpus.html +> - Systems running inside a virtualized environment must have nested hardware virtualization enabled, with support for a hardware hypervisor such as KVM (or an equivalent). +> - Each Virtual core should correspond to a reservation of one physical core, not a thread. +> - Note that some performance will be lost with mitigations for Spectre, Meltdown, and other security patching, so plan reserve capacity as appropriate. +> - Disk space requirements are referring to formatted, local disk space. The use of network file storage solutions may be incompatible with Wire deployment and operation. +> - Disk space must be SSD or NVMe. We recommend against using spinning disks. + +General Hardware Requirements: +- Minimum 3 physical servers required in 3 availability zones. +- Wire has a minimum requirement for a total of 23 Ubuntu 22.04 virtual machines across the 3 servers (in accordance with the table above) + +### Capacity and scalability + +This architecture is specified to support **4,000 or fewer messaging (chat, IM) users**, divided into teams of **less than 1,500 users per team**. For larger deployments, a modified architecture and additional capacity planning may be advised. + +This infrastructure is **scalable**. The 4,000-user figure is an initial sizing point, not a hard upper bound. You can scale out further (more nodes, more resources) based on your requirements. For a full infrastructure guide and scaling plan tailored to your use case, contact **Wire deployment support** via . -General Hardware Requirements +A production installation will look a bit like this: -- Minimum 3 physical servers required -- Wire has a minimum requirement for a total of 16 Ubuntu 18.04 virtual machines across the 3 servers (in accordance with the table above) +![Wire Server Architecture HA](img/architecture-prod-ha.png) +![Calling Architecture HA](img/architecture-calling-ha.png) -Conference Calling Hardware Requirements +If you use a private datacenter (not a cloud provider), the easiest is to have three physical servers, each with one virtual machine for each server component (cassandra, minio, elasticsearch, rabbitmq, postgreSQL). -- Kubernetes Hosts may need additional resources for SFT services. For concurrent SFT users (SFT = Selective Forwarding Turn-server, ie. Conference calling), we recommend an extra 3% of CPU allocation, evenly distributed across the nodes (i.e. 1% more CPU per kubernetes server). So for every 100 users plan on adding one CPU core on each Kubernetes node. The SFT component runs inside of Kubernetes, and does not require a separate virtual machine for operation. +It’s up to you how you create these VMs - kvm on a bare metal machine, VM on a cloud provider, etc. Make sure they run ubuntu 22.04. -A production installation will look a bit like this: +In a typical production network layout: + +- Core messaging components and databases run in a protected cluster. +- Calling services (coturn/SFTD) are deployed on nodes or a cluster placed in a **DMZ** or dedicated edge network, with: + - Public reachability on UDP 3478 and the configured media port range. + - Controlled, internal connectivity back to the messaging cluster for signalling. + +See [Backend Routing](../../understand/overview.md#backend-routing) for more details on how messaging and calling traffic are separated in production. + +For detailed production cluster installation steps, see: + +- **Guide:** [`Production Installation`](prod-intro.md) + +## Artifact bundle and offline deployment -![image](img/architecture-server-ha.png) +An **artifact** is a collection of binaries, Ansible playbooks, Bash scripts, Helm charts, and container images that has been verified to work together and is compatible for deploying a specific version of Wire Server. Having an artifact also allows deployments in **private or offline environments** without needing direct access to the public internet. -If you use a private datacenter (not a cloud provider), the easiest is -to have three physical servers, each with one virtual machine for each -server component (cassandra, minio, elasticsearch, redis, kubernetes, -restund) +A typical production or WIAB Staging artifact bundle contains (folder names may vary slightly by release): -It’s up to you how you create these VMs - kvm on a bare metal machine, -VM on a cloud provider, etc. Make sure they run ubuntu 18.04. +- `containers-adminhost/wire-server-deploy-*.tar` - A container image containing ansible, helm, and other tools and their dependencies in versions verified to be compatible with the current wire stack. Published to quay.io/wire/wire-server-deploy as well, but shipped in the artifacts tarball for convenience. +- `ansible/` – These contain all the ansible playbooks the rest of the guide refers to, as well as an example inventory, which should be configured according to the environment this is installed into. +- `bin/` – helper scripts for Ansible runs, VM provisioning, secret generation, and Helm operations. +- `binaries.tar` – This contains static binaries, both used during the kubespray-based kubernetes bootstrapping, as well as to provide some binaries that are installed during other ansible playbook runs. +- `charts/` – Helm charts for Wire and supporting components. We don't use an external helm repository, every helm chart dependency is resolved already. +- `containers-helm.tar` – These are the container images our charts (and charts we depend on) refer to. Also come as tarballs, and are seeded like the system containers. +- `containers-system.tar` – These are the container images needed to bootstrap kubernetes itself (currently using kubespray). +- `dashboards/` – Grafana dashboard definitions. +- `debs-jammy.tar` – This acts as a self-contained dump of all packages required to install kubespray, as well as all other packages that are installed by ansible playbooks on nodes that don't run kubernetes. There's an ansible playbook copying these assets to an "assethost", starting a little webserver there serving it, and configuring all nodes to use it as a package repo. +- `values/` – Contains helm chart values and secrets. Needs to be tweaked to the environment. +- `versions/` – version metadata for images and binaries. -Ensure that your VMs have IP addresses that do not change. +The default Ansible inventories on the **master** branch of `wire-server-deploy` reference **pre-configured artifacts** for **WIAB Dev** and **WIAB Staging**. These bundles differ per solution type and are tied to a specific Wire backend version and its dependencies. -Avoid `10.x.x.x` network address schemes, and instead use something like `192.168.x.x` or `172.x.x.x`. This is because internally, Kubernetes already uses a `10.x.x.x` address scheme, creating a potential conflict. +If the default bundle for WIAB Dev or WIAB Staging does not meet your needs (for example, you require a different Wire version), please reach out to **Wire support**. They can provide a download link for a compatible artifact bundle to assist with your deployment. -### Next steps for high-available production installation +Relationship to each solution: -Your next step will be [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md#ansible-vms) +- **WIAB Dev (Demo)** – uses the bundle primarily for Helm charts and container images, but still requires internet access for system and tooling packages during installation (see the WIAB Dev guide). +- **WIAB Staging** – relies heavily on the artifact for offline deployment of Kubernetes components, data services, and Wire itself after the initial VM provisioning. +- **Production** – can use artifact bundles to perform fully offline or tightly controlled deployments where direct internet access from the cluster nodes is not allowed. diff --git a/src/how-to/install/prod-intro.md b/src/how-to/install/prod-intro.md index 0bbfbd2..14fb475 100644 --- a/src/how-to/install/prod-intro.md +++ b/src/how-to/install/prod-intro.md @@ -1,26 +1,31 @@ # Introduction -#### WARNING -It is *strongly recommended* to have followed and completed the demo installation [How to install Wire in a box (Demo)](demo-wiab.md) before continuing with this page. The demo installation is simpler, and already makes you aware of a few things you need (TLS certs, DNS, a VM, …). +## WARNING +It is *strongly recommended* to have followed and completed the wiab staging installation [How to install Wire in a box Staging](wiab-staging.md) before continuing with this page. The wiab-staging installation is simpler, and already makes you aware of a few things you need like TLS certs, DNS, VMs, ansible and helm chart etc. -#### NOTE +## NOTE All required dependencies for doing an installation can be found here [Dependencies on operator’s machine](dependencies.md#dependencies). +## Getting started with installation + +![Wire Server Architecture HA](img/architecture-prod-ha.png) +![Calling Architecture HA](img/architecture-calling-ha.png) + A production installation consists of several parts: -Part 1 - you’re on your own here, and need to create a set of VMs as detailed in [Production installation (persistent data, high-availability)](planning.md#planning-prod) +Part 1 - you’re on your own here, and need to create a set of VMs as detailed in [Production installation (persistent data, high-availability)](planning.md#production-installation-persistent-data-high-availability) -Part 2 ([Installing kubernetes and databases on VMs with ansible](ansible-VMs.md#ansible-vms)) deals with installing components directly on a set of virtual machines, such as kubernetes itself, as well as databases. It makes use of ansible to achieve that. +Part 2 ([Installing kubernetes and databases on VMs with ansible](ansible-VMs.md)) deals with installing components directly on a set of virtual machines, such as kubernetes itself, as well as databases. It makes use of ansible to achieve that. -Part 3 ([Installing wire-server (production) components using Helm](helm-prod.md#helm-prod)) is similar to the demo installation, and uses the tool `helm` to install software on top of kubernetes. +Part 3 ([Installing wire-server (production) components using Helm](helm-prod.md)) is similar to the demo installation, and uses the tool `helm` to install software on top of kubernetes. -Part 4 ([Infrastructure configuration options](infrastructure-configuration.md#configuration-options)) details other possible configuration options and settings to fit your needs. +Part 4 ([Infrastructure configuration options](infrastructure-configuration.md)) details other possible configuration options and settings to fit your needs. ## What will be installed by following these parts? - highly-available and persistent databases (cassandra, elasticsearch) - kubernetes -- restund (audio/video calling) servers ( see also [Restund (TURN) servers](../../understand/restund.md#understand-restund)) +- - Calling services i.e. coturn & SFTD - wire-server (API) - user accounts, authentication, conversations - assets handling (images, files, …) @@ -29,7 +34,7 @@ Part 4 ([Infrastructure configuration options](infrastructure-configuration.md#c - wire-webapp - fully functioning web client (like `https://app.wire.com`) - wire-account-pages - - user account management (a few pages relating to e.g. password reset) +- team-settings page for team management ## What will not be installed? @@ -38,7 +43,6 @@ Part 4 ([Infrastructure configuration options](infrastructure-configuration.md#c ## What will not be installed by default? - 3rd party proxying - requires accounts with third-party providers -- team-settings page for team management (including invitations, requires access to a private repository - get in touch with us for access) ## Getting support @@ -46,4 +50,4 @@ Part 4 ([Infrastructure configuration options](infrastructure-configuration.md#c ## Next steps for high-available production installation -Your next step will be part 2, [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md#ansible-vms) +Your next step will be part 2, [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md) diff --git a/src/how-to/install/demo-wiab.md b/src/how-to/install/wiab-dev.md similarity index 62% rename from src/how-to/install/demo-wiab.md rename to src/how-to/install/wiab-dev.md index c4a1228..372cfdb 100644 --- a/src/how-to/install/demo-wiab.md +++ b/src/how-to/install/wiab-dev.md @@ -1,8 +1,12 @@ -# Demo Wire-in-a-Box Deployment Guide +# WIAB Dev (Wire-in-a-Box development) Deployment Guide ## Introduction -The following will install a demo version of all the wire-server components including the databases. This setup is not recommended in production but will get you started. Demo version means no data persistence, as everything is stored in memory and will be lost. It does not require any external storage solutions to function. Read the section [Cleaning/Uninstalling Wire-in-a-Box](#cleaninguninstalling-wire-in-a-box) to clean the installation post testing the demo solution. +This guide provides detailed instructions for deploying Wire-in-a-Box (WIAB Dev) using Ansible on an Ubuntu 24.04 system. The deployment corresponds to **WIAB Dev (previously called *WIAB Demo*)** in the [planning overview](planning.md#wiab-dev-single-vm-wire-in-a-box). It is a single-node, Minikube-based environment intended for functional testing and evaluation only. + +This will install the wire-server components including the databases in a fashion ready for development or evaluation. This setup is not recommended for production use, but will get you started with learning the product. This configuration has NO data persistence, as everything is stored in memory and will be lost if restarted. It does not require any external storage solutions to function. + +> Read the section [Cleaning/Uninstalling Wire-in-a-Box](#cleaninguninstalling-wire-in-a-box) to clean the installation once your testing or evaluation is complete. ### What will be installed? @@ -12,63 +16,33 @@ The following will install a demo version of all the wire-server components incl - notifications over websocket - Wire-webapp, a fully functioning web client (like `https://app.wire.com`) - Wire-account-pages, user account management (a few pages relating to e.g. password reset), team-settings page -- Email relay service i.e. demo-smtp -- Group calling component i.e. coturn +- Email relay service i.e. smtp service +- Calling services i.e. coturn & SFTD - Ephemeral datastores -- A cert-manager with `letsencrypt` as `issuser`. +- A cert-manager with `letsencrypt` as an issuer (if `use_cert_manager=true`) ### What will not be installed? - notifications over native push notifications via [FCM](https://firebase.google.com/docs/cloud-messaging/)/[APNS](https://developer.apple.com/notifications/) - persistent datastores in k8s -- highly availablity - -### Diagram -The flow diagram of the Demo setup: - -```mermaid -graph TB - Client["🖥️ Clients"] - Admin["📋 Admin
⬇️ Download wire-server-deploy"] - - subgraph Node ["deploy_node"] - IPTables["🔄 iptables rules"] - Download["📥 Artifacts
Helm Charts
Docker Images"] - subgraph K8s ["Minikube K8s"] - Seeds["🐳 Container Images + 📦 Helm Charts
wire-server | wire-utility
databases | coturn"] - Wire["🚀 Wire Services
💬 Messaging | ☎️ Calls"] - end - end - - Admin -->|"SSH/Ansible"| IPTables - Client -->|"HTTPS/UDP"| IPTables - IPTables --> K8s - - Download --> Seeds - Seeds --> Wire - - classDef client fill:#e3f2fd,stroke:#1976d2,stroke-width:2px - classDef admin fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px - classDef network fill:#fff9c4,stroke:#f57f17,stroke-width:2px - classDef download fill:#e0f2f1,stroke:#00897b,stroke-width:2px - classDef k8s fill:#ffe0b2,stroke:#e65100,stroke-width:2px - classDef seeds fill:#ffccbc,stroke:#bf360c,stroke-width:2px - classDef wire fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px - - class Client client - class Admin admin - class IPTables network - class Download download - class K8s k8s - class Seeds seeds - class Wire wire -``` +- high availability + +### Relation to WIAB Staging and Production + +- **WIAB Staging** runs a similar set of components across multiple VMs and is suitable for more realistic infrastructure validation but without HA. See the [WIAB Staging Deployment](planning.md#wiab-staging-multi-vm-kvm-based-test-cluster) guide. +- **Production** uses dedicated database VMs and a full Kubernetes cluster to provide the maximum capacity and security. See [production installation](planning.md#production-installation-persistent-data-high-availability) for more details. -This guide provides detailed instructions for deploying Wire-in-a-Box (WIAB) using Ansible on an Ubuntu 24.04 system. The deployment process is structured into multiple blocks within the Ansible playbook, offering flexibility in execution. It is designed to configure a remote node, such as example.com (referred to as deploy_node), to install Wire with a custom domain, example.com (referred to as target_domain). These variables must be verified in the file [ansible/inventory/demo/host.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/demo/host.yml) before running the pipeline. +## Architecture overview -Note: this guide and the shipped playbooks are highly tailored to make testing straightforward on a single VM that has a public IP address. Using a public IP simplifies obtaining HTTPS certificates (for example via cert-manager HTTP challenges) and making external call configurations during tests. If you need to deploy in a private or restricted network, the playbooks can be tuned: skip or enable components via Ansible tags and adjust Helm chart values (see the `--tags` / `--skip-tags` usage below and the `values/` files generated by the playbooks). +![Wire in a Box Dev Architecture](img/architecture-wiab-dev.png) -Typically, the deployment process runs seamlessly without requiring any external flags. However, if needed, you can skip certain tasks using their associated tags. For example, if you wish to use your own certificates instead of Let's Encrypt, you can use `--skip-tags cert_manager_networking` to skip cert-manager deployment and related networking configuration. For detailed instructions, see [Bring your own certificates](https://github.com/wireapp/wire-server-deploy/blob/master/offline/docs_ubuntu_22.04.md#bring-your-own-certificates). +The deployment process is structured into multiple sections within the Ansible playbooks, offering flexibility in execution. It is designed to configure a remote node (referred to as deploy_node), to install Wire with a given domain(referred to as target_domain). These variables must be verified in the file `ansible/inventory/demo/host.yml` (as explained below) before running the pipeline. + +> **Note:** This guide and the shipped playbooks are highly tailored to make testing straightforward on a single VM that has a public IP address. +> Using a public IP simplifies obtaining HTTPS certificates (for example via cert-manager HTTP challenges) and making external call configurations during tests. +> If you need to deploy in a private or restricted network, the playbooks can be tuned accordingly: skip or enable components via Ansible tags and adjust Helm chart values (see the `--tags` / `--skip-tags` usage below and the `values/` files). + +Typically, the deployment process runs *seamlessly* without requiring any external flags. However, if needed, you can skip certain tasks using their associated tags. For example, if you wish to use your own certificates instead of Let's Encrypt, you can use `--skip-tags cert_manager_networking` to skip cert-manager deployment and related networking configuration. For more detailed instructions on each task, please refer to the [Deployment Flow section](#deployment-flow). @@ -77,32 +51,20 @@ For more detailed instructions on each task, please refer to the [Deployment Flo - **Ansible Playbooks**: - The `ansible` directory from [wire-server-deploy repository](https://github.com/wireapp/wire-server-deploy) - Obtain it using **either** method: - - **Download as ZIP:** [wire-server-deploy/archive/master.zip](https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip) (requires unzip) + - **Download as ZIP:** [wire-server-deploy/archive/master.zip](https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip) (requires unzip, wget) - **Clone with Git:** `git clone https://github.com/wireapp/wire-server-deploy.git` (requires git) - The inventory file [ansible/inventory/demo/host.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/demo/host.yml) to update and verify the following variables (required unless noted optional): - **ansible_host**: aka **deploy_node** i.e. IP address or hostname of VM where Wire will be deployed (Required) - - **ansible_user**: username to access the deploy_node (Required) + - **ansible_user**: username to access the deploy_node with sudo access (Required) - **ansible_ssh_private_key_file**: SSH key file path for ansible_user@deploy_node (Required) - - **target_domain**: The domain you want to use for wire installation eg. example.com (Required) + - **target_domain**: The domain you want to use for wire installation eg. example.com (Required). Check [How to set up DNS records](includes/helm_dns-ingress-troubleshooting.inc.md#how-to-set-up-dns-records) for more details. - **wire_ip**: Gateway IP address for Wire, could be same as deploy_node's IP (Optional). If not specified, the playbook will attempt to detect it (network ACLs permitting). If your deploy_node is only reachable on a private network, set this explicitly. - **use_cert_manager**: Controls TLS certificate management behavior (Optional, default: true) - - **true** (default): Deploys cert-manager and nginx-ingress-services for automatic HTTPS certificate generation via Let's Encrypt. This is the recommended option for most deployments with internet-accessible domains. - - **false**: Skips cert-manager deployment and nginx-ingress-services chart. When disabled, you must manually provide TLS certificates for your domain and configure ingress manually. See [Bring your own certificates](https://github.com/wireapp/wire-server/tree/develop/charts/nginx-ingress-services) for instructions. - - **artifact_hash**: Check with wire support about this value (used by the download step) - -Note: The playbook installs a comprehensive set of system tools and libraries during the `install_pkgs` tasks. See [Package Installation](#4-package-installation) for the complete list. If you already have these tools on the deploy node you may skip the `install_pkgs` tag when running the playbook. - -## DNS Requirements + - **true** (default): Deploys `cert-manager` and `nginx-ingress-services` helm chart for automatic HTTPS certificate generation via Let's Encrypt. This is the recommended option for most deployments where the target domain is publicly reachable and the deploy_node has outbound internet access. + - **false**: Skips `cert-manager` deployment and `nginx-ingress-services` chart. When disabled, you must manually provide TLS certificates for your domain and configure `nginx-ingress-services` helm chart manually. + - **artifact_hash**: A default hash is already configured to enable the testing with the latest stable version. Check with Wire support about this value, if a different version is required. Read about more about artifact at [Artifact bundle and offline deployment](planning.md#artifact-bundle-and-offline-deployment) and [Offline bundle and alternative chart-only deployment](#offline-bundle-and-alternative-chart-only-deployment) for an alertnative approach. -- two DNS records for the so-called "nginz" component of wire-server (the main REST API entry point), these are usually called `nginz-https.` and `nginz-ssl.` -- one DNS record for the asset store (images, audio files etc. that your users are sharing); usually `assets.` -- one DNS record for the webapp (equivalent of https://app.wire.com, i.e. the javascript app running in the browser), usually called `webapp.` -- one DNS record for the account pages (hosts some html/javascript pages for e.g. password reset), usually called `account.` -- one DNS record for team settings, usually called `teams.` -- one DNS record for SFTD (conference calling), usually called `sftd.` -- one DNS TXT record with the contents: `v=spf1 a mx ip4:SERVER-IP-ADDRESS-HERE -all`. It is used to define which mail servers are permitted to send emails from the domain, helping to prevent unauthorized use and enhance email security. - -**Note**: The above DNS requirements are verified in [DNS verification step](#1-dns-verification). +> **Internet connectivity requirement:** While the Wire services and other related components can be installed without continuous internet access (Thanks to the Artifact), this WIAB Dev playbook assumes that the deploy_node has outbound internet connectivity during installation to provision a k8s cluster. It downloads tooling such as Minikube, Docker, kubectl, Python packages, and system packages from public repositories, and (if `use_cert_manager=true`) contacts Let’s Encrypt via cert-manager to obtain certificates. If you must operate in a fully offline environment, consider using WIAB Staging or Production with offline artifacts instead. ## Getting Started @@ -125,21 +87,24 @@ cd wire-server-deploy **Step 2: Configure your deployment** -Edit the file `ansible/inventory/demo/host.yml` as explained in [Requirements](#requirements) to set up your deployment variables. +Edit the file `ansible/inventory/demo/host.yml` as explained in [Requirements](#deployment-requirements) to set up your deployment variables. **Step 3: Run the deployment** ```bash +# the playbook can run locally on your system against ansible_host where wire-server will be installed. The playbook can directly be used on ansible_host to configure it locally. ansible-playbook -i ansible/inventory/demo/host.yml ansible/wiab-demo/deploy_wiab.yml ``` +> Check [Custom Installation](#custom-installation) if you already have a k8s cluster and don't want to follow the full process. + ## Deployment Flow The deployment process follows these steps as defined in the main playbook: ### 1. Wire IP Access Verification (Always Runs) -- Imports [verify_wire_ip.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/verify_wire_ip.yml) to check Wire IP access +- Imports [verify_wire_ip.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/verify_wire_ip.yml) to check `wire_ip` access - **Always runs** - This step is crucial for identifying network ingress and cannot be skipped - Sets up variables (facts) for Kubernetes nodes based on the Minikube profile - If `wire_ip` is not already specified, the playbook attempts to detect it and saves it on the node @@ -149,40 +114,33 @@ The deployment process follows these steps as defined in the main playbook: The playbook starts by verifying DNS records to ensure proper name resolution: - Imports [verify_dns.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/verify_dns.yml) - Can be skipped using `--skip-tags verify_dns` -- Checks for basic DNS record requirements as explained in [DNS Requirements](#dns-requirements) +- Checks [How to set up DNS records](includes/helm_dns-ingress-troubleshooting.inc.md#how-to-set-up-dns-records) for more details. ### 3. Package Installation - Imports [install_pkgs.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/install_pkgs.yml) to install required dependencies - Can be skipped using `--skip-tags install_pkgs` +- Versions can be defined in the ansible inventory file `ansible/inventory/demo/host.yml` **Packages Installed:** - **Binaries:** - - Helm v3.15.0 (downloaded with checksum verification) - - Minikube (latest release) - - kubectl (latest stable release) + - Helm + - Minikube + - kubectl - **APT Packages:** - jq (JSON query tool) - - python3-pip (Python package manager) - python3-venv (Python virtual environments) - - python3-full (Complete Python installation) - docker-ce (Docker Container Engine) - - docker-ce-cli (Docker CLI) - containerd.io (Container runtime) -- **Python Libraries (via pip):** - - kubernetes >= 18.0.0 (Kubernetes Python client) - - pyyaml >= 5.4.1 (YAML parser) - -> **Virtual Environment Approach:** Python packages are installed in an isolated virtual environment at `/opt/ansible-venv` instead of system-wide installation. This eliminates conflicts with system Python packages and respects [PEP668](https://peps.python.org/pep-0668/) constraints on Ubuntu 24.04. The playbook automatically detects the best Python interpreter available (system Python if kubernetes is installed, or venv otherwise) and configures Ansible accordingly. If neither has the kubernetes module, it fails with clear remediation instructions. +> **Virtual Environment Approach:** Python packages are installed in an isolated virtual environment at `/opt/ansible-venv` instead of system-wide installation. This eliminates conflicts with system Python packages and respects [PEP668](https://peps.python.org/pep-0668/) constraints on Ubuntu 24.04. ### 4. SSH Key Management (Automatic Dependency) - Imports [setup_ssh.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/setup_ssh.yml) to manage SSH keys for Minikube node and SSH proxying -- **Dependency task:** This task has no tag and runs automatically when `minikube`, `asset_host`, or `seed_containers` are selected +- **Dependency task:** This task has no tag and runs automatically when `minikube`, `asset_host`, or `seed_containers` are selected (when any component that needs it is selected) - Cannot be run independently or skipped manually - it's controlled entirely by dependent components -- **Smart dependency:** SSH setup runs when any component that needs it is selected, and is automatically skipped otherwise ### 5. Minikube Cluster Configuration @@ -205,13 +163,10 @@ The playbook starts by verifying DNS records to ensure proper name resolution: ### 8. Minikube Node Inventory Setup (Automatic Dependency) -- **Dependency task:** This setup has no tag and runs automatically when `asset_host` or `seed_containers` are selected +- This setup has no tag and runs automatically when `asset_host` or `seed_containers` are selected - Adds Minikube node(s) to Ansible inventory dynamically - Extracts internal IP addresses from all Kubernetes nodes - Configures SSH proxy access to cluster nodes -- **Automatic dependency:** Runs when `asset_host` or `seed_containers` are selected -- Creates temporary directory for SSH keys on localhost -- Cannot be run independently or skipped manually - controlled entirely by dependent components ### 9. Asset Host Setup @@ -230,7 +185,7 @@ The playbook starts by verifying DNS records to ensure proper name resolution: - Imports [wire_values.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-demo/wire_values.yml) to prepare Helm chart values - Updates configurations for: - Wire services (domain names, IP addresses) - - SFT daemon (node affinity, domain settings) + - SFT configuration (node affinity, domain settings) - Coturn (IP addresses, node affinity) - Ingress controller (node affinity) - TLS/cert-manager settings @@ -244,7 +199,6 @@ The playbook starts by verifying DNS records to ensure proper name resolution: - Ed25519 cryptographic keys for zAuth - Random strings for security credentials - PostgreSQL credentials and Kubernetes secrets - - Prometheus authentication credentials - The playbook is idempotent: won't regenerate secrets if they already exist - If existing secret files are present, the playbook backs them up before replacing them - Can be skipped using `--skip-tags wire_secrets` @@ -258,31 +212,19 @@ The playbook starts by verifying DNS records to ensure proper name resolution: - Reports deployment status and pod health - Can be skipped using `--skip-tags helm_install` -**Cert Manager Hairpin Networking Configuration:** -- If `use_cert_manager` is true, automatically configures hairpin (NAT) behavior on the host so workloads (pods) can reach external/public IPs that resolve back to the same node -- Runs automatically at the end of helm chart installation when cert-manager is enabled +**Cert Manager Hairpin Networking Configuration (WIAB Dev):** +- When `use_cert_manager` is true, automatically configures hairpin (NAT) behavior on the host so workloads (pods) can reach external/public IPs that resolve back to the same node +- These rules ensure that pods in the Minikube cluster can reach `https://` even when DNS resolves to the same host's public IP address. +- This hairpin behaviour is required for HTTP-01 challenges used by Let's Encrypt in a single-node setup where ingress and workloads share the same node. ### 14. Temporary Cleanup - Locates all temporary SSH key directories created during deployment -- Lists and removes these directories - Stops `serve-assets` systemd service on `deploy_node` - Can be skipped using `--skip-tags cleanup` -## SSH Proxy Configuration - -The deployment uses an SSH proxy mechanism to access: -1. Kubernetes node within the Minikube cluster -2. The asset host for resource distribution - -SSH proxying is configured with: -- Dynamic discovery of SSH key paths (uses `ansible_ssh_private_key_file` if defined) -- StrictHostKeyChecking disabled for convenience -- UserKnownHostsFile set to /dev/null to prevent host key verification issues +## Ansible notes -## Notes - -- This deployment is only meant for testing, all the datastores are ephemeral - **Tag-Based Execution with Dependency Protection:** The playbook uses a hybrid approach where main components have tags for user control, while dependency tasks have no tags and are controlled automatically through `when` conditions. This prevents accidental skipping of critical dependencies while maintaining a clean user interface. - You can use Ansible tags to control the execution flow of the playbook. You can run specific tasks using `--tags` or skip specific tasks using `--skip-tags` as explained in the [Deployment Flow](#deployment-flow) section. By default, if no tags are specified, all tasks will run in sequence. @@ -290,12 +232,12 @@ SSH proxying is configured with: ```bash ansible-playbook -i ansible/inventory/demo/host.yml ansible/wiab-demo/deploy_wiab.yml --skip-tags verify_dns,install_pkgs,minikube ``` - Or if you just want to run the final deployment steps: + Or if you just want to run the final helm chart deployment steps: ```bash ansible-playbook -i ansible/inventory/demo/host.yml ansible/wiab-demo/deploy_wiab.yml --tags helm_install ``` -- All the iptables rules are not persisted after reboots, , but they can be regenerated by running just the minikube setup (and `cert_manager_networking` if required) or restored from the `/home/ansible_user/wire-iptables-rules/rules_post_wire.v4` file. +- All the iptables rules are not persisted after reboots, but they can be regenerated by running just the minikube setup (and `cert_manager_networking` if required) or restored from the `/home/ansible_user/wire-iptables-rules/rules_post_wire.v4` file. ```bash ansible-playbook -i ansible/inventory/demo/host.yml ansible/wiab-demo/deploy_wiab.yml --tags minikube,cert_manager_networking # or @@ -306,47 +248,11 @@ iptables-restore < /home/ansible_user/wire-iptables-rules/rules_post_wire.v4 - Temporary SSH keys are created and cleaned up automatically - The deployment creates a single-node Kubernetes cluster with all Wire services -## Offline bundle and alternative chart-only deployment - -The deployment playbook downloads an offline bundle that contains: - -- Helm chart tarballs (the charts used by the deployment) -- Docker/container image archives (used to seed Minikube/node container runtime) -- Helper scripts such as `bin/wiab-demo/offline_deploy_k8s.sh` which are sourced during the playbook - -If you already have a working **Kubernetes cluster** and prefer to use it instead of creating a local Minikube node, you can skip the Minikube and seeding tasks, and run only the Helm chart installation (tags `wire_values`, `wire_secrets` and `helm_install`). However, the offline bundle is still required to obtain the charts and the docker image archive(s) so you can: - -1. Extract charts from the bundle and point Helm to the extracted chart directories, and -2. Load container images into your cluster from the image archive. - -Typical steps to load images manually (examples — adapt for your runtime): - -```bash -# extract the image archive (example filename, check inside the bundle you downloaded) -tar -xf containers-helm.tar -C /tmp/wiab-images - -# For Docker (on the machine that will load images into the cluster): -for img in /tmp/wiab-images/*.tar*; do docker load -i "$img"; done - -# For containerd (ctr) on a node that uses containerd: -for img in /tmp/wiab-images/*.tar; do sudo ctr -n=k8s.io images import "$img"; done -``` -Note: Optionally the playbooks `10. Asset Host Setup` and `11. Container Seeding` can also perform these image-extraction and loading steps for you: `setup-offline-sources.yml` will unarchive and host the images via a simple HTTP asset host, and `seed-offline-containerd.yml` will pull/load those images into Minikube node. Those playbooks are tuned for Minikube but can be adapted to work with your own cluster by creating an appropriate inventory and adjusting paths. - -## kubeconfig path used by Helm in this deployment - -Helm commands in the playbook are executed inside a helper Docker container and expect the kubeconfig to be mounted at `{{ ansible_user_dir }}/.kube/config` on the deploy node (the playbook mounts this into the container as `/root/.kube/config`). If you are using your own Kubernetes cluster instead of Minikube, ensure that the kubeconfig for your cluster is available at that path on the deploy node before running the `helm_install` step. - -Small note on values and secrets -- The playbook generates Helm values and secrets files under `{{ ansible_user_dir }}/wire-server-deploy/values/` (for example `values/wire-server/values.yaml` and `values/wire-server/secrets.yaml`). These files can be edited manually before running the `helm_install` step if you need to change chart values or secrets. - -## Available Tags +### Main Component Tags The following tags are available for controlling playbook execution: -### Main Component Tags - | Tag | Description | Automatic Dependencies | Skippable | |-----|-------------|----------------------|-----------| | `verify_dns` | DNS record verification | None | Yes (`--skip-tags verify_dns`) | @@ -358,6 +264,7 @@ The following tags are available for controlling playbook execution: | `wire_values` | Setup Wire Helm values | None | Yes (`--skip-tags wire_values`) | | `wire_secrets` | Create Wire secrets | None | Yes (`--skip-tags wire_secrets`) | | `helm_install` | Helm chart installation + cert-manager hairpin networking | None | Yes (`--skip-tags helm_install`) | +| `cert_manager_networking` | Cert Manager hairpin networking | None | Yes (`use_cert_manager=true`) | | `cleanup` | Temporary file cleanup | None | Yes (`--skip-tags cleanup`) | @@ -372,9 +279,48 @@ The following tags are available for controlling playbook execution: - **Quick helm values and secrets update:** `ansible-playbook ... --tags wire_values,wire_secrets` - **Resume from artifact download:** `ansible-playbook ... --skip-tags verify_dns,install_pkgs,minikube` -## Trying Things Out +## Custom installation + +If you already have a working **Kubernetes cluster** and prefer to use it instead of creating a local Minikube node, you can skip most of tasks like Minikube creation, container seeding etc, and run only the Helm chart installation tasks (tags `wire_values`, `wire_secrets` and `helm_install`). However, the offline bundle is still required to obtain the charts and the docker image archive(s). + +### Offline bundle and alternative chart-only deployment + +The deployment playbook downloads an offline bundle that contains: + +- Helm chart tarballs (the charts used by the deployment) +- Docker/container image archives (used to seed Minikube/node container runtime) +- Helper scripts such as `bin/wiab-demo/offline_deploy_k8s.sh` which are sourced during the playbook. + +How to access the helm charts and container images: +1. Download the artifact using the ansible tag `download`. +2. Extract charts from the bundle and point Helm to the extracted chart directories +3. Load container images into your cluster from the image archive, so that pods can find the images locally + +Typical steps to load images manually (examples — adapt for your runtime): + +```bash +# extract the image archive (example filename, check inside the bundle you downloaded) +tar -xf containers-helm.tar -C /tmp/wiab-images -At this point, with a bit of luck, everything should be working. If not, refer to the ‘Troubleshooting’ section below. +# For Docker (on the machine that will load images into the cluster): +for img in /tmp/wiab-images/*.tar*; do docker load -i "$img"; done + +# For containerd (ctr) on a node that uses containerd: +for img in /tmp/wiab-images/*.tar; do sudo ctr -n=k8s.io images import "$img"; done +``` + +> Note: Optionally the steps `10. Asset Host Setup` and `11. Container Seeding` can also perform these image-extraction and loading steps for you: `setup-offline-sources.yml` will unarchive and host the images via a simple HTTP asset host, and `seed-offline-containerd.yml` will pull/load those images into Minikube node. Those playbooks are tuned for Minikube but can be adapted to work with your own cluster by creating an appropriate inventory and adjusting paths. + +### kubeconfig path used by Helm in this deployment + +If you are using your own Kubernetes cluster instead of Minikube, ensure that the kubeconfig for your cluster is available at `"/home/{{ ansible_user }}/.kube/config"` before running the `helm_install` step. + +### Edit the helm chart values and secrets manually +The playbook generates Helm values and secrets files under `{{ ansible_user_dir }}/wire-server-deploy/values/` (for example `values/wire-server/values.yaml` and `values/wire-server/secrets.yaml`). These files can be edited manually before running the `helm_install` step if you need to change chart values or secrets. + +## Trying Things Out post installation + +At this point, wire-server installation should be working. If not, refer to the [Troubleshooting](#troubleshooting) section below. Can you reach the nginz server? @@ -397,27 +343,27 @@ Can you access the webapp? Open https://webapp. in your browser (Fi #### Why is my ansible-playbook failing? -- Check the error message and review the [Requirements](#requirements) section to confirm that all requirements are met. -- See [Notes](#notes) to run only the failing tasks. -- If `ansible-playbook` fails at the last step of [Helm Chart Installation](#14-helm-chart-installation), proceed to [Are Wire services running fine?](#are-wire-services-running-fine). +- Check the error message and review the [Requirements](#deployment-requirements) section to confirm that all requirements are met. +- See [Ansible Notes](#ansible-notes) to re-run only the failing tasks. +- If `ansible-playbook` fails at the last step of [Helm Chart Installation](#13-helm-chart-installation), proceed to [Are Wire services running fine?](#are-wire-services-running-fine). #### What to do if ansible-playbook finished successfully but still unable to access Wire? SSH into the `deploy_node` with user `ansible_user` and continue with the following steps. -#### Which version am I on? +##### Which version am I on? -There are multiple components that together form a running Wire-server deployment. The definitions for these can be found in the file `/home/ansible_user/wire-server-deploy/versions/containers_helm_images.json` after [downloading](#8-wire-artifact-download) the archive. +There are multiple components that together form a running Wire-server deployment. The definitions for these can be found in the file `/home/ansible_user/wire-server-deploy/versions/containers_helm_images.json` after [downloading](#7-wire-artifact-download) the archive. -#### Is networking working fine? +##### Is networking working fine? -- Verify that the [Network Access Requirements](#requirements) are met for the deploy_node. Check the verbose (-vvvv) output from the `ansible-playbook` command for the [Network Verification](#3-network-verification). -- Ensure that [DNS Requirements](#dns-requirements) has been followed. Check the verbose (-vvvv) output from the `ansible-playbook` command for the [DNS verification step](#1-dns-verification). +- Verify that the [Network Access Requirements](#deployment-requirements) are met for the deploy_node. Check the verbose (-vvvv) output from the `ansible-playbook` command for the [Wire IP Access Verification](#1-wire-ip-access-verification-always-runs). +- Ensure that [DNS Requirements](#deployment-requirements) has been followed. Check the verbose (-vvvv) output from the `ansible-playbook` command for the [DNS verification step](#2-dns-verification). - Check if iptables rules from Wire installation are in place using the following command: ```bash sudo iptables -t nat -L -nv --line-numbers | grep "Wire Iptables Rules" ``` -- If they are not visible or if you are unable to access the Wire services, refer to [Notes](#notes) to reset the iptables rules. +- If they are not visible or if you are unable to access the Wire services, refer to [Ansible Notes](#ansible-notes) to reset the iptables rules. #### How to check the status of minikube k8s cluster or get access to kubectl? @@ -432,11 +378,11 @@ minikube status --profile=k8s-wire cat ~/.kube/config kubectl --kubeconfig='~/.kube/config' get pods -A ``` -- If you are unable to access the k8s cluster, try reinstalling minikube using [Ansible run selective tasks](#ansible-run-selective-tasks) with the flags for [skip_minikube=false](#6-minikube-cluster-configuration) and [skip_helm_install=false](#13-helm-chart-installation). +- If you are unable to access the k8s cluster, try reinstalling minikube using [Minikube Cluster Configuration](#5-minikube-cluster-configuration) with the required tags. #### Are Wire services running fine? -Start by checking the state of all the pods: +Start by checking the state of all the pods, assuming that kubeconfig is available in your environment: ```default kubectl get pods --all-namespaces @@ -454,17 +400,17 @@ and/or: kubectl --namespace describe ``` -- If Wire pods or datastore pods are failing due to Docker image issues, try running some of the steps again using [Ansible run selective tasks](#ansible-run-selective-tasks) and set the flags for [skip_asset_host=true](#10-asset-host-setup) and [skip_setup_offline_seed=true](#11-container-seeding). +- If Wire pods or datastore pods are failing due to Docker image issues, try running some of the steps again using [Container Seeding](#10-container-seeding) and then run [Helm Chart Installation](#13-helm-chart-installation). #### Confirm if datastore services are working? -Wire-in-a-Box relies on several backend datastore services to function properly. If you experience issues with service connectivity or user operations, you can use **wire-utility** to troubleshoot and validate the health of these services. +Wire-in-a-Box relies on several backend datastore services to function properly. If you experience issues with service connectivity or user operations, you can use **[wire-utility](../administrate/wire-utility-tool.md)** to troubleshoot and validate the health of these services. **Available datastore services to check:** PostgreSQL, Cassandra, Elasticsearch, RabbitMQ, MinIO and Redis. **Note** - Deployed services can differ based on the Wire backend version deployed. **Using wire-utility for diagnostics:** -If wire-utility was successfully deployed (see [Deploy wire-utility](#14-helm-chart-installation) task), you can leverage it to inspect and validate all datastore services. Wire-utility provides comprehensive tooling for: +If wire-utility was successfully deployed (see [Helm Chart Installation](#13-helm-chart-installation) task and ansible inventory `demo/host.yml`), you can leverage it to inspect and validate all datastore services. Wire-utility provides comprehensive tooling for: - Querying datastore status and connectivity - Running diagnostics to identify service-level issues - Troubleshooting authentication and access problems @@ -497,8 +443,6 @@ If datastore pods are consistently failing, consider redeploying them using the The cleanup playbook uses a **safe-by-default** approach with the special `never` tag - **nothing is destroyed unless you explicitly specify tags**. This prevents accidental destruction of your deployment. -⚠️ **Important:** All cleanup tasks are tagged with `never`, which means they will not run unless explicitly requested. Running the cleanup playbook without any tags will do nothing. - ### Basic Usage **No destruction by default:** diff --git a/src/how-to/install/wiab-staging.md b/src/how-to/install/wiab-staging.md new file mode 100644 index 0000000..722d2a2 --- /dev/null +++ b/src/how-to/install/wiab-staging.md @@ -0,0 +1,595 @@ +# WIAB Staging Deployment Guide + +## Introduction + +**Wire in a Box (WIAB) Staging** is an installation of Wire running on a single physical machine using KVM-based virtual machines. This setup replicates the multi-node production Wire architecture in a consolidated environment suitable for testing, evaluation, and learning about Wire's infrastructure—but **not for production use**. The main use of this package is to verify that automation inside and outside of the wire product functions in the fashion you expect, before you run said automation in production. This will not test your network environment, load based behaviors, or the interface between wire and it's callingservices when using a DMZ'd network configuration. + +> **Important:** This is a sandbox environment. Data from a staging installation **cannot be migrated to production**. WIAB Staging is designed for experimentation, validation, and understanding Wire's deployment model. + +For a high-level comparison of WIAB Dev, WIAB Staging, and Production, see the [planning overview](planning.md) (in particular, [Artifact bundle and offline deployment](planning.md#artifact-bundle-and-offline-deployment)). + +## Architecture overview + +![Wire in a Box Staging Architecture](img/architecture-wiab-stag.png) + +## Relation to production + +WIAB Staging is designed to be structurally similar to a production deployment: + +- Separate nodes for Kubernetes and stateful services. +- Use of offline artifacts, Ansible, and Helm for installation similar to production workflows. +- Able to test upgrade proceedures for production. + +Key differences from production: + +- All VMs run on a single physical host (single failure domain). +- Resource sizing is optimized for staging, not for peak production traffic. +- Calling services run with limited resources (replicas=1). +- [Calling services](../../understand/overview.md#calling) will share the same k8s cluster as Wire stateful services hence, all infrastructure will be DMZ (De-militarized zone). + +If you need a fully supported, highly-available, secure, multi-datacenter deployment, use the **Production** path instead (see `ansible-VMs.md` and `helm-prod.md`). + +## Requirements + +- One physical machine with hypervisor support (KVM): + - **Memory:** 55 GiB RAM + - **Compute:** 29 vCPUs (At least processor 6 cores) + - **Storage:** 850 GB disk space (thin-provisioned) + - 7 VMs with [Ubuntu 22](https://releases.ubuntu.com/jammy/) as per (#VM-Provisioning) +- **DNS Records**: + - A method to create DNS records for your domain name (e.g. wire.example.com) + - See [DNS Requirements for Wire Deployments](includes/helm_dns-ingress-troubleshooting.inc.md#how-to-set-up-dns-records) for the full list of required hostnames and examples. +- **SSL/TLS certificates**: + - A method to create SSL/TLS certificates for your domain name (to allow connecting via https://) + - To ease out the process of managing certs, we recommend using [Let's Encrypt](https://letsencrypt.org/getting-started/) & [cert-manager](https://cert-manager.io/docs/tutorials/acme/http-validation/) +- **Network**: + - SSH access to the physical host. + - HTTP/HTTPS and UDP traffic from clients to the cluster, without interference from host-level firewalls (UFW, nftables) on required ports. + - IP forwarding between the host’s network interfaces when following the reference topology. Check [Network Traffic Configuration](#network-traffic-configuration) for more details. +- **Wire-server-deploy artifact**: Access to a `wire-server-deploy` offline artifact bundle (contact Wire support for the latest stable artifact). This tarball contains all required Bash scripts, deb packages, Ansible playbooks, Helm charts and container images to perform the installation. See the [planning overview](planning.md#artifact-bundle-and-offline-deployment) for how artifacts are used across WIAB Dev, WIAB Staging and Production. + +> Note: Check [WIAB Dev](./planning.md#wiab-dev-single-vm-wire-in-a-box) solution if looking for a lighter weight installation of wire-server, for development uses. + +## VM Provisioning + +Our deployment will be into 7 VMs, shown in the below VM Archetecture and Resource Allocation table, You can choose to use your own hypervisor to manage the VMs or use our [Wiab staging ansible provisioning playbook](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-staging-provision.yml) against your physical node to setup the VMs. + +**VM Architecture and Resource Allocation:** + +| Hostname | Role | RAM | vCPUs | Disk | +|----------|------|-----|-------|------| +| assethost | Asset/Storage Server | 4 GiB | 2 | 100 GB | +| kubenode1 | Kubernetes Node 1 | 9 GiB | 5 | 150 GB | +| kubenode2 | Kubernetes Node 2 | 9 GiB | 5 | 150 GB | +| kubenode3 | Kubernetes Node 3 | 9 GiB | 5 | 150 GB | +| datanode1 | Data Node 1 | 8 GiB | 4 | 100 GB | +| datanode2 | Data Node 2 | 8 GiB | 4 | 100 GB | +| datanode3 | Data Node 3 | 8 GiB | 4 | 100 GB | +| **Total** | | **55 GiB** | **29** | **850 GB** | + +*Note: These specifications are optimized for testing and validation purposes, not for performance benchmarking.* + +**VM Service Distribution:** + +- **kubenodes (kubenode1, kubenode2, kubenode3):** Run the Kubernetes cluster and host Wire backend services +- **datanodes (datanode1, datanode2, datanode3):** Run distributed data services: + - Cassandra + - PostgreSQL + - Elasticsearch + - Minio + - RabbitMQ +- **assethost:** Hosts static assets like container images and debian binaries to be used by kubenodes and datanodes during installation. + +### Internet access for VMs: + +In most cases, Wire Server components do not require internet access, except in the following situations: +- **External email services** – If your users’ email providers are hosted on the public internet (for example, `user@gmail.com`). If outbound internet access is not allowed and no internal email service is available on your local network, email-based flows such as verification codes, invitations, and some login emails will not be delivered. In that case, you must retrieve the required codes from the logs instead. Read more at [I deployed demo-smtp and I want to skip email configuration and retrieve verification codes directly](troubleshooting.md#i-deployed-demo-smtp-and-i-want-to-skip-email-configuration-and-retrieve-verification-codes-directly). +- **Mobile push notifications (FCM/APNS)** – Required to enable notifications for Android and Apple mobile devices. Wire uses [AWS services](infrastructure-configuration.md#enable-push-notifications-using-the-public-appstore-playstore-mobile-wire-clients) to relay notifications to Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS). +- **Third-party content previews** – If you want clients to display previews for services such as Giphy, Google, Spotify, or SoundCloud. Wire provides a proxy service for third-party content so clients do not communicate directly with these services, preventing exposure of IP addresses, cookies, or other metadata. +- **Federation with other Wire servers** – Required if your deployment needs to federate with another Wire server hosted on the public internet. + +> **Note:** Internet access is also required by the cert-manager pods (via Let's Encrypt) to issue TLS certificates when manual certificates are not used. +> +> This internet access is temporarily enabled as described in [cert-manager behaviour in NAT or bridge environments](#cert-manager-behaviour-in-nat-or-bridge-environments) to allow certificate issuance. Once the certificates are successfully issued by cert-manager, the internet access is removed from the VMs. + + +## Getting the Ansible playbooks + +> **Note:** If you prefer to manage VMs yourself with another hypervisor, you can skip this step and instead create the VMs manually with the specs shown above. Along with VM creation, you need to perform a few more steps like downloading the [wire-server-deploy artifact](planning.md#artifact-bundle-and-offline-deployment) and [setup network configuration](#network-traffic-configuration) including the firewall rules. + +On an admin machine (your workstation or the physical host), obtain the `wire-server-deploy` repository: + +- **Download as ZIP:** + ```bash + # requirements: wget and unzip + wget https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip + unzip master.zip + cd wire-server-deploy-master + ``` +- **Or clone with Git:** + ```bash + # requirements: git + git clone https://github.com/wireapp/wire-server-deploy.git + cd wire-server-deploy + ``` + +You will use the `ansible` directory from this repository, along with the offline artifact bundle. + +### WIAB staging ansible playbook overview + +The ansible playbook will perform the following operations for you: + +**System Setup & Networking**: + - Updates all system packages and installs required tools (git, curl, docker, qemu, libvirt, yq, etc.) + - Configures SSH, firewall (nftables), and user permissions (sudo, kvm, docker groups) + +**wire-server-deploy Artifact & Ubuntu Cloud Image**: + - Downloads wire-server-deploy static artifact and Ubuntu cloud image using public internet + - Extracts artifacts and sets proper file permissions + - *Note: The wire-server-deploy artifact downloaded corresponds to the currently supported wire-server backend version* + +**Libvirt Network Setup and VM Creation**: + - Removes default libvirt network and creates custom "wirebox" network + - Launches VMs using the [offline-vm-setup.sh](https://github.com/wireapp/wire-server-deploy/blob/master/bin/offline-vm-setup.sh) bash script with KVM + - Creates an SSH key directory at `/home/ansible_user/wire-server-deploy/ssh` for VM access + +**Ansible Inventory Generation**: + - Generates inventory.yml with actual VM IPs replacing placeholders + - Configures network interface variables for all k8s-nodes and datanodes + +*Note: Skip the Ansible playbook step if you are managing VMs with your own hypervisor.* + +### Provisioning the VMs with the WIAB Staging playbook + +If you want Ansible to create and configure the 7 VMs for you on a single physical host, use the WIAB Staging playbook. +> Note: the SSH user for ansible `ansible_user` should have password-less `sudo` access. The physical host should be running Ubuntu 22.04. + +1. Prepare an inventory for the physical host, based on [ansible/inventory/demo/wiab-staging.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/demo/wiab-staging.yml). +2. Adjust values such as: + - Physical host address of adminhost eg. `example.com` and SSH user eg.`demo` + - Ssh key to access the node `ansible_ssh_private_key_file='~/.ssh/id_ed25519'` +3. Run the provisioning playbook: + ```bash + ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml + ``` + +*Note: Ansible core version 2.16.3 or compatible is required for this step* + +### Getting started with Ansible playbook + +**Step 1: Obtain the ansible directory** + +We need the whole ansible directory as ansible-playbook uses some templates for its operations. Choose one method to download the `wire-server-deploy/ansible` directory: + +**Option A: Download as ZIP** +```bash +# requirements: wget and unzip +wget https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip +unzip master.zip +cd wire-server-deploy-master +``` + +**Option B: Clone with Git** +```bash +# requirements: git +git clone https://github.com/wireapp/wire-server-deploy.git +cd wire-server-deploy +``` + +**Step 2: Configure your Ansible inventory for your physical machine** + +A sample inventory is available at [ansible/inventory/demo/wiab-staging.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/demo/wiab-staging.yml). +Replace example.com with your physical machine (`adminhost`) address where KVM is available and adjust other variables like `ansible_user` and `ansible_ssh_private_key_file`. The SSH user for ansible `ansible_user` should have password-less `sudo` access. The adminhost should be running Ubuntu 22.04. From here on, we would refer the physical machine as `adminhost`. + +The `private_deployment` variable determines whether the VMs created below will have internet access. When set to `true` (default value), no internet access is available to VMs. Check [Network Traffic Configuration](#network-traffic-configuration) to understand more about it. + +**Step 3: Run the VM and network provision** + +```bash +ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml +``` + +*Note: Ansible core version 2.16.3 or compatible is required for this step* + +## When VMs are ready + +Now you should have 7 VMs running on your `adminhost`. If you have used the ansible playbook, you should also have a directory `/home/ansible_user/wire-server-deploy` with all resources required for further deployment. If you didn't use the above playbook, download the `wire-server-deploy` artifact shared by Wire support and extract it with tar. + +```bash +wget https://s3-eu-west-1.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-.tgz +tar xvzf wire-server-deploy-static-.tgz +cd wire-server-deploy +``` + +Ensure the inventory file `ansible/inventory/offline/inventory.yml` in the directory `/home/ansible_user/wire-server-deploy` contains values corresponding to your VMs. If you have already used the [Ansible playbook above](#getting-started-with-ansible-playbook) to set up VMs, this file should have been prepared for you. + +The purpose of this secondary Ansible inventory is to interact only with the 7 VMs after they have been created. It is used by the offline deployment steps to install Kubernetes and the stateful services. Our kubernetes solution uses `Calico` as the default `Container Network Interface (CNI)` plugin for cluster networking and ensure the [kernel requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#kernel-dependencies) are met on the VMs before deploying Kubernetes. + +If the provisioning playbook did not generate it for you, create it from the template [ansible/inventory/offline/staging.yml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/inventory/offline/staging.yml): + +```bash +cp ansible/inventory/offline/staging.yml ansible/inventory/offline/inventory.yml +``` + +Then edit `ansible/inventory/offline/inventory.yml` and replace all placeholder values. + +**Critical values to review in the inventory:** + +- `all.vars.ansible_user`: the SSH user present on every VM, should be part of `sudoers` list. +- `all.vars.ansible_ssh_private_key_file`: uncomment and set this if you authenticate with a private key, for example `ssh/id_ed25519`. +- `assethost.hosts.assethost.ansible_host`: IP address of the asset host VM. +- `kube-node.hosts.kubenode1|2|3.ansible_host`: IP addresses of the three Kubernetes VMs. +- `datanodes.hosts.datanode1|2|3.ansible_host`: IP addresses of the three data VMs. +- `cassandra.vars.cassandra_network_interface`, `elasticsearch.vars.elasticsearch_network_interface`, `minio.vars.minio_network_interface`, `rmq-cluster.vars.rabbitmq_network_interface`, `postgresql.vars.postgresql_network_interface`: the network interface name used by those services inside each data VM, for example `enp1s0`. Do not assume this value; verify it on your machines. +- `rmq-cluster.vars.rabbitmq_cluster_master`: the RabbitMQ primary node. Keep this aligned with the hostname of one of the data nodes, typically `datanode1`. +- `postgresql.vars.repmgr_node_config`: The repmgr config to ensure HA PostgreSQL cluster. + +> **Note:** If your environment uses a non-standard MTU (e.g. cloud providers, VPNs, or overlay networks), you must [configure the MTU](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/calico.md#configuring-interface-mtu) for Calico in `k8s-cluster.vars`. Ensure all VMs have the same MTU on their primary interface: +> ```bash +> ip link show +> ``` +> Then set: +> ```yaml +> # k8s-cluster.vars. +> calico_mtu: +> calico_veth_mtu: +> ``` +> As a rule of thumb: +> - `calico_mtu = underlying network MTU - encapsulation overhead` +> - `calico_veth_mtu` ≤ `calico_mtu` + +**Hostnames matter:** + +- The inventory hostnames `assethost`, `kubenode1`, `kubenode2`, `kubenode3`, `datanode1`, `datanode2`, and `datanode3` should match the actual hostnames configured inside the VMs. +- This is especially important for RabbitMQ, because the nodes in `rmq-cluster` must match each VM's real hostname. +- `datanode1` is also referenced as the Cassandra seed and as the default RabbitMQ cluster master in the template, so change those only if your topology differs. + +**SSH authentication options:** + +- If the VMs are reachable with a private key, set `ansible_ssh_private_key_file` in the inventory and run Ansible normally. +- If you rely on an SSH agent, keep `ansible_ssh_private_key_file` commented out and ensure the agent on the `adminhost` can reach all VMs. +- If you do not use a private key entry in the inventory and password authentication is enabled on the VMs, add `--ask-pass` when running ansible-playbooks manually and `--ask-become-pass` for sudo access. +- Our installation scripts are non-interactive, define `ansible_password` and `ansible_become_pass` in the inventory instead of relying on interactive password prompts. + +Before running the offline deployment scripts, verify that the inventory resolves to the expected machines. The commands below assume you are running them from `/home/ansible_user/wire-server-deploy` on the `adminhost`. + +```bash +# confirm the inventory hostnames match the actual VM hostnames +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'hostname' + +# verify the default IPv4 interface and address reported by Ansible +d ansible all -i ansible/inventory/offline/inventory.yml -m setup -a 'filter=ansible_default_ipv4' + +# verify time and timezone consistency across the machines +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'date' + +# verify if the MTU is consistent across all the VMs +d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a "ip link show | grep mtu" +``` + +If any hostname, IP address, SSH setting, or interface name is wrong at this stage, correct `ansible/inventory/offline/inventory.yml` before continuing. The next deployment steps assume this inventory is accurate. + +For a deeper explanation of offline inventories and VM roles in production-like setups, see [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md). + +## Next steps + +Once the secondary inventory is ready, please continue with the following steps: + +> **Note**: All next steps assume that the wire-server-deploy artifact has been downloaded on the `adminhost` (your physical machine) and extracted at `/home/ansible_user/wire-server-deploy`. All commands from here on will be issued from this directory on the `adminhost`, ssh on the node before proceeding. + +### Environment Setup + +- **Making tooling available in your environment** + - Source the `bin/offline-env.sh` shell script by running following command to set up a `d` alias that runs commands inside a Docker container with all necessary tools for offline deployment. + ```bash + source bin/offline-env.sh + ``` + - You can always use this alias `d` later to interact with the ansible playbooks, k8s cluster and the helm charts. + - The docker container mounts everything here from the `wire-server-deploy` directory, hence this acts an entry point for all the future interactions with ansible, k8s and helm charts. + - Please ensure that this environment doesn't contain `quay.io/wire/wire-server-deploy` docker image from previous installations, if it does then such images need to be removed. + +- **Generating secrets** + - Run `bin/offline-secrets.sh` to generate fresh secrets for Minio and coturn services. It uses the docker container images shipped inside the `wire-server-deploy` directory. + ```bash + ./bin/offline-secrets.sh + ``` + - This creates following secret files: + - `ansible/inventory/group_vars/all/secrets.yaml` + - `values/wire-server/secrets.yaml` + - `values/coturn/prod-secrets.example.yaml` + +### Kubernetes and Data Services Deployment + +- **Deploying Kubernetes and stateful services** + ```bash + d ./bin/offline-cluster.sh + ``` + - Run the above command to deploy Kubernetes and stateful services (Cassandra, PostgreSQL, Elasticsearch, Minio, RabbitMQ). This script deploys all infrastructure needed for Wire backend operations. + +To confirm if the kubernetes cluster has been setup correctly. All pods should be in `Running` or `Completed` state. Any `CrashLoopBackOff`, `Error`, or `Pending` states indicate a problem.: +```bash +d kubectl -n kube-system get pods +``` + +Check [Deploying Kubernetes and stateful services](ansible-VMs.md#deploying-kubernetes-and-stateful-services) for more details into the exact structure of `offline-cluster.sh` and sub commands. + +### Helm Operations to install wire services and supporting helm charts + +**Helm chart deployment (automated):** The script `bin/helm-operations.sh` will deploy the charts for you. It prepares `values.yaml`/`secrets.yaml`, customizes them for your domain/IPs, then runs Helm installs/upgrades in the correct order. Prepare the values before running it. + +**User-provided inputs (set these before running):** +- `TARGET_SYSTEM`: your domain (e.g., `wire.example.com` or `example.dev`) using which you have created subdomains, check more at [How to set up DNS records](https://docs.wire.com/latest/how-to/install/demo-wiab.html#dns-requirements). +- `CERT_MASTER_EMAIL`: email used by cert-manager for ACME registration (by default=TRUE). +- `DEPLOY_CALLING_SERVICES`: set to `TRUE` or `FALSE` to control deployment of the calling services (`sftd` and `coturn`). Default is `TRUE`. +- `HOST_IP`: the IP address on which traffic for Wire calling services is expected to arrive. This should match your public DNS A record since we are expected to deploy Wire and calling services behind a single firewall. The calling traffic configuration described in [Network Traffic Configuration](#network-traffic-configuration) and [Configure the port redirection in Nftables](https://github.com/wireapp/wire-server-deploy/blob/master/offline/coturn.md#configure-the-port-redirection-in-nftables). It is not required if `DEPLOY_CALLING_SERVICES=FALSE` + +**Calling services behavior:** +- When `DEPLOY_CALLING_SERVICES=TRUE` and `HOST_IP` is not passed, the script tries to detect the publicly visible address for this setup by running `wget -qO- https://api.ipify.org`. +- When `DEPLOY_CALLING_SERVICES=FALSE`, the script skips deployment of `sftd` and `coturn`, and it does not evaluate any `HOST_IP`-dependent logic. + +**TLS / certificate behavior (cert-manager vs. Bring Your Own):** +- By default, `bin/helm-operations.sh` has `DEPLOY_CERT_MANAGER=TRUE`, which installs cert-manager and configures a Let’s Encrypt (HTTP-01) issuer for the ingress charts. +- If you **do not** want Let’s Encrypt / cert-manager for TLS certs for the ingress, disable this step by passing the environment variable `DEPLOY_CERT_MANAGER=FALSE` when running `bin/helm-operations.sh`. + - When choosing `DEPLOY_CERT_MANAGER=FALSE`, ensure your ingress is configured with your own TLS secret(s) as described at [Acquiring / Deploying SSL Certificates](https://github.com/wireapp/wire-server-deploy/blob/master/offline/docs_ubuntu_22.04.md#acquiring--deploying-ssl-certificates). The `nginx-ingress-services` should be deployed manually. + - When choosing `DEPLOY_CERT_MANAGER=TRUE`, ensure if further network configuration is required by following [cert-manager behaviour in NAT / bridge environments](#cert-manager-behaviour-in-nat--bridge-environments). + +**To run the automated helm chart deployment with your variables**: +```bash +# example command - verify the variables before running it +d sh -c 'TARGET_SYSTEM="example.dev" CERT_MASTER_EMAIL="certmaster@example.dev" DEPLOY_CERT_MANAGER=TRUE DEPLOY_CALLING_SERVICES=TRUE HOST_IP="a.a.a.a" ./bin/helm-operations.sh' +``` + +If you do not want to deploy the calling services, run: + +```bash +d sh -c 'TARGET_SYSTEM="example.dev" CERT_MASTER_EMAIL="certmaster@example.dev" DEPLOY_CERT_MANAGER=TRUE DEPLOY_CALLING_SERVICES=FALSE ./bin/helm-operations.sh' +``` + +**Charts deployed by the script:** +- External datastores and helpers: `cassandra-external`, `elasticsearch-external`, `postgresql-external`, `minio-external`, `rabbitmq-external`, `databases-ephemeral`, `reaper`, `fake-aws`, `smtp`. +- Wire services: `wire-server`, `webapp`, `account-pages`, `team-settings`. +- Ingress and certificates: `ingress-nginx-controller`, `cert-manager`, `nginx-ingress-services`. +- Calling services: `sftd`, `coturn` when `DEPLOY_CALLING_SERVICES=TRUE`. + +**Values and secrets generation:** +- Creates `values.yaml` and `secrets.yaml` from `prod-values.example.yaml` and `prod-secrets.example.yaml` for each chart under `values/`. +- Backs up any existing `values.yaml`/`secrets.yaml` before replacing them. + +*Note: The `bin/helm-operations.sh` script above deploys these charts; you do not need to run the Helm commands manually unless you want to customize or debug.* + +**Manually removing non-required helm charts**: +- If some helm charts are not required in your environment like `demo-smtp` for email relaying then use the following command to uninstall them: +```bash +#d helm uninstall CHART_NAME +d helm uninstall demo-smtp +``` + +## Network Traffic Configuration + +### Bring traffic from the adminhost to Wire services in the k8s cluster + +Our Wire services are ready to receive traffic but we must enable network access from the `adminhost` network interface to the k8s pods running in the virtual network. We can achieve it by setting up [nftables](https://documentation.ubuntu.com/security/security-features/network/firewall/nftables/) rules on the `adminhost`. When using any other type of firewall tools, please ensure following network configuration is achieved. + +**Required Network Configuration:** + +The `adminhost` must forward traffic from external clients to the Kubernetes cluster running Wire services. This involves: + +1. **HTTP/HTTPS Traffic (Ingress)** – Forward external web traffic to Kubernetes ingress with load balancing across nodes + - Port 80 (TCP, from any external source to adminhost WAN IP) → DNAT to any Kubernetes node on port 31772 → HTTP ingress + - Port 443 (TCP, from any external source to adminhost WAN IP) → DNAT to any Kubernetes node on port 31773 → HTTPS ingress + +2. **Calling Services Traffic (Coturn/SFT)** – Forward TURN control and media traffic to the dedicated calling node + - Port 3478 (TCP/UDP, from any external source to adminhost WAN IP) → DNAT to calling node → TURN control traffic + - Ports 32768–65535 (UDP, from any external source to adminhost WAN IP) → DNAT to calling node → WebRTC media relay + +3. **Normal Access Rules (Host-Level Access)** – Restrict direct access to adminhost + - Port 22 (TCP, from allowed sources to adminhost) → allow → SSH access + - Traffic from loopback and VM bridge interfaces → allow → internal communication + - Any traffic within VM network → allowed → ensures inter-node communication + - All other inbound traffic to adminhost → drop → default deny policy + +4. **Masquerading (If [Internet access for VMs](#internet-access-for-vms) is required)** – Enable outbound connectivity for VMs + - Any traffic from VM subnet leaving via WAN interface → SNAT/masquerade → ensures return traffic from internet. + - Controlled by the variable `private_deployment` + +5. **Conditional Rules (cert-manager / HTTP-01 in NAT setups)** – Temporary adjustments for certificate validation + - DNAT hairpin traffic (VM → public IP → VM) → may require SNAT/masquerade on VM bridge → ensures return path during HTTP-01 self-checks + - Asymmetric routing scenarios → may require relaxed reverse path filtering → prevents packet drops during validation + +```mermaid +flowchart TB + +%% External Clients +Client[External Client] +LetsEncrypt["(Optional)
Let's Encrypt"] +Internet["(If Required)
Internet Services
(AWS/FCM/APNS, Email Services etc)"] + +%% Admin Host +AdminHost["AdminHost
(Firewall)"] + +%% VM Network +subgraph VM_Network ["VM Network (virbr0)"] + K1[KubeNode1] + K2[KubeNode2] + K3["KubeNode3
(CALLING NODE)"] +end + +%% Ingress Traffic +Client -->|HTTPS → wire-records.example.com| AdminHost +AdminHost -->|"DNAT →31772/31773"| K1 +AdminHost -->|"DNAT →31772/31773"| K2 +AdminHost -->|"DNAT →31772/31773"| K3 + +%% Calling Traffic +Client -->|TCP/UDP Calling| AdminHost +AdminHost -->|DNAT → Calling Node| K3 + +%% Outbound Traffic (Masquerade) +K1 -.->|SNAT via AdminHost| Internet +K2 -.->|SNAT via AdminHost| Internet +K3 -.->|SNAT via AdminHost| Internet + +%% Cert-Manager Flow +K1 <-.->|HTTP-01 self-check| AdminHost +AdminHost-.->|Request TLS certificate| LetsEncrypt +``` + +**Implementation:** + +The nftables rules are detailed in [wiab_server_nftables.conf.j2](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/files/wiab_server_nftables.conf.j2). Please ensure no other firewall services like `ufw` or `iptables` are configured on the node before continuing. + +If you have already used the `wiab-staging-provision.yml` ansible playbook to create the VMs, then you can apply these rules using the same playbook (with the tag `nftables`) against your adminhost, by following: + +```bash +ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml --tags nftables +``` + +> **Note:** You can use this playbook to change the internet access to VMs by modifying the variable `private_deployment` and re-run the above playbook. + +Alternatively, if you have not used the `wiab-staging-provision.yml` ansible playbook to create the VMs but would like to configure nftables rules, you can invoke the ansible playbook [wiab-staging-nftables.yaml](https://github.com/wireapp/wire-server-deploy/blob/master/ansible/wiab-staging-nftables.yaml) against the physical node. The playbook is available in the directory `wire-server-deploy/ansible`. + +The inventory file `inventory.yml` should define the following variables: +```yaml +wiab-staging: + hosts: + deploy_node: + # this should be the adminhost + ansible_host: example.com + ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o TCPKeepAlive=yes' + ansible_user: 'demo' + ansible_ssh_private_key_file: "~/.ssh/id_ed25519" + vars: + # Kubernetes node IPs + kubenode1_ip: 192.168.122.11 + kubenode2_ip: 192.168.122.12 + kubenode3_ip: 192.168.122.13 + # Calling services node(kubenode3) + calling_node_ip: 192.168.122.13 + wire_comment: "wiab-stag" + # it will disable internet access to VMs created on the private network + private_deployment: true + # the playbook will try to find the default interface i.e. INF_WAN from ansible_default_ipv4.interface +``` + +To implement the nftables rules, execute the following command: +```bash +# assuming the inventory.yml storead at wire-server-deploy and run command from the same directory +ansible-playbook -i inventory.yml ansible/wiab-staging-nftables.yaml +``` + +### cert-manager behaviour in NAT or bridge environments + +When cert-manager performs HTTP-01 self-checks inside the cluster, traffic can hairpin: + +- Pod → Node → host public IP → DNAT → Node → Ingress + +> **Note**: Using Let's Encrypt with `cert-manager` requires internet access ([to at least `acme-v02.api.letsencrypt.org`](https://letsencrypt.org/docs/acme-protocol-updates/)) to issue TLS certificates. If you have chosen to keep the network private, that is `private_deployment=true` for the VMs when applying nftables rules, then you need to make a temporary exception for this traffic. The same outbound access will also be required later for certificate renewals (after 180 days). +> +> To temporarily provide outbound internet access from the VMs, add the following nftables masquerading rule on the `adminhost`. Replace `INF_WAN` with the WAN interface that should carry this traffic, or make an equivalent change in your firewall: +> +> ```bash +> # Host WAN interface name +> INF_WAN=enp41s0 +> sudo nft insert rule ip nat POSTROUTING position 0 \ +> oifname $INF_WAN \ +> counter masquerade \ +> comment "wire-masquerade-for-letsencrypt" +> ``` +> +> If you are using a different implementation than nftables then please ensure Internet access to VMs. + +In NAT/bridge setups (for example, using `virbr0` on the host): + +- If nftables DNAT rules exist in `PREROUTING` without a matching SNAT on `virbr0 → virbr0`, return packets may bypass the host and break conntrack, causing HTTP-01 timeouts and certificate verification failures. +- too strict of `rp_filter` settings can drop asymmetric return packets. + +Before changing anything, first verify whether certificate issuance is actually failing: + +1. Check whether certificates are successfully issued: + ```bash + d kubectl get certificates + ``` +2. Check if k8s pods can access to its own domain: + ```bash + # Replace below. To find the aws-sns pod id, run the command: + # d kubectl get pods -l 'app=fake-aws-sns' + d kubectl exec -ti fake-aws-sns- -- sh -c 'curl --connect-timeout 10 -v webapp.' + ``` +3. If certificates are not in `Ready=True` state, inspect cert-manager logs for HTTP-01 self-check or timeout errors: + ```bash + # To find the , run the following command: + # d kubectl get pods -n cert-manager-ns -l 'app=cert-manager' + d kubectl logs -n cert-manager-ns + ``` + +If you observe HTTP-01 challenge timeouts or self-check failures in a NAT/bridge environment, hairpin SNAT and relaxed reverse-path filtering handling may be required. One possible approach is by making following changes to the adminhost: + +> **Note:** All `nft` and `sysctl` commands should run on the adminhost. + +- Relax reverse-path filtering to loose mode to allow asymmetric flows: + ```bash + sudo sysctl -w net.ipv4.conf.all.rp_filter=2 + sudo sysctl -w net.ipv4.conf.virbr0.rp_filter=2 + ``` + These settings help conntrack reverse DNAT correctly and avoid drops during cert-manager’s HTTP-01 challenges in NAT/bridge (`virbr0`) environments. + +- Enable Hairpin SNAT (temporary for cert-manager HTTP-01): + ```bash + sudo nft insert rule ip nat POSTROUTING position 0 \ + iifname "virbr0" oifname "virbr0" \ + ip daddr 192.168.122.0/24 ct status dnat \ + counter masquerade \ + comment "wire-hairpin-dnat-virbr0" + ``` + This forces DNATed traffic that hairpins over the bridge to be masqueraded, ensuring return traffic flows back through the host and conntrack can correctly reverse the DNAT. + + Verify the rule was added: + ```bash + sudo nft list chain ip nat POSTROUTING + ``` + You should see a rule similar to: + ``` + iifname "virbr0" oifname "virbr0" ip daddr 192.168.122.0/24 ct status dnat counter masquerade # handle + ``` + +- Remove the rule after certificates are issued, confirm by running the following: + ```bash + d kubectl get certificates + ``` + + Once Let’s Encrypt validation completes and certificates are issued, remove the temporary hairpin SNAT rule. Use the following pipeline to locate the rule handle and delete it safely: + ```bash + sudo nft -a list chain ip nat POSTROUTING | \ + grep wire-hairpin-dnat-virbr0 | \ + sed -E 's/.*handle ([0-9]+).*/\1/' | \ + xargs -r -I {} sudo nft delete rule ip nat POSTROUTING handle {} + ``` + +> **Note**: If you added an nftables rule above to allow temporary internet access for the VMs, remove it after certificate issuance is complete. +> +> To remove the nftables masquerading rule for all outgoing traffic run the following command: +> +> ```bash +> # remove the masquerading rule +> sudo nft -a list chain ip nat POSTROUTING | \ +> grep wire-masquerade-for-letsencrypt | \ +> sed -E 's/.*handle ([0-9]+).*/\1/' | \ +> xargs -r -I {} sudo nft delete rule ip nat POSTROUTING handle {} +> ``` +> +> If you are using a different implementation than nftables then please ensure temporary Internet access to VMs has been removed. + +> **Note**: If email delivery is not working, or if Android/iOS push notifications are still not working after you have configured the required AWS credentials, ensure the required outbound access is allowed as explained at [Internet access for VMs](#internet-access-for-vms). + +## Next steps and troubleshooting + +- To understand individual Helm charts and their configuration options, see [Installing wire-server (production) components using Helm](helm-prod.md). +- For Kubernetes and database Ansible roles, see [Installing kubernetes and databases on VMs with ansible](ansible-VMs.md). + +If something goes wrong: + +- Check pod status: `d kubectl get pods -A`. +- Inspect Helm releases: `d helm list -A`. +- Review Ansible output for failed tasks. + +WIAB Staging is intended to be a safe place to experiment – you can always tear down the VMs and redeploy from scratch once you have corrected your configuration. diff --git a/src/understand/overview.md b/src/understand/overview.md index 010105b..4ef92df 100644 --- a/src/understand/overview.md +++ b/src/understand/overview.md @@ -204,7 +204,7 @@ Henry has a different thing happening. For Henry, the call is on the SFT server The following diagram shows a usual setup with multiple VMs (Virtual Machines): -![image](../how-to/install/img/architecture-server-ha.png) +![image](../how-to/install/img/architecture-prod-ha.png) All the server components on one physical machine can connect to all the databases (also those on a different physical machine). The databases each connect to each-other, e.g. cassandra on machine 1 will connect to the cassandra VMs on machines 2 and 3.