Merge pull request #9570 from cwilkers/rhv-provider-rebase

RHV Provider Role and Playbooks

playbooks/rhv/README.md

@@ -0,0 +1,85 @@
+# RHV Playbooks
+## Provisioning
+This subdirectory contains the Ansible playbooks used to deploy 
+an OpenShift Container Platform environment on Red Hat Virtualization
+
+### Where do I start?
+Choose a host from which Ansible plays will be executed. This host must have
+the ability to access the web interface of the RHV cluster engine and the
+network on which the OpenShift nodes will be installed. We will refer to
+this host as the *bastion*.
+
+#### oVirt Ansible Roles
+The oVirt project maintains Ansible roles for managing an oVirt or RHV cluster.
+These should be installed on the *bastion* host according to the instructions
+at the [oVirt Ansible Roles page](https://github.com/ovirt/ovirt-ansible/).
+
+#### DNS Server
+An external DNS server is required to provide name resolution to nodes and
+applications. See the
+[OpenShift Installation Documentation](https://docs.openshift.com/container-platform/latest/install_config/install/prerequisites.html#prereq-dns)
+for details.
+
+### Let's Provision!
+#### High-level overview
+After populating inventory and variables files with the proper values,
+(see [The OpenShift Advanced Installation Documentation](https://docs.openshift.com/container-platform/latest/install_config/install/advanced_install.html)
+) a series of Ansible playbooks from this subdirectory will provision a set of
+nodes on the RHV (or oVirt) cluster, prepare them for OpenShift installation,
+and deploy an OpenShift cluster on them.
+
+#### Step 1 Inventory
+The [`inventory.example`](inventory.example) file here is provided as an example of a three master, three inventory
+environment. It is up to the user to add additional OpenShift specific variables to this file to configure
+required elements such as the registry, storage, authentication, and networking.
+
+One required variable added for this environment is the `openshift_rhv_dns_zone`. As this is used to construct
+hostnames during VM creation, it is essential that this be set to the default dns zone for those nodes' hostnames.
+
+#### Step 2 RHV Provisioning Variables
+
+Fill out a provisioning variables file (example [`provisioning-vars.yaml.example`](provisioning-vars.yaml.example)
+with values from your RHV environment, making sure to fill in all commented values.
+
+*Red Hat Virtualization Certificate*
+
+A copy of the `/etc/pki/ovirt-engine/ca.pem` from the RHV engine will need to
+be downloaded to the *bastion* and its location set in the `engine_cafile` variable. Replace the
+example server in the following command to download the certificate:
+
+```
+$ curl --output ca.pem 'http://engine.example.com/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA'
+
+```
+
+#### Step 3 Provision Virtual Machines in RHV
+Once all the variables in the `provisioning_vars.yaml` file are set, use the
+[`ovirt-vm-infra.yml`](openshift-cluster/ovirt-vm-infra.yml) playbook to begin
+provisioning.
+
+```
+ansible-playbook -i inventory -e@provisioning_vars.yml ${PATH_TO_OPENSHIFT_ANSIBLE}/playbooks/rhv/openshift-cluster/ovirt-vm-infra.yml
+```
+
+#### Step 4 Update DNS
+
+At this stage, ensure DNS is set up properly for the following access:
+
+* Nodes are available to each other by their hostnames.
+* The nodes running router services (typically the infrastructure nodes) are reachable by the wildcard entry.
+* The load balancer node is reachable as the openshift-master host entry for console access.
+
+#### Step 5 Install Prerequisite Services
+```
+ansible-playbook -i inventory ${PATH_TO_OPENSHIFT_ANSIBLE}/playbooks/prerequisites.yml
+```
+
+#### Step 6 Deploy OpenShift
+```
+ansible-playbook -i inventory ${PATH_TO_OPENSHIFT_ANSIBLE}/playbooks/deploy_cluster.yml
+```
+
+### Ready To Work!
+
+## Uninstall / Deprovisioning
+In case of a failed installation due to a missing variable, it is occasionally necessary to start from a fresh set of virtual machines. Uninstalling the virtual machines and reprovisioning them may be perfomed by running the [`openshift-cluster/unregister-vms.yaml`](openshift-cluster/unregister-vms.yaml) playbook (to recover RHSM entitlements) followed by the [`openshift-cluster/ovirt-vm-uninstall.yaml`](openshift-cluster/ovirt-vm-uninstall.yaml) playbook.

playbooks/rhv/inventory.example

@@ -0,0 +1,61 @@
+[all:vars]
+openshift_rhv_dns_zone=  # example.com
+
+[OSEv3:children]
+nodes
+masters
+etcd
+lb
+
+[OSEv3:vars]
+# General variables
+ansible_user=root
+openshift_deployment_type=origin
+#openshift_deployment_type=openshift-enterprise
+openshift_enable_service_catalog=False
+
+# Hostnames
+load_balancer_hostname=lb0.{{openshift_rhv_dns_zone}}
+openshift_master_cluster_hostname="{{ load_balancer_hostname }}"
+openshift_master_cluster_public_hostname=openshift-master.{{ openshift_rhv_dns_zone }}
+openshift_master_default_subdomain=apps.{{ openshift_rhv_dns_zone }}
+openshift_public_hostname="{{openshift_master_cluster_public_hostname}}"
+
+# Docker setup for extra disks on nodes
+container_runtime_docker_storage_setup_device=/dev/vdb
+container_runtime_docker_storage_type=overlay2
+openshift_node_local_quota_per_fsgroup=512Mi
+
+[masters:vars]
+container_runtime_extra_storage="[{'device': '/dev/vdc', 'path': '/var/lib/origin/openshift.local.volumes', 'options': 'gquota', 'filesystem': 'xfs', 'format': 'True'}, {'device': '/dev/vdd', 'path': '/var/lib/etcd', 'hosts': 'masters', 'filesystem': 'xfs', 'format': 'True'}]"
+
+[nodes:vars]
+container_runtime_extra_storage="[{'device': '/dev/vdc', 'path': '/var/lib/origin/openshift.local.volumes', 'options': 'gquota', 'filesystem': 'xfs', 'format': 'True'}]"
+
+[masters]
+master0.example.com
+master1.example.com
+master2.example.com
+
+[etcd]
+master0.example.com
+master1.example.com
+master2.example.com
+
+[infras]
+infra0.example.com
+infra1.example.com
+infra2.example.com
+
+[lb]
+lb0.example.com
+
+[nodes]
+master0.example.com openshift_node_group_name=node-config-master
+master1.example.com openshift_node_group_name=node-config-master
+master2.example.com openshift_node_group_name=node-config-master
+infra0.example.com openshift_node_group_name=node-config-infra
+infra1.example.com openshift_node_group_name=node-config-infra
+infra2.example.com openshift_node_group_name=node-config-infra
+compute0.example.com openshift_node_group_name=node-config-compute
+# vim: set syntax=dosini

playbooks/rhv/openshift-cluster/ovirt-vm-infra.yml

@@ -0,0 +1,33 @@
+---
+- name: Deploy oVirt template and virtual machines
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  pre_tasks:
+    - name: Log in to oVirt
+      ovirt_auth:
+        url: "{{ engine_url }}"
+        username: "{{ engine_user }}"
+        password: "{{ engine_password }}"
+        ca_file: "{{ engine_cafile | default(omit) }}"
+        insecure: "{{ engine_insecure | default(true) }}"
+      tags:
+        - always
+    - name: Build virtual machine facts
+      import_role:
+        name: openshift_rhv
+        tasks_from: build_vm_list.yml
+
+  roles:
+    - oVirt.image-template
+    - oVirt.vm-infra
+
+  post_tasks:
+    - name: Logout from oVirt
+      ovirt_auth:
+        state: absent
+        ovirt_auth: "{{ ovirt_auth }}"
+      tags:
+        - always
+...

playbooks/rhv/openshift-cluster/ovirt-vm-uninstall.yml

@@ -0,0 +1,37 @@
+---
+- name: Destroy oVirt VMs
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  pre_tasks:
+    - name: Log in to oVirt
+      ovirt_auth:
+        url: "{{ engine_url }}"
+        username: "{{ engine_user }}"
+        password: "{{ engine_password }}"
+        ca_file: "{{ engine_cafile | default(omit) }}"
+        insecure: "{{ engine_insecure | default(true) }}"
+      tags:
+        - always
+    - name: Build virtual machine facts
+      import_role:
+        name: openshift_rhv
+        tasks_from: build_vm_list.yml
+
+  tasks:
+    - name: Erase vms
+      ovirt_vms:
+        auth: "{{ ovirt_auth }}"
+        state: absent
+        name: "{{ item.name }}"
+      with_items:
+        - "{{ ovirt_vm_infra_vms }}"
+
+  post_tasks:
+    - name: Logout from oVirt
+      ovirt_auth:
+        state: absent
+        ovirt_auth: "{{ ovirt_auth }}"
+      tags:
+        - always

playbooks/rhv/openshift-cluster/roles

@@ -0,0 +1 @@
+../../../roles

playbooks/rhv/openshift-cluster/unregister-vms.yml

@@ -0,0 +1,11 @@
+---
+- name: Unregister VMs
+  gather_facts: true
+  hosts:
+    - nodes
+    - lb
+  tasks:
+    - import_role:
+        name: rhel_unsubscribe
+      ignore_errors: yes
+...

playbooks/rhv/provisioning-vars.yaml.example

@@ -0,0 +1,82 @@
+---
+###########################
+# Engine Connection
+###########################
+compatibility_version: 4.2
+engine_url:                     # https://engine.example.com/ovirt-engine/api
+engine_user:                    # admin@internal
+engine_password:                # secret
+
+# CA file copied from engine:/etc/pki/ovirt-engine/ca.pem
+# path is relative to openshift-cluster directory
+engine_cafile:                  # ../ca.pem
+
+data_center_name:               # Default
+openshift_rhv_cluster:          # Default
+openshift_rhv_data_store:       # vmstore
+openshift_rhv_ssh_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+
+##########################
+# Template Creation
+# https://github.com/oVirt/ovirt-ansible-image-template
+##########################
+qcow_url:                       # https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2c
+image_path: "{{ lookup('env', 'HOME') }}/Downloads/{{ template_name }}.qcow2"
+template_name:                  # rhel75
+template_cluster: "{{ openshift_rhv_cluster }}"
+template_memory: 8GiB
+template_cpu: 1
+template_disk_storage: "{{ openshift_rhv_data_store }}"
+template_disk_size: 60GiB
+template_nics:
+  - name: nic1
+    profile_name:                # ovirtmgmt
+    interface: virtio
+
+##########################
+# Virtual Machines
+##########################
+wait_for_ip: true                # Required to gather IP addresses from amchines after
+vm_infra_wait_for_ip_retries: 10 # Higher than default to wait for all IPs on all VMs
+vm_infra_wait_for_ip_delay: 8
+
+# Virtual machine profile for master nodes
+# Differs from node profile by 16GiB RAM, extra disk for etcd
+master_vm:
+  cluster: "{{ openshift_rhv_cluster }}"
+  template: "{{ template_name }}"
+  memory: 16GiB
+  cores: 2
+  high_availability: true
+  disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+    - size: 25GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: etcd_disk
+      interface: virtio
+  state: running
+
+# Virtual Machine profile for rest of nodes
+node_vm:
+  cluster: "{{ openshift_rhv_cluster }}"
+  template: "{{ template_name }}"
+  memory: 8GiB
+  cores: 2
+  disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+  state: running
+...

roles/openshift_rhv/README.md

@@ -0,0 +1,84 @@
+OpenShift RHV
+=============
+
+OpenShift Provisioned on Red Hat Virtualization and oVirt
+
+Role Tasks
+----------
+
+* `build_vm_list.yml`: Creates a list of virtual machine definitions and
+  affinity groups based on a simple manifest (below)
+
+Role Variables
+--------------
+
+For documentation on virtual machine profile options, see the [oVirt Ansible VM-Infra Documentation](https://github.com/oVirt/ovirt-ansible-vm-infra)
+
+| Name                      | Default value |                                                                                         |
+|---------------------------|---------------|-----------------------------------------------------------------------------------------|
+| openshift_rhv_vm_profile  | See below.    | Dictionary of dictionaries providing common VM parameters for virtual machine creation. |
+| openshift_rhv_vm_manifest | See below.    | List of dictionaries specifying node base name, count, and which of the above profiles to apply. The default creates three master nodes, three infrastructure nodes, one application node, and a load balancer. |
+
+```
+openshift_rhv_vm_profile:
+  master:
+    cluster: "{{ openshift_rhv_cluster }}"
+    template: "{{ ovirt_template_name }}"
+    memory: 16GiB
+    cores: 2
+    high_availability: true
+    disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+    - size: 25GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: etcd_disk
+      interface: virtio
+    state: running
+  node:
+    cluster: "{{ openshift_rhv_cluster }}"
+    template: "{{ ovirt_template_name }}"
+    memory: 8GiB
+    cores: 2
+    high_availability: true
+    disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+    state: running
+```
+
+```
+openshift_rhv_vm_manifest:
+- name: 'master'
+  count: 3
+  profile: 'master'
+- name: 'infra'
+  count: 3
+  profile: 'node'
+- name: 'compute'
+  count: 1
+  profile: 'node'
+- name: 'lb'
+  count: 1
+  profile: 'node'
+```
+
+Example Playbook
+----------------
+
+License
+-------
+
+Apache License, Version 2.0

roles/openshift_rhv/defaults/main.yml

@@ -0,0 +1,60 @@
+---
+# Defaults for oVirt/RHV provider integration
+compatibility_version: 4.2
+data_center_name: Default
+wait_for_ip: true
+openshift_rhv_cluster: Default
+template_name: openshift_node
+
+# oVirt VM Profiles
+openshift_rhv_vm_profile:
+  master:
+    cluster: "{{ openshift_rhv_cluster }}"
+    template: "{{ template_name }}"
+    memory: 16GiB
+    cores: 2
+    high_availability: true
+    disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+    - size: 25GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: etcd_disk
+      interface: virtio
+    state: running
+  node:
+    cluster: "{{ openshift_rhv_cluster }}"
+    template: "{{ template_name }}"
+    memory: 8GiB
+    cores: 2
+    high_availability: true
+    disks:
+    - size: 15GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: docker_disk
+      interface: virtio
+    - size: 30GiB
+      storage_domain: "{{ openshift_rhv_data_store }}"
+      name: localvol_disk
+      interface: virtio
+    state: running
+
+openshift_rhv_vm_manifest:
+- name: 'master'
+  count: 3
+  profile: 'master'
+- name: 'infra'
+  count: 3
+  profile: 'node'
+- name: 'compute'
+  count: 1
+  profile: 'node'
+- name: 'lb'
+  count: 1
+  profile: 'node'

roles/openshift_rhv/tasks/build_vm_list.yml

@@ -0,0 +1,45 @@
+---
+# Creates a dictionary for use with oVirt.vm-infra role
+# https://github.com/oVirt/ovirt-ansible-vm-infra
+- fail:
+    msg: "The openshift_rhv_dns_zone variable is required."
+  when:
+  - openshift_rhv_dns_zone is not defined
+- name: Create virtual machine list fact
+  set_fact:
+    vms: >-
+      {{ vms|default([]) }} + [
+      {% for iter in range(item.count) -%}
+      {% if iter > 0 -%},{% endif -%}
+      {
+      'name': '{{ item.name }}{{ iter }}.{{ openshift_rhv_dns_zone }}',
+      'tag': 'openshift_{{ item.profile }}',
+      'cloud_init':
+      {
+      'host_name': '{{ item.name }}{{ iter }}.{{ openshift_rhv_dns_zone }}',
+      'authorized_ssh_keys': '{{ openshift_rhv_ssh_key }}'
+      },
+      'profile':  {{ openshift_rhv_vm_profile[ item.profile ] }} ,
+      }
+      {% endfor -%}
+      ]
+    affinity_groups: >-
+      {{ affinity_groups|default([]) }} + [
+      {% if item.count > 1 -%}
+      {
+      'name': '{{ item.name }}_ag',
+      'cluster': '{{ openshift_rhv_cluster }}',
+      'vm_enforcing': 'false',
+      'vm_rule': 'negative',
+      'vms': [
+      {% for iter in range(item.count) -%}
+      '{{ item.name }}{{ iter }}.{{ openshift_rhv_dns_zone }}',
+      {% endfor -%}
+      ]
+      }
+      {% endif -%}
+      ]
+  with_items: "{{ openshift_rhv_vm_manifest }}"
+  tags:
+  - openshift_rhv
+...

setup.py

@@ -74,7 +74,7 @@ def find_playbooks():
     all_playbooks = set()
     included_playbooks = set()
 
-    exclude_dirs = ('adhoc', 'tasks')
+    exclude_dirs = ('adhoc', 'tasks', 'rhv')
     for yaml_file in find_files(
             os.path.join(os.getcwd(), 'playbooks'),
             exclude_dirs, None, r'^[^\.].*\.ya?ml$'):