Configuration updates for latest builds and major refactor

Configuration updates for latest builds
- Switch to using create-node-config
- Switch sdn services to use etcd over SSL
- This re-uses the client certificate deployed on each node
- Additional node registration changes
- Do not assume that metadata service is available in openshift_facts module
- Call systemctl daemon-reload after installing openshift-master, openshift-sdn-master, openshift-node, openshift-sdn-node
- Fix bug overriding openshift_hostname and openshift_public_hostname in byo playbooks
- Start moving generated configs to /etc/openshift
- Some custom module cleanup
- Add known issue with ansible-1.9 to README_OSE.md
- Update to genericize the kubernetes_register_node module
  - Default to use kubectl for commands
  - Allow for overriding kubectl_cmd
  - In openshift_register_node role, override kubectl_cmd to openshift_kube
- Set default openshift_registry_url for enterprise when deployment_type is enterprise
- Fix openshift_register_node for client config change
- Ensure that master certs directory is created
- Add roles and filter_plugin symlinks to playbooks/common/openshift-master and node
- Allow non-root user with sudo nopasswd access
- Updates for README_OSE.md
- Update byo inventory for adding additional comments
- Updates for node cert/config sync to work with non-root user using sudo
- Move node config/certs to /etc/openshift/node
- Don't use path for mktemp. addresses: https://github.com/openshift/openshift-ansible/issues/154

Create common playbooks
- create common/openshift-master/config.yml
- create common/openshift-node/config.yml
- update playbooks to use new common playbooks
- update launch playbooks to call update playbooks
- fix openshift_registry and openshift_node_ip usage

Set default deployment type to origin
- openshift_repo updates for enabling origin deployments
  - also separate repo and gpgkey file structure
  - remove kubernetes repo since it isn't currently needed
- full deployment type support for bin/cluster
  - honor OS_DEPLOYMENT_TYPE env variable
  - add --deployment-type option, which will override OS_DEPLOYMENT_TYPE if set
  - if neither OS_DEPLOYMENT_TYPE or --deployment-type is set, defaults to
    origin installs

Additional changes:
- Add separate config action to bin/cluster that runs ansible config but does
  not update packages
- Some more duplication reduction in cluster playbooks.
- Rename task files in playbooks dirs to have tasks in their name for clarity.
- update aws/gce scripts to use a directory for inventory (otherwise when
  there are no hosts returned from dynamic inventory there is an error)

libvirt refactor and update

- add libvirt dynamic inventory
- updates to use dynamic inventory for libvirt

README_OSE.md

@@ -7,15 +7,17 @@
 * [Creating the default variables for the hosts and host groups](#creating-the-default-variables-for-the-hosts-and-host-groups)
 * [Running the ansible playbooks](#running-the-ansible-playbooks)
 * [Post-ansible steps](#post-ansible-steps)
+* [Overriding detected ip addresses and hostnames](#overriding-detected-ip-addresses-and-hostnames)
 
 ## Requirements
 * ansible
-  * Tested using ansible-1.8.2-1.fc20.noarch, but should work with version 1.8+
+  * Tested using ansible-1.8.4-1.fc20.noarch, but should work with version 1.8+
+  * There is currently a known issue with ansible-1.9.0, you can downgrade to 1.8.4 on Fedora by installing one of the bulids from Koji: http://koji.fedoraproject.org/koji/packageinfo?packageID=13842
   * Available in Fedora channels
   * Available for EL with EPEL and Optional channel
 * One or more RHEL 7.1 VMs
-* ssh key based auth for the root user needs to be pre-configured from the host
-  running ansible to the remote hosts
+* Either ssh key based auth for the root user or ssh key based auth for a user
+  with sudo access (no password)
 * A checkout of openshift-ansible from https://github.com/openshift/openshift-ansible/
   
   ```sh
@@ -48,9 +50,6 @@ subscription-manager repos \
 ```
 * Configuration of router is not automated yet
 * Configuration of docker-registry is not automated yet
-* End-to-end testing has not been completed yet using this module
-* root user is used for all ansible actions; eventually we will support using
-  a non-root user with sudo.
 
 ## Configuring the host inventory
 [Ansible docs](http://docs.ansible.com/intro_inventory.html)
@@ -64,6 +63,38 @@ option to ansible-playbook.
 ```ini
 # This is an example of a bring your own (byo) host inventory
 
+# Create an OSEv3 group that contains the maters and nodes groups
+[OSEv3:children]
+masters
+nodes
+
+# Set variables common for all OSEv3 hosts
+[OSEv3:vars]
+# SSH user, this user should allow ssh based auth without requiring a password
+ansible_ssh_user=root
+
+# If ansible_ssh_user is not root, ansible_sudo must be set to true
+#ansible_sudo=true
+
+# To deploy origin, change deployment_type to origin
+deployment_type=enterprise
+
+# Pre-release registry URL
+openshift_registry_url=docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-${component}:${version}
+
+# Pre-release additional repo
+openshift_additional_repos=[{'id': 'ose-devel', 'name': 'ose-devel',
+'baseurl':
+'http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/3.0/latest/RH7-RHOSE-3.0/$basearch/os',
+'enabled': 1, 'gpgcheck': 0}]
+
+# Origin copr repo
+#openshift_additional_repos=[{'id': 'openshift-origin-copr', 'name':
+'OpenShift Origin COPR', 'baseurl':
+'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/epel-7-$basearch/',
+'enabled': 1, 'gpgcheck': 1, gpgkey:
+'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/pubkey.gpg'}]
+
 # host group for masters
 [masters]
 ose3-master.example.com
@@ -76,51 +107,13 @@ ose3-node[1:2].example.com
 The hostnames above should resolve both from the hosts themselves and
 the host where ansible is running (if different).
 
-## Creating the default variables for the hosts and host groups
-[Ansible docs](http://docs.ansible.com/intro_inventory.html#id9)
-
-#### Group vars for all hosts
-/etc/ansible/group_vars/all:
-```yaml
----
-# Assume that we want to use the root as the ssh user for all hosts
-ansible_ssh_user: root
-
-# Default debug level for all OpenShift hosts
-openshift_debug_level: 4
-
-# Set the OpenShift deployment type for all hosts
-openshift_deployment_type: enterprise
-
-# Override the default registry for development
-openshift_registry_url: docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-${component}:${version}
-
-# To use the latest OpenShift Enterprise Errata puddle:
-#openshift_additional_repos:
-#- id: ose-devel
-#  name: ose-devel
-#  baseurl: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterpriseErrata/3.0/latest/RH7-RHOSE-3.0/$basearch/os
-#  enabled: 1
-#  gpgcheck: 0
-# To use the latest OpenShift Enterprise Whitelist puddle:
-openshift_additional_repos:
-- id: ose-devel
-  name: ose-devel
-  baseurl: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/3.0/latest/RH7-RHOSE-3.0/$basearch/os
-  enabled: 1
-  gpgcheck: 0
-
-```
-
 ## Running the ansible playbooks
 From the openshift-ansible checkout run:
 ```sh
 ansible-playbook playbooks/byo/config.yml
 ```
-**Note:** this assumes that the host inventory is /etc/ansible/hosts and the
-group_vars are defined in /etc/ansible/group_vars, if using a different
-inventory file (and a group_vars directory that is in the same directory as
-the directory as the inventory) use the -i option for ansible-playbook.
+**Note:** this assumes that the host inventory is /etc/ansible/hosts, if using a different
+inventory file use the -i option for ansible-playbook.
 
 ## Post-ansible steps
 #### Create the default router
@@ -140,3 +133,109 @@ openshift ex registry --create=true \
   --images='docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-${component}:${version}' \
   --mount-host=/var/lib/openshift/docker-registry
 ```
+
+## Overriding detected ip addresses and hostnames
+Some deployments will require that the user override the detected hostnames
+and ip addresses for the hosts. To see what the default values will be you can
+run the openshift_facts playbook:
+```sh
+ansible-playbook playbooks/byo/openshift_facts.yml
+```
+The output will be similar to:
+```
+ok: [10.3.9.45] => {
+    "result": {
+        "ansible_facts": {
+            "openshift": {
+                "common": {
+                    "hostname": "jdetiber-osev3-ansible-005dcfa6-27c6-463d-9b95-ef059579befd.os1.phx2.redhat.com",
+                    "ip": "172.16.4.79",
+                    "public_hostname": "jdetiber-osev3-ansible-005dcfa6-27c6-463d-9b95-ef059579befd.os1.phx2.redhat.com",
+                    "public_ip": "10.3.9.45",
+                    "use_openshift_sdn": true
+                },
+                "provider": {
+                  ... <snip> ...
+                }
+            }
+        },
+        "changed": false,
+        "invocation": {
+            "module_args": "",
+            "module_name": "openshift_facts"
+        }
+    }
+}
+ok: [10.3.9.42] => {
+    "result": {
+        "ansible_facts": {
+            "openshift": {
+                "common": {
+                    "hostname": "jdetiber-osev3-ansible-c6ae8cdc-ba0b-4a81-bb37-14549893f9d3.os1.phx2.redhat.com",
+                    "ip": "172.16.4.75",
+                    "public_hostname": "jdetiber-osev3-ansible-c6ae8cdc-ba0b-4a81-bb37-14549893f9d3.os1.phx2.redhat.com",
+                    "public_ip": "10.3.9.42",
+                    "use_openshift_sdn": true
+                },
+                "provider": {
+                  ...<snip>...
+                }
+            }
+        },
+        "changed": false,
+        "invocation": {
+            "module_args": "",
+            "module_name": "openshift_facts"
+        }
+    }
+}
+ok: [10.3.9.36] => {
+    "result": {
+        "ansible_facts": {
+            "openshift": {
+                "common": {
+                    "hostname": "jdetiber-osev3-ansible-bc39a3d3-cdd7-42fe-9c12-9fac9b0ec320.os1.phx2.redhat.com",
+                    "ip": "172.16.4.73",
+                    "public_hostname": "jdetiber-osev3-ansible-bc39a3d3-cdd7-42fe-9c12-9fac9b0ec320.os1.phx2.redhat.com",
+                    "public_ip": "10.3.9.36",
+                    "use_openshift_sdn": true
+                },
+                "provider": {
+                    ...<snip>...
+                }
+            }
+        },
+        "changed": false,
+        "invocation": {
+            "module_args": "",
+            "module_name": "openshift_facts"
+        }
+    }
+}
+```
+Now, we want to verify the detected common settings to verify that they are
+what we expect them to be (if not, we can override them).
+
+* hostname
+  * Should resolve to the internal ip from the instances themselves.
+  * openshift_hostname will override.
+* ip
+  * Should be the internal ip of the instance.
+  * openshift_ip will override.
+* public hostname
+  * Should resolve to the external ip from hosts outside of the cloud
+  * provider openshift_public_hostname will override.
+* public_ip
+  * Should be the externally accessible ip associated with the instance
+  * openshift_public_ip will override
+* use_openshift_sdn
+  * Should be true unless the cloud is GCE.
+  * openshift_use_openshift_sdn overrides
+
+To override the the defaults, you can set the variables in your inventory:
+```
+...snip...
+[masters]
+ose3-master.example.com openshift_ip=1.1.1.1 openshift_hostname=ose3-master.example.com openshift_public_ip=2.2.2.2 openshift_public_hostname=ose3-master.public.example.com
+...snip...
+```

README_libvirt.md

@@ -1,4 +1,3 @@
-
 LIBVIRT Setup instructions
 ==========================
 
@@ -9,19 +8,21 @@ This makes `libvirt` useful to develop, test and debug Openshift and openshift-a
 Install dependencies
 --------------------
 
-1. Install [dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html)
-2. Install [ebtables](http://ebtables.netfilter.org/)
-3. Install [qemu](http://wiki.qemu.org/Main_Page)
-4. Install [libvirt](http://libvirt.org/)
-5. Enable and start the libvirt daemon, e.g:
-   * ``systemctl enable libvirtd``
-   * ``systemctl start libvirtd``
-6. [Grant libvirt access to your user¹](https://libvirt.org/aclpolkit.html)
-7. Check that your `$HOME` is accessible to the qemu user²
+1.	Install [dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html)
+2.	Install [ebtables](http://ebtables.netfilter.org/)
+3.	Install [qemu](http://wiki.qemu.org/Main_Page)
+4.	Install [libvirt](http://libvirt.org/)
+5.	Enable and start the libvirt daemon, e.g:
+	-	`systemctl enable libvirtd`
+	-	`systemctl start libvirtd`
+6.	[Grant libvirt access to your user¹](https://libvirt.org/aclpolkit.html)
+7.	Check that your `$HOME` is accessible to the qemu user²
+8.	Configure dns resolution on the host³
 
 #### ¹ Depending on your distribution, libvirt access may be denied by default or may require a password at each access.
 
 You can test it with the following command:
+
 ```
 virsh -c qemu:///system pool-list
 ```
@@ -67,12 +68,7 @@ If your `$HOME` is world readable, everything is fine. If your `$HOME` is privat
 error: Cannot access storage file '$HOME/libvirt-storage-pool-openshift/lenaic-master-216d8.qcow2' (as uid:99, gid:78): Permission denied
 ```
 
-In order to fix that issue, you have several possibilities:
-* set `libvirt_storage_pool_path` inside `playbooks/libvirt/openshift-cluster/launch.yml` and `playbooks/libvirt/openshift-cluster/terminate.yml` to a directory:
-  * backed by a filesystem with a lot of free disk space
-  * writable by your user;
-  * accessible by the qemu user.
-* Grant the qemu user access to the storage pool.
+In order to fix that issue, you have several possibilities:* set `libvirt_storage_pool_path` inside `playbooks/libvirt/openshift-cluster/launch.yml` and `playbooks/libvirt/openshift-cluster/terminate.yml` to a directory: * backed by a filesystem with a lot of free disk space * writable by your user; * accessible by the qemu user.* Grant the qemu user access to the storage pool.
 
 On Arch:
 
@@ -80,13 +76,55 @@ On Arch:
 setfacl -m g:kvm:--x ~
 ```
 
-Test the setup
+#### ³ Enabling DNS resolution to your guest VMs with NetworkManager
+
+-	Verify NetworkManager is configured to use dnsmasq:
+
+```sh
+$ sudo vi /etc/NetworkManager/NetworkManager.conf
+[main]
+dns=dnsmasq
+```
+
+-	Configure dnsmasq to use the Virtual Network router for example.com:
+
+```sh
+sudo vi /etc/NetworkManager/dnsmasq.d/libvirt_dnsmasq.conf server=/example.com/192.168.55.1
+```
+
+Test The Setup
 --------------
 
+1.	cd openshift-ansible/
+2.	Try to list all instances (Passing an empty string as the cluster_id argument will result in all libvirt instances being listed)
+
+```
+  bin/cluster list libvirt ''
 ```
-cd openshift-ansible
 
-bin/cluster create -m 1 -n 3 libvirt lenaic
+Creating a cluster
+------------------
+
+1.	To create a cluster with one master and two nodes
 
-bin/cluster terminate libvirt lenaic
+```
+  bin/cluster create libvirt lenaic
+```
+
+Updating a cluster
+------------------
+
+1.	To update the cluster
+
+```
+  bin/cluster update libvirt lenaic
+```
+
+Terminating a cluster
+---------------------
+
+1.	To terminate the cluster
+
+```
+  bin/cluster terminate libvirt lenaic
 ```

bin/cluster

@@ -22,13 +22,28 @@ class Cluster(object):
                 '-o ControlPersist=600s '
             )
 
+    def get_deployment_type(self, args):
+        """
+        Get the deployment_type based on the environment variables and the
+        command line arguments
+        :param args: command line arguments provided by the user
+        :return: string representing the deployment type
+        """
+        deployment_type = 'origin'
+        if args.deployment_type:
+            deployment_type = args.deployment_type
+        elif 'OS_DEPLOYMENT_TYPE' in os.environ:
+            deployment_type = os.environ['OS_DEPLOYMENT_TYPE']
+        return deployment_type
+
     def create(self, args):
         """
         Create an OpenShift cluster for given provider
         :param args: command line arguments provided by user
         :return: exit status from run command
         """
-        env = {'cluster_id': args.cluster_id}
+        env = {'cluster_id': args.cluster_id,
+               'deployment_type': self.get_deployment_type(args)}
         playbook = "playbooks/{}/openshift-cluster/launch.yml".format(args.provider)
         inventory = self.setup_provider(args.provider)
 
@@ -43,7 +58,8 @@ class Cluster(object):
         :param args: command line arguments provided by user
         :return: exit status from run command
         """
-        env = {'cluster_id': args.cluster_id}
+        env = {'cluster_id': args.cluster_id,
+               'deployment_type': self.get_deployment_type(args)}
         playbook = "playbooks/{}/openshift-cluster/terminate.yml".format(args.provider)
         inventory = self.setup_provider(args.provider)
 
@@ -55,19 +71,34 @@ class Cluster(object):
         :param args: command line arguments provided by user
         :return: exit status from run command
         """
-        env = {'cluster_id': args.cluster_id}
+        env = {'cluster_id': args.cluster_id,
+               'deployment_type': self.get_deployment_type(args)}
         playbook = "playbooks/{}/openshift-cluster/list.yml".format(args.provider)
         inventory = self.setup_provider(args.provider)
 
         return self.action(args, inventory, env, playbook)
 
+    def config(self, args):
+        """
+        Configure or reconfigure OpenShift across clustered VMs
+        :param args: command line arguments provided by user
+        :return: exit status from run command
+        """
+        env = {'cluster_id': args.cluster_id,
+               'deployment_type': self.get_deployment_type(args)}
+        playbook = "playbooks/{}/openshift-cluster/config.yml".format(args.provider)
+        inventory = self.setup_provider(args.provider)
+
+        return self.action(args, inventory, env, playbook)
+
     def update(self, args):
         """
         Update to latest OpenShift across clustered VMs
         :param args: command line arguments provided by user
         :return: exit status from run command
         """
-        env = {'cluster_id': args.cluster_id}
+        env = {'cluster_id': args.cluster_id,
+               'deployment_type': self.get_deployment_type(args)}
         playbook = "playbooks/{}/openshift-cluster/update.yml".format(args.provider)
         inventory = self.setup_provider(args.provider)
 
@@ -81,19 +112,19 @@ class Cluster(object):
         """
         config = ConfigParser.ConfigParser()
         if 'gce' == provider:
-            config.readfp(open('inventory/gce/gce.ini'))
+            config.readfp(open('inventory/gce/hosts/gce.ini'))
 
             for key in config.options('gce'):
                 os.environ[key] = config.get('gce', key)
 
-            inventory = '-i inventory/gce/gce.py'
+            inventory = '-i inventory/gce/hosts'
         elif 'aws' == provider:
-            config.readfp(open('inventory/aws/ec2.ini'))
+            config.readfp(open('inventory/aws/hosts/ec2.ini'))
 
             for key in config.options('ec2'):
                 os.environ[key] = config.get('ec2', key)
 
-            inventory = '-i inventory/aws/ec2.py'
+            inventory = '-i inventory/aws/hosts'
         elif 'libvirt' == provider:
             inventory = '-i inventory/libvirt/hosts'
         else:
@@ -145,29 +176,49 @@ if __name__ == '__main__':
     parser = argparse.ArgumentParser(
         description='Python wrapper to ensure proper environment for OpenShift ansible playbooks',
     )
-    parser.add_argument('-v', '--verbose', action='count', help='Multiple -v options increase the verbosity')
+    parser.add_argument('-v', '--verbose', action='count',
+                        help='Multiple -v options increase the verbosity')
     parser.add_argument('--version', action='version', version='%(prog)s 0.2')
 
     meta_parser = argparse.ArgumentParser(add_help=False)
     meta_parser.add_argument('provider', choices=providers, help='provider')
     meta_parser.add_argument('cluster_id', help='prefix for cluster VM names')
-
-    action_parser = parser.add_subparsers(dest='action', title='actions', description='Choose from valid actions')
-
-    create_parser = action_parser.add_parser('create', help='Create a cluster', parents=[meta_parser])
-    create_parser.add_argument('-m', '--masters', default=1, type=int, help='number of masters to create in cluster')
-    create_parser.add_argument('-n', '--nodes', default=2, type=int, help='number of nodes to create in cluster')
+    meta_parser.add_argument('-t', '--deployment-type',
+                             choices=['origin', 'online', 'enterprise'],
+                             help='Deployment type. (default: origin)')
+
+    action_parser = parser.add_subparsers(dest='action', title='actions',
+                                          description='Choose from valid actions')
+
+    create_parser = action_parser.add_parser('create', help='Create a cluster',
+                                             parents=[meta_parser])
+    create_parser.add_argument('-m', '--masters', default=1, type=int,
+                               help='number of masters to create in cluster')
+    create_parser.add_argument('-n', '--nodes', default=2, type=int,
+                               help='number of nodes to create in cluster')
     create_parser.set_defaults(func=cluster.create)
 
-    terminate_parser = action_parser.add_parser('terminate', help='Destroy a cluster', parents=[meta_parser])
-    terminate_parser.add_argument('-f', '--force', action='store_true', help='Destroy cluster without confirmation')
+    config_parser = action_parser.add_parser('config',
+                                             help='Configure or reconfigure a cluster',
+                                             parents=[meta_parser])
+    config_parser.set_defaults(func=cluster.config)
+
+    terminate_parser = action_parser.add_parser('terminate',
+                                                help='Destroy a cluster',
+                                                parents=[meta_parser])
+    terminate_parser.add_argument('-f', '--force', action='store_true',
+                                  help='Destroy cluster without confirmation')
     terminate_parser.set_defaults(func=cluster.terminate)
 
-    update_parser = action_parser.add_parser('update', help='Update OpenShift across cluster', parents=[meta_parser])
-    update_parser.add_argument('-f', '--force', action='store_true', help='Update cluster without confirmation')
+    update_parser = action_parser.add_parser('update',
+                                             help='Update OpenShift across cluster',
+                                             parents=[meta_parser])
+    update_parser.add_argument('-f', '--force', action='store_true',
+                               help='Update cluster without confirmation')
     update_parser.set_defaults(func=cluster.update)
 
-    list_parser = action_parser.add_parser('list', help='List VMs in cluster', parents=[meta_parser])
+    list_parser = action_parser.add_parser('list', help='List VMs in cluster',
+                                           parents=[meta_parser])
     list_parser.set_defaults(func=cluster.list)
 
     args = parser.parse_args()

filter_plugins/oo_filters.py

@@ -5,6 +5,7 @@
 from ansible import errors, runner
 import json
 import pdb
+import re
 
 def oo_pdb(arg):
     ''' This pops you into a pdb instance where arg is the data passed in from the filter.
@@ -101,6 +102,18 @@ def oo_prepend_strings_in_list(data, prepend):
     retval = [prepend + s for s in data]
     return retval
 
+def oo_get_deployment_type_from_groups(data):
+    ''' This takes a list of groups and returns the associated
+        deployment-type
+    '''
+    if not issubclass(type(data), list):
+        raise errors.AnsibleFilterError("|failed expects first param is a list")
+    regexp = re.compile('^tag_deployment-type[-_]')
+    matches = filter(regexp.match, data)
+    if len(matches) > 0:
+        return regexp.sub('', matches[0])
+    return "Unknown"
+
 class FilterModule (object):
     def filters(self):
         return {
@@ -109,5 +122,6 @@ class FilterModule (object):
                 "oo_flatten": oo_flatten,
                 "oo_len": oo_len,
                 "oo_pdb": oo_pdb,
-                "oo_prepend_strings_in_list": oo_prepend_strings_in_list
+                "oo_prepend_strings_in_list": oo_prepend_strings_in_list,
+                "oo_get_deployment_type_from_groups": oo_get_deployment_type_from_groups
                 }

inventory/aws/group_vars/all

@@ -1,2 +0,0 @@
----
-ansible_ssh_user: root

inventory/aws/hosts/hosts

@@ -0,0 +1 @@
+localhost ansible_sudo=no ansible_python_interpreter=/usr/bin/python2

inventory/byo/group_vars/all

@@ -1,28 +0,0 @@
----
-# lets assume that we want to use the root as the ssh user for all hosts
-ansible_ssh_user: root
-
-# default debug level for all OpenShift hosts
-openshift_debug_level: 4
-
-# set the OpenShift deployment type for all hosts
-openshift_deployment_type: enterprise
-
-# Override the default registry for development
-openshift_registry_url: docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-${component}:${version}
-
-# Use latest Errata puddle as an additional repo:
-#openshift_additional_repos:
-#- id: ose-devel
-#  name: ose-devel
-#  baseurl: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterpriseErrata/3.0/latest/RH7-RHOSE-3.0/$basearch/os
-#  enabled: 1
-#  gpgcheck: 0
-
-# Use latest Whitelist puddle as an additional repo:
-openshift_additional_repos:
-- id: ose-devel
-  name: ose-devel
-  baseurl: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/3.0/latest/RH7-RHOSE-3.0/$basearch/os
-  enabled: 1
-  gpgcheck: 0

inventory/byo/hosts

@@ -1,5 +1,30 @@
 # This is an example of a bring your own (byo) host inventory
 
+# Create an OSEv3 group that contains the maters and nodes groups
+[OSEv3:children]
+masters
+nodes
+
+# Set variables common for all OSEv3 hosts
+[OSEv3:vars]
+# SSH user, this user should allow ssh based auth without requiring a password
+ansible_ssh_user=root
+
+# If ansible_ssh_user is not root, ansible_sudo must be set to true
+#ansible_sudo=true
+
+# To deploy origin, change deployment_type to origin
+deployment_type=enterprise
+
+# Pre-release registry URL
+openshift_registry_url=docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-${component}:${version}
+
+# Pre-release additional repo
+openshift_additional_repos=[{'id': 'ose-devel', 'name': 'ose-devel', 'baseurl': 'http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/3.0/latest/RH7-RHOSE-3.0/$basearch/os', 'enabled': 1, 'gpgcheck': 0}]
+
+# Origin copr repo
+#openshift_additional_repos=[{'id': 'openshift-origin-copr', 'name': 'OpenShift Origin COPR', 'baseurl': 'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/epel-7-$basearch/', 'enabled': 1, 'gpgcheck': 1, gpgkey: 'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/pubkey.gpg'}]
+
 # host group for masters
 [masters]
 ose3-master-ansible.test.example.com
@@ -7,4 +32,3 @@ ose3-master-ansible.test.example.com
 # host group for nodes
 [nodes]
 ose3-node[1:2]-ansible.test.example.com
-

inventory/gce/group_vars/all

@@ -1,2 +0,0 @@
----
-ansible_ssh_user: root

inventory/gce/hosts/hosts

@@ -0,0 +1 @@
+localhost ansible_sudo=no ansible_python_interpreter=/usr/bin/python2

inventory/libvirt/group_vars/all

@@ -1,2 +0,0 @@
----
-ansible_ssh_user: root

inventory/libvirt/hosts

@@ -1,2 +0,0 @@
-# Eventually we'll add the GCE, AWS, etc dynamic inventories, but for now...
-localhost ansible_python_interpreter=/usr/bin/python2

inventory/libvirt/hosts/hosts

@@ -0,0 +1 @@
+localhost ansible_sudo=no ansible_python_interpreter=/usr/bin/python2 connection=local

inventory/libvirt/hosts/libvirt.ini

@@ -0,0 +1,20 @@
+# Ansible libvirt external inventory script settings
+#
+
+[libvirt]
+
+uri = qemu:///system
+
+# API calls to libvirt can be slow. For this reason, we cache the results of an API
+# call. Set this to the path you want cache files to be written to. Two files
+# will be written to this directory:
+#   - ansible-libvirt.cache
+#   - ansible-libvirt.index
+cache_path = /tmp
+
+# The number of seconds a cache file is considered valid. After this many
+# seconds, a new API call will be made, and the cache file will be updated.
+cache_max_age = 900
+
+
+

inventory/libvirt/hosts/libvirt_generic.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python
+
+"""
+libvirt external inventory script
+=================================
+
+Ansible has a feature where instead of reading from /etc/ansible/hosts
+as a text file, it can query external programs to obtain the list
+of hosts, groups the hosts are in, and even variables to assign to each host.
+
+To use this, copy this file over /etc/ansible/hosts and chmod +x the file.
+This, more or less, allows you to keep one central database containing
+info about all of your managed instances.
+
+"""
+
+# (c) 2015, Jason DeTiberus <jdetiber@redhat.com>
+#
+# This file is part of Ansible,
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+######################################################################
+
+import argparse
+import ConfigParser
+import os
+import re
+import sys
+from time import time
+import libvirt
+import xml.etree.ElementTree as ET
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+
+
+class LibvirtInventory(object):
+
+    def __init__(self):
+        self.inventory = dict()  # A list of groups and the hosts in that group
+        self.cache = dict()  # Details about hosts in the inventory
+
+        # Read settings and parse CLI arguments
+        self.read_settings()
+        self.parse_cli_args()
+
+        if self.args.host:
+            print self.json_format_dict(self.get_host_info(), self.args.pretty)
+        elif self.args.list:
+            print self.json_format_dict(self.get_inventory(), self.args.pretty)
+        else:  # default action with no options
+            print self.json_format_dict(self.get_inventory(), self.args.pretty)
+
+    def read_settings(self):
+        config = ConfigParser.SafeConfigParser()
+        config.read(
+            os.path.dirname(os.path.realpath(__file__)) + '/libvirt.ini'
+        )
+        self.libvirt_uri = config.get('libvirt', 'uri')
+
+    def parse_cli_args(self):
+        parser = argparse.ArgumentParser(
+            description='Produce an Ansible Inventory file based on libvirt'
+        )
+        parser.add_argument(
+            '--list',
+            action='store_true',
+            default=True,
+            help='List instances (default: True)'
+        )
+        parser.add_argument(
+            '--host',
+            action='store',
+            help='Get all the variables about a specific instance'
+        )
+        parser.add_argument(
+            '--pretty',
+            action='store_true',
+            default=False,
+            help='Pretty format (default: False)'
+        )
+        self.args = parser.parse_args()
+
+    def get_host_info(self):
+        inventory = self.get_inventory()
+        if self.args.host in inventory['_meta']['hostvars']:
+            return inventory['_meta']['hostvars'][self.args.host]
+
+    def get_inventory(self):
+        inventory = dict(_meta=dict(hostvars=dict()))
+
+        conn = libvirt.openReadOnly(self.libvirt_uri)
+        if conn is None:
+            print "Failed to open connection to %s" % libvirt_uri
+            sys.exit(1)
+
+        domains = conn.listAllDomains()
+        if domains is None:
+            print "Failed to list domains for connection %s" % libvirt_uri
+            sys.exit(1)
+
+        arp_entries = self.parse_arp_entries()
+
+        for domain in domains:
+            hostvars = dict(libvirt_name=domain.name(),
+                            libvirt_id=domain.ID(),
+                            libvirt_uuid=domain.UUIDString())
+            domain_name = domain.name()
+
+            # TODO: add support for guests that are not in a running state
+            state, _ = domain.state()
+            # 2 is the state for a running guest
+            if state != 1:
+                continue
+
+            hostvars['libvirt_status'] = 'running'
+
+            root = ET.fromstring(domain.XMLDesc())
+            ns = {'ansible': 'https://github.com/ansible/ansible'}
+            for tag_elem in root.findall('./metadata/ansible:tag', ns):
+                tag = tag_elem.text
+                self.push(inventory, "tag_%s" % tag, domain_name)
+                self.push(hostvars, 'libvirt_tags', tag)
+
+            # TODO: support more than one network interface, also support
+            # interface types other than 'network'
+            interface = root.find("./devices/interface[@type='network']")
+            if interface is not None:
+                mac_elem = interface.find('mac')
+                if mac_elem is not None:
+                    mac = mac_elem.get('address')
+                    if mac in arp_entries:
+                        ip_address = arp_entries[mac]['ip_address']
+                        hostvars['ansible_ssh_host'] = ip_address
+                        hostvars['libvirt_ip_address'] = ip_address
+
+            inventory['_meta']['hostvars'][domain_name] = hostvars
+
+        return inventory
+
+    def parse_arp_entries(self):
+        arp_entries = dict()
+        with open('/proc/net/arp', 'r') as f:
+            # throw away the header
+            f.readline()
+
+            for line in f:
+                ip_address, _, _, mac, _, device = line.strip().split()
+                arp_entries[mac] = dict(ip_address=ip_address, device=device)
+
+        return arp_entries
+
+    def push(self, my_dict, key, element):
+        if key in my_dict:
+            my_dict[key].append(element)
+        else:
+            my_dict[key] = [element]
+
+    def json_format_dict(self, data, pretty=False):
+        if pretty:
+            return json.dumps(data, sort_keys=True, indent=2)
+        else:
+            return json.dumps(data)
+
+LibvirtInventory()

playbooks/aws/openshift-cluster/config.yml

@@ -0,0 +1,36 @@
+---
+- name: Populate oo_masters_to_config host group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_masters_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_config
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type_{{ cluster_id }}-openshift-master"] | default([])
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_config
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type_{{ cluster_id }}-openshift-node"] | default([])
+  - name: Evaluate oo_first_master
+    add_host:
+      name: "{{ groups['tag_env-host-type_' ~ cluster_id ~ '-openshift-master'][0] }}"
+      groups: oo_first_master
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    when: "'tag_env-host-type_{{ cluster_id }}-openshift-master' in groups"
+
+- include: ../../common/openshift-cluster/config.yml
+  vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
+    openshift_hostname: "{{ ec2_private_ip_address }}"
+    openshift_public_hostname: "{{ ec2_ip_address }}"

playbooks/aws/openshift-cluster/launch.yml

@@ -4,59 +4,26 @@
   connection: local
   gather_facts: no
   vars_files:
-      - vars.yml
+  - vars.yml
   tasks:
-    - set_fact: k8s_type="master"
-
-    - name: Generate master instance names(s)
-      set_fact: scratch={{ cluster_id }}-{{ k8s_type }}-{{ '%05x' |format( 1048576 |random) }}
-      register: master_names_output
-      with_sequence: start=1 end={{ num_masters }}
-
-    # These set_fact's cannot be combined
-    - set_fact:
-        master_names_string: "{% for item in master_names_output.results %}{{ item.ansible_facts.scratch }} {% endfor %}"
-
-    - set_fact:
-        master_names: "{{ master_names_string.strip().split(' ') }}"
-
-    - include: launch_instances.yml
-      vars:
-        instances: "{{ master_names }}"
-        cluster: "{{ cluster_id }}"
-        type: "{{ k8s_type }}"
-
-    - set_fact: k8s_type="node"
-
-    - name: Generate node instance names(s)
-      set_fact: scratch={{ cluster_id }}-{{ k8s_type }}-{{ '%05x' |format( 1048576 |random) }}
-      register: node_names_output
-      with_sequence: start=1 end={{ num_nodes }}
-
-    # These set_fact's cannot be combined
-    - set_fact:
-        node_names_string: "{% for item in node_names_output.results %}{{ item.ansible_facts.scratch }} {% endfor %}"
-
-    - set_fact:
-        node_names: "{{ node_names_string.strip().split(' ') }}"
-
-    - include: launch_instances.yml
-      vars:
-        instances: "{{ node_names }}"
-        cluster: "{{ cluster_id }}"
-        type: "{{ k8s_type }}"
-
-- hosts: "tag_env_{{ cluster_id }}"
-  roles:
-  - openshift_repos
-  - os_update_latest
-
-- include: ../openshift-master/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type_{{ cluster_id }}-openshift-master\"]"
-
-- include: ../openshift-node/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type_{{ cluster_id }}-openshift-node\"]"
+  - fail:
+      msg: Deployment type not supported for libvirt provider yet
+    when: deployment_type == 'enterprise'
+
+  - include: ../../common/openshift-cluster/set_master_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ master_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
+
+  - include: ../../common/openshift-cluster/set_node_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ node_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
+
+- include: update.yml
 
 - include: list.yml

playbooks/aws/openshift-cluster/list.yml

@@ -2,16 +2,23 @@
 - name: Generate oo_list_hosts group
   hosts: localhost
   gather_facts: no
+  vars_files:
+  - vars.yml
   tasks:
   - set_fact: scratch_group=tag_env_{{ cluster_id }}
     when: cluster_id != ''
   - set_fact: scratch_group=all
-    when: scratch_group is not defined
-  - add_host: name={{ item }} groups=oo_list_hosts
-    with_items: groups[scratch_group] | difference(['localhost'])
+    when: cluster_id == ''
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_list_hosts
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost'])
 
 - name: List Hosts
   hosts: oo_list_hosts
   gather_facts: no
   tasks:
-  - debug: msg="public:{{hostvars[inventory_hostname].ec2_ip_address}} private:{{hostvars[inventory_hostname].ec2_private_ip_address}}"
+  - debug:
+      msg: "public ip:{{ hostvars[inventory_hostname].ec2_ip_address }} private ip:{{ hostvars[inventory_hostname].ec2_private_ip_address }} deployment-type: {{ hostvars[inventory_hostname].group_names | oo_get_deployment_type_from_groups }}"

playbooks/aws/openshift-cluster/launch_instances.yml

@@ -1,8 +1,9 @@
 ---
+# TODO: modify machine_image based on deployment_type
 - set_fact:
-    machine_type: "{{ lookup('env', 'ec2_instance_type')|default('m3.large', true) }}"
-    machine_image: "{{ lookup('env', 'ec2_ami')|default('ami-307b3658', true) }}"
-    machine_region: "{{ lookup('env', 'ec2_region')|default('us-east-1', true) }}"
+    machine_type: "{{ lookup('env', 'ec2_instance_type') | default('m3.large', true) }}"
+    machine_image: "{{ lookup('env', 'ec2_ami') | default(deployment_vars[deployment_type].image, true) }}"
+    machine_region: "{{ lookup('env', 'ec2_region') | default(deployment_vars[deployment_type].region, true) }}"
     machine_keypair: "{{ lookup('env', 'ec2_keypair')|default('libra', true) }}"
     created_by: "{{ lookup('env', 'LOGNAME')|default(cluster, true) }}"
     security_group: "{{ lookup('env', 'ec2_security_group')|default('public', true) }}"
@@ -25,6 +26,7 @@
       env: "{{ env }}"
       host-type: "{{ host_type }}"
       env-host-type: "{{ env_host_type }}"
+      deployment-type: "{{ deployment_type }}"
   register: ec2
 
 - name: Add Name tag to instances
@@ -37,12 +39,14 @@
       Name: "{{ item.0 }}"
 
 - set_fact:
-    instance_groups: tag_created-by_{{ created_by }}, tag_env_{{ env }}, tag_host-type_{{ host_type }}, tag_env-host-type_{{ env_host_type }}
+    instance_groups: tag_created-by_{{ created_by }}, tag_env_{{ env }}, tag_host-type_{{ host_type }}, tag_env-host-type_{{ env_host_type }}, tag_deployment-type_{{ deployment_type }}
 
 - name: Add new instances groups and variables
   add_host:
     hostname: "{{ item.0 }}"
     ansible_ssh_host: "{{ item.1.dns_name }}"
+    ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+    ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
     groups: "{{ instance_groups }}"
     ec2_private_ip_address: "{{ item.1.private_ip }}"
     ec2_ip_address: "{{ item.1.public_ip }}"
@@ -54,10 +58,12 @@
   wait_for: "port=22 host={{ item.dns_name }}"
   with_items: ec2.instances
 
-- name: Wait for root user setup
-  command: "ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null root@{{ item.dns_name }} echo root user is setup"
+- name: Wait for user setup
+  command: "ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null {{ hostvars[item.0].ansible_ssh_user }}@{{ item.1.dns_name }} echo {{ hostvars[item.0].ansible_ssh_user }} user is setup"
   register: result
   until: result.rc == 0
   retries: 20
   delay: 10
-  with_items: ec2.instances
+  with_together:
+  - instances
+  - ec2.instances

playbooks/aws/openshift-cluster/terminate.yml

@@ -1,14 +1,26 @@
 ---
 - name: Terminate instance(s)
   hosts: localhost
-
+  gather_facts: no
   vars_files:
-    - vars.yml
+  - vars.yml
+  tasks:
+  - set_fact: scratch_group=tag_env-host-type_{{ cluster_id }}-openshift-node
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_terminate
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost'])
+
+  - set_fact: scratch_group=tag_env-host-type_{{ cluster_id }}-openshift-master
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_terminate
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost'])
 
 - include: ../openshift-node/terminate.yml
-  vars:
-    oo_host_group_exp: 'groups["tag_env-host-type_{{ cluster_id }}-openshift-node"]'
 
 - include: ../openshift-master/terminate.yml
-  vars:
-    oo_host_group_exp: 'groups["tag_env-host-type_{{ cluster_id }}-openshift-master"]'

playbooks/aws/openshift-cluster/update.yml

@@ -1,13 +1,18 @@
 ---
-- hosts: "tag_env_{{ cluster_id }}"
-  roles:
-  - openshift_repos
-  - os_update_latest
+- name: Populate oo_hosts_to_update group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_hosts_to_update
+    add_host:
+      name: "{{ item }}"
+      groups: oo_hosts_to_update
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type_{{ cluster_id }}-openshift-master"] | union(groups["tag_env-host-type_{{ cluster_id }}-openshift-node"]) | default([])
 
-- include: ../openshift-master/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type_{{ cluster_id }}-openshift-master\"]"
+- include: ../../common/openshift-cluster/update_repos_and_packages.yml
 
-- include: ../openshift-node/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type_{{ cluster_id }}-openshift-node\"]"
+- include: config.yml

playbooks/aws/openshift-cluster/vars.yml

@@ -1 +1,20 @@
 ---
+deployment_vars:
+  origin:
+    # fedora, since centos requires marketplace
+    image: ami-acd999c4
+    region: us-east-1
+    ssh_user: fedora
+    sudo: yes
+  online:
+    # private ami
+    image: ami-307b3658
+    region: us-east-1
+    ssh_user: root
+    sudo: no
+  enterprise:
+    # rhel-7.1, requires cloud access subscription
+    image: ami-10663b78
+    region: us-east-1
+    ssh_user: ec2-user
+    sudo: yes

playbooks/aws/openshift-master/config.yml

@@ -1,24 +1,19 @@
 ---
-- name: Populate oo_masters_to_config host group if needed
+- name: Populate oo_masters_to_config host group
   hosts: localhost
   gather_facts: no
   tasks:
-  - name: "Evaluate oo_host_group_exp if it's set"
-    add_host: "name={{ item }} groups=oo_masters_to_config"
-    with_items: "{{ oo_host_group_exp | default('') }}"
-    when: oo_host_group_exp is defined
+  - name: Evaluate oo_masters_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_config
+      ansible_ssh_user: root
+    with_items: oo_host_group_exp | default([])
 
-- name: Configure instances
-  hosts: oo_masters_to_config
+- include: ../../common/openshift-master/config.yml
   vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
     openshift_hostname: "{{ ec2_private_ip_address }}"
     openshift_public_hostname: "{{ ec2_ip_address }}"
-    # TODO: this should be removed once openshift-sdn packages are available
-    openshift_use_openshift_sdn: False
-  vars_files:
-  - vars.yml
-  roles:
-    - openshift_master
-    #- openshift_sdn_master
-    - pods
-    - os_env_extras

playbooks/aws/openshift-master/launch.yml

@@ -4,14 +4,12 @@
   connection: local
   gather_facts: no
 
+# TODO: modify atomic_ami based on deployment_type
   vars:
     inst_region: us-east-1
     atomic_ami: ami-86781fee
     user_data_file: user_data.txt
 
-  vars_files:
-    - vars.yml
-
   tasks:
     - name: Launch instances
       ec2:
@@ -40,7 +38,7 @@
           Name: "{{ item.0 }}"
 
     - name: Add other tags to instances
-      ec2_tag: "resource={{ item.id }} region={{ inst_region }} state=present"
+      ec2_tag: resource={{ item.id }} region={{ inst_region }} state=present
       with_items: ec2.instances
       args:
         tags: "{{ oo_new_inst_tags }}"
@@ -57,7 +55,7 @@
         - ec2.instances
 
     - name: Wait for ssh
-      wait_for: "port=22 host={{ item.dns_name }}"
+      wait_for: port=22 host={{ item.dns_name }}
       with_items: ec2.instances
 
     - name: Wait for root user setup

playbooks/aws/openshift-master/terminate.yml

@@ -1,15 +1,15 @@
 ---
-- name: Populate oo_masters_to_terminate host group if needed
+- name: Populate oo_masters_to_terminate host group
   hosts: localhost
   gather_facts: no
   tasks:
-    - name: Evaluate oo_host_group_exp if it's set
-      add_host: "name={{ item }} groups=oo_masters_to_terminate"
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
+    - name: Evaluate oo_masters_to_terminate
+      add_host: name={{ item }} groups=oo_masters_to_terminate
+      with_items: oo_host_group_exp | default([])
 
-- name: Gather facts for instances to terminate
+- name: Gather dynamic inventory variables for hosts to terminate
   hosts: oo_masters_to_terminate
+  gather_facts: no
 
 - name: Terminate instances
   hosts: localhost
@@ -27,11 +27,12 @@
       ignore_errors: yes
       register: ec2_term
       with_items: host_vars
+      when: "'oo_masters_to_terminate' in groups"
 
     # Fail if any of the instances failed to terminate with an error other
     # than 403 Forbidden
     - fail: msg=Terminating instance {{ item.item.ec2_id }} failed with message {{ item.msg }}
-      when: "item.failed and not item.msg | search(\"error: EC2ResponseError: 403 Forbidden\")"
+      when: "'oo_masters_to_terminate' in groups and item.failed and not item.msg | search(\"error: EC2ResponseError: 403 Forbidden\")"
       with_items: ec2_term.results
 
     - name: Stop instance if termination failed
@@ -42,6 +43,7 @@
       register: ec2_stop
       when: item.failed
       with_items: ec2_term.results
+      when: "'oo_masters_to_terminate' in groups"
 
     - name: Rename stopped instances
       ec2_tag: resource={{ item.item.item.ec2_id }} region={{ item.item.item.ec2_region }} state=present
@@ -49,4 +51,5 @@
         tags:
           Name: "{{ item.item.item.ec2_tag_Name }}-terminate"
       with_items: ec2_stop.results
+      when: "'oo_masters_to_terminate' in groups"
 

playbooks/aws/openshift-master/vars.yml

@@ -1,3 +0,0 @@
----
-openshift_debug_level: 4
-openshift_cluster_id: "{{ cluster_id }}"

playbooks/aws/openshift-node/config.yml

@@ -1,107 +1,25 @@
 ---
-- name: Populate oo_nodes_to_config host group if needed
+- name: Populate oo_nodes_to_config and oo_first_master host groups
   hosts: localhost
   gather_facts: no
   tasks:
-  - name: Evaluate oo_host_group_exp
-    add_host: "name={{ item }} groups=oo_nodes_to_config"
-    with_items: "{{ oo_host_group_exp | default('') }}"
-    when: oo_host_group_exp is defined
-  - add_host:
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_config
+      ansible_ssh_user: root
+    with_items: oo_host_group_exp | default([])
+  - name: Evaluate oo_first_master
+    add_host:
       name: "{{ groups['tag_env-host-type_' ~ cluster_id ~ '-openshift-master'][0] }}"
       groups: oo_first_master
-    when: oo_host_group_exp is defined
+      ansible_ssh_user: root
 
 
-- name: Gather and set facts for hosts to configure
-  hosts: oo_nodes_to_config
-  roles:
-  - openshift_facts
-  tasks:
-  # Since the master is registering the nodes before they are configured, we
-  # need to make sure to set the node properties beforehand if we do not want
-  # the defaults
-  - openshift_facts:
-      role: "{{ item.role }}"
-      local_facts: "{{ item.local_facts }}"
-    with_items:
-    - role: common
-      local_facts:
-        hostname: "{{ ec2_private_ip_address }}"
-        public_hostname: "{{ ec2_ip_address }}"
-        # TODO: this should be removed once openshift-sdn packages are available
-        use_openshift_sdn: False
-    - role: node
-      local_facts:
-        external_id: "{{ openshift_node_external_id | default(None) }}"
-        resources_cpu: "{{ openshfit_node_resources_cpu | default(None) }}"
-        resources_memory: "{{ openshfit_node_resources_memory | default(None) }}"
-        pod_cidr: "{{ openshfit_node_pod_cidr | default(None) }}"
-        labels: "{{ openshfit_node_labels | default(None) }}"
-        annotations: "{{ openshfit_node_annotations | default(None) }}"
-
-
-- name: Register nodes
-  hosts: oo_first_master
-  vars:
-    openshift_nodes: "{{ hostvars
-          | oo_select_keys(groups['oo_nodes_to_config']) }}"
-  roles:
-  - openshift_register_nodes
-  tasks:
-  - name: Create local temp directory for syncing certs
-    local_action: command /usr/bin/mktemp -d /tmp/openshift-ansible-XXXXXXX
-    register: mktemp
-
-  - name: Sync master certs to localhost
-    synchronize:
-      mode: pull
-      checksum: yes
-      src: /var/lib/openshift/openshift.local.certificates
-      dest: "{{ mktemp.stdout }}"
-
-
-- name: Configure instances
-  hosts: oo_nodes_to_config
-  vars_files:
-  - vars.yml
+- include: ../../common/openshift-node/config.yml
   vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
     openshift_hostname: "{{ ec2_private_ip_address }}"
     openshift_public_hostname: "{{ ec2_ip_address }}"
-    sync_tmpdir: "{{ hostvars[groups['oo_first_master'][0]].mktemp.stdout }}"
-    cert_parent_rel_path: openshift.local.certificates
-    cert_rel_path: "{{ cert_parent_rel_path }}/node-{{ openshift.common.hostname }}"
-    cert_base_path: /var/lib/openshift
-    cert_parent_path: "{{ cert_base_path }}/{{ cert_parent_rel_path }}"
-    cert_path: "{{ cert_base_path }}/{{ cert_rel_path }}"
-  pre_tasks:
-  - name: Ensure certificate directories exists
-    file:
-      path: "{{ item }}"
-      state: directory
-    with_items:
-    - "{{ cert_path }}"
-    - "{{ cert_parent_path }}/ca"
-
-  # TODO: notify restart openshift-node and/or restart openshift-sdn-node,
-  # possibly test service started time against certificate/config file
-  # timestamps in openshift-node or openshift-sdn-node to trigger notify
-  - name: Sync certs to nodes
-    synchronize:
-      checksum: yes
-      src: "{{ item.src }}"
-      dest: "{{ item.dest }}"
-      owner: no
-      group: no
-    with_items:
-    - src: "{{ sync_tmpdir }}/{{ cert_rel_path }}"
-      dest: "{{ cert_parent_path }}"
-    - src: "{{ sync_tmpdir }}/{{ cert_parent_rel_path }}/ca/cert.crt"
-      dest: "{{ cert_parent_path }}/ca/cert.crt"
-  - local_action: file name={{ sync_tmpdir }} state=absent
-    run_once: true
-  roles:
-    - openshift_node
-    #- openshift_sdn_node
-    - os_env_extras
-    - os_env_extras_node

playbooks/aws/openshift-node/launch.yml

@@ -4,14 +4,12 @@
   connection: local
   gather_facts: no
 
+# TODO: modify atomic_ami based on deployment_type
   vars:
     inst_region: us-east-1
     atomic_ami: ami-86781fee
     user_data_file: user_data.txt
 
-  vars_files:
-    - vars.yml
-
   tasks:
     - name: Launch instances
       ec2:
@@ -33,7 +31,7 @@
       with_items: ec2.instances
 
     - name: Add Name and environment tags to instances
-      ec2_tag: "resource={{ item.1.id }} region={{ inst_region }} state=present"
+      ec2_tag: resource={{ item.1.id }} region={{ inst_region }} state=present
       with_together:
         - oo_new_inst_names
         - ec2.instances
@@ -42,7 +40,7 @@
           Name: "{{ item.0 }}"
 
     - name: Add other tags to instances
-      ec2_tag: "resource={{ item.id }} region={{ inst_region }} state=present"
+      ec2_tag: resource={{ item.id }} region={{ inst_region }} state=present
       with_items: ec2.instances
       args:
         tags: "{{ oo_new_inst_tags }}"
@@ -59,7 +57,7 @@
         - ec2.instances
 
     - name: Wait for ssh
-      wait_for: "port=22 host={{ item.dns_name }}"
+      wait_for: port=22 host={{ item.dns_name }}
       with_items: ec2.instances
 
     - name: Wait for root user setup

playbooks/aws/openshift-node/terminate.yml

@@ -1,15 +1,15 @@
 ---
-- name: Populate oo_nodes_to_terminate host group if needed
+- name: Populate oo_nodes_to_terminate host group
   hosts: localhost
   gather_facts: no
   tasks:
-    - name: Evaluate oo_host_group_exp if it's set
-      add_host: "name={{ item }} groups=oo_nodes_to_terminate"
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
+    - name: Evaluate oo_nodes_to_terminate
+      add_host: name={{ item }} groups=oo_nodes_to_terminate
+      with_items: oo_host_group_exp | default([])
 
-- name: Gather facts for instances to terminate
+- name: Gather dynamic inventory variables for hosts to terminate
   hosts: oo_nodes_to_terminate
+  gather_facts: no
 
 - name: Terminate instances
   hosts: localhost
@@ -27,11 +27,12 @@
       ignore_errors: yes
       register: ec2_term
       with_items: host_vars
+      when: "'oo_nodes_to_terminate' in groups"
 
     # Fail if any of the instances failed to terminate with an error other
     # than 403 Forbidden
     - fail: msg=Terminating instance {{ item.item.ec2_id }} failed with message {{ item.msg }}
-      when: "item.failed and not item.msg | search(\"error: EC2ResponseError: 403 Forbidden\")"
+      when: "'oo_nodes_to_terminate' in groups and item.failed and not item.msg | search(\"error: EC2ResponseError: 403 Forbidden\")"
       with_items: ec2_term.results
 
     - name: Stop instance if termination failed
@@ -42,6 +43,7 @@
       register: ec2_stop
       when: item.failed
       with_items: ec2_term.results
+      when: "'oo_nodes_to_terminate' in groups"
 
     - name: Rename stopped instances
       ec2_tag: resource={{ item.item.item.ec2_id }} region={{ item.item.item.ec2_region }} state=present
@@ -49,4 +51,5 @@
         tags:
           Name: "{{ item.item.item.ec2_tag_Name }}-terminate"
       with_items: ec2_stop.results
+      when: "'oo_nodes_to_terminate' in groups"
 

playbooks/aws/openshift-node/vars.yml

@@ -1,3 +0,0 @@
----
-openshift_debug_level: 4
-openshift_cluster_id: "{{ cluster_id }}"

playbooks/byo/openshift-master/config.yml

@@ -1,9 +1,15 @@
 ---
-- name: Gather facts for node hosts
-  hosts: nodes
+- name: Populate oo_masters_to_config host group
+  hosts: localhost
+  gather_facts: no
+  tasks:
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_config
+    with_items: groups['masters']
 
-- name: Configure master instances
-  hosts: masters
-  roles:
-  - openshift_master
-  - openshift_sdn_master
+- include: ../../common/openshift-master/config.yml
+  vars:
+    openshift_cluster_id: "{{ cluster_id | default('default') }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"

playbooks/byo/openshift-node/config.yml

@@ -1,79 +1,21 @@
 ---
-- name: Gather facts for node hosts
-  hosts: nodes
-  roles:
-  - openshift_facts
+- name: Populate oo_nodes_to_config and oo_first_master host groups
+  hosts: localhost
+  gather_facts: no
   tasks:
-  # Since the master is registering the nodes before they are configured, we
-  # need to make sure to set the node properties beforehand if we do not want
-  # the defaults
-  - openshift_facts:
-      role: 'node'
-      local_facts:
-        hostname: "{{ openshift_hostname | default(None) }}"
-        external_id: "{{ openshift_node_external_id | default(None) }}"
-        resources_cpu: "{{ openshfit_node_resources_cpu | default(None) }}"
-        resources_memory: "{{ openshfit_node_resources_memory | default(None) }}"
-        pod_cidr: "{{ openshfit_node_pod_cidr | default(None) }}"
-        labels: "{{ openshfit_node_labels | default(None) }}"
-        annotations: "{{ openshfit_node_annotations | default(None) }}"
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_config
+    with_items: groups.nodes
+  - name: Evaluate oo_first_master
+    add_host:
+      name: "{{ groups.masters[0] }}"
+      groups: oo_first_master
 
 
-- name: Register nodes
-  hosts: masters[0]
+- include: ../../common/openshift-node/config.yml
   vars:
-    openshift_nodes: "{{ hostvars | oo_select_keys(groups['nodes']) }}"
-  roles:
-  - openshift_register_nodes
-  tasks:
-  - name: Create local temp directory for syncing certs
-    local_action: command /usr/bin/mktemp -d /tmp/openshift-ansible-XXXXXXX
-    register: mktemp
-
-  - name: Sync master certs to localhost
-    synchronize:
-      mode: pull
-      checksum: yes
-      src: /var/lib/openshift/openshift.local.certificates
-      dest: "{{ mktemp.stdout }}"
-
-
-- name: Configure node instances
-  hosts: nodes
-  vars:
-    sync_tmpdir: "{{ hostvars[groups['masters'][0]].mktemp.stdout }}"
-    cert_parent_rel_path: openshift.local.certificates
-    cert_rel_path: "{{ cert_parent_rel_path }}/node-{{ openshift.common.hostname }}"
-    cert_base_path: /var/lib/openshift
-    cert_parent_path: "{{ cert_base_path }}/{{ cert_parent_rel_path }}"
-    cert_path: "{{ cert_base_path }}/{{ cert_rel_path }}"
-    openshift_sdn_master_url: http://{{ hostvars[groups['masters'][0]].openshift.common.hostname }}:4001
-  pre_tasks:
-  - name: Ensure certificate directories exists
-    file:
-      path: "{{ item }}"
-      state: directory
-    with_items:
-    - "{{ cert_path }}"
-    - "{{ cert_parent_path }}/ca"
-
-  # TODO: notify restart openshift-node and/or restart openshift-sdn-node,
-  # possibly test service started time against certificate/config file
-  # timestamps in openshift-node or openshift-sdn-node to trigger notify
-  - name: Sync certs to nodes
-    synchronize:
-      checksum: yes
-      src: "{{ item.src }}"
-      dest: "{{ item.dest }}"
-      owner: no
-      group: no
-    with_items:
-    - src: "{{ sync_tmpdir }}/{{ cert_rel_path }}"
-      dest: "{{ cert_parent_path }}"
-    - src: "{{ sync_tmpdir }}/{{ cert_parent_rel_path }}/ca/cert.crt"
-      dest: "{{ cert_parent_path }}/ca/cert.crt"
-  - local_action: file name={{ sync_tmpdir }} state=absent
-    run_once: true
-  roles:
-  - openshift_node
-  - openshift_sdn_node
+    openshift_cluster_id: "{{ cluster_id | default('default') }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"

playbooks/byo/openshift_facts.yml

@@ -0,0 +1,10 @@
+---
+- name: Gather OpenShift facts
+  hosts: all
+  gather_facts: no
+  roles:
+  - openshift_facts
+  tasks:
+  - openshift_facts:
+    register: result
+  - debug: var=result

playbooks/common/openshift-cluster/config.yml

@@ -0,0 +1,4 @@
+---
+- include: ../openshift-master/config.yml
+
+- include: ../openshift-node/config.yml

playbooks/common/openshift-cluster/set_master_launch_facts_tasks.yml

@@ -0,0 +1,11 @@
+---
+- set_fact: k8s_type="master"
+
+- name: Generate master instance names(s)
+  set_fact:
+    scratch_name: "{{ cluster_id }}-{{ k8s_type }}-{{ '%05x' | format(1048576 | random) }}"
+  register: master_names_output
+  with_sequence: start=1 end={{ num_masters }}
+
+- set_fact:
+    master_names: "{{ master_names_output.results | oo_collect('ansible_facts') | oo_collect('scratch_name') }}"

playbooks/common/openshift-cluster/set_node_launch_facts_tasks.yml

@@ -0,0 +1,11 @@
+---
+- set_fact: k8s_type="node"
+
+- name: Generate node instance names(s)
+  set_fact:
+    scratch_name: "{{ cluster_id }}-{{ k8s_type }}-{{ '%05x' | format(1048576 | random) }}"
+  register: node_names_output
+  with_sequence: start=1 end={{ num_nodes }}
+
+- set_fact:
+    node_names: "{{ node_names_output.results | oo_collect('ansible_facts') | oo_collect('scratch_name') }}"

playbooks/common/openshift-cluster/update_repos_and_packages.yml

@@ -0,0 +1,7 @@
+---
+- hosts: oo_hosts_to_update
+  vars:
+    openshift_deployment_type: "{{ deployment_type }}"
+  roles:
+  - openshift_repos
+  - os_update_latest

playbooks/common/openshift-master/config.yml

@@ -0,0 +1,19 @@
+---
+- name: Configure master instances
+  hosts: oo_masters_to_config
+  vars:
+    openshift_sdn_master_url: https://{{ openshift.common.hostname }}:4001
+  roles:
+  - openshift_master
+  - { role: openshift_sdn_master, when: openshift.common.use_openshift_sdn | bool }
+  tasks:
+  - name: Create group for deployment type
+    group_by: key=oo_masters_deployment_type_{{ openshift.common.deployment_type }}
+    changed_when: False
+
+# Additional instance config for online deployments
+- name: Additional instance config
+  hosts: oo_masters_deployment_type_online
+  roles:
+  - pods
+  - os_env_extras

playbooks/common/openshift-master/roles

@@ -0,0 +1 @@
+../../../roles/

playbooks/common/openshift-node/config.yml

@@ -0,0 +1,121 @@
+---
+- name: Gather and set facts for node hosts
+  hosts: oo_nodes_to_config
+  roles:
+  - openshift_facts
+  tasks:
+  # Since the master is registering the nodes before they are configured, we
+  # need to make sure to set the node properties beforehand if we do not want
+  # the defaults
+  - openshift_facts:
+      role: "{{ item.role }}"
+      local_facts: "{{ item.local_facts }}"
+    with_items:
+      - role: common
+        local_facts:
+          hostname: "{{ openshift_hostname | default(None) }}"
+          public_hostname: "{{ openshift_public_hostname | default(None) }}"
+      - role: node
+        local_facts:
+          external_id: "{{ openshift_node_external_id | default(None) }}"
+          resources_cpu: "{{ openshift_node_resources_cpu | default(None) }}"
+          resources_memory: "{{ openshift_node_resources_memory | default(None) }}"
+          pod_cidr: "{{ openshift_node_pod_cidr | default(None) }}"
+          labels: "{{ openshift_node_labels | default(None) }}"
+          annotations: "{{ openshift_node_annotations | default(None) }}"
+          deployment_type: "{{ openshift_deployment_type }}"
+
+
+- name: Create temp directory for syncing certs
+  hosts: localhost
+  gather_facts: no
+  tasks:
+  - name: Create local temp directory for syncing certs
+    local_action: command mktemp -d /tmp/openshift-ansible-XXXXXXX
+    register: mktemp
+    changed_when: False
+
+
+- name: Register nodes
+  hosts: oo_first_master
+  vars:
+    openshift_nodes: "{{ hostvars | oo_select_keys(groups['oo_nodes_to_config']) }}"
+    sync_tmpdir: "{{ hostvars.localhost.mktemp.stdout }}"
+  roles:
+  - openshift_register_nodes
+  tasks:
+  - name: Create the temp directory on the master
+    file:
+      path: "{{ sync_tmpdir }}"
+      owner: "{{ ansible_ssh_user }}"
+      mode: 0700
+      state: directory
+    changed_when: False
+
+  - name: Create a tarball of the node config directories
+    command: tar -czvf {{ sync_tmpdir }}/{{ item.openshift.common.hostname }}.tgz ./
+    args:
+      chdir: "{{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}"
+    with_items: openshift_nodes
+    changed_when: False
+
+  - name: Retrieve the node config tarballs from the master
+    fetch:
+      src: "{{ sync_tmpdir }}/{{ item.openshift.common.hostname }}.tgz"
+      dest: "{{ sync_tmpdir }}/"
+      flat: yes
+      fail_on_missing: yes
+      validate_checksum: yes
+    with_items: openshift_nodes
+    changed_when: False
+
+  - name: Remove the temp directory on the master
+    file:
+      path: "{{ sync_tmpdir }}"
+      state: absent
+    changed_when: False
+
+
+- name: Configure node instances
+  hosts: oo_nodes_to_config
+  gather_facts: no
+  vars:
+    sync_tmpdir: "{{ hostvars.localhost.mktemp.stdout }}"
+    openshift_sdn_master_url: "https://{{ hostvars[groups['oo_first_master'][0]].openshift.common.hostname }}:4001"
+  pre_tasks:
+  - name: Ensure certificate directory exists
+    file:
+      path: "{{ openshift_node_cert_dir }}"
+      state: directory
+
+  # TODO: notify restart openshift-node and/or restart openshift-sdn-node,
+  # possibly test service started time against certificate/config file
+  # timestamps in openshift-node or openshift-sdn-node to trigger notify
+  - name: Unarchive the tarball on the node
+    unarchive:
+      src: "{{ sync_tmpdir }}/{{ openshift.common.hostname }}.tgz"
+      dest: "{{ openshift_node_cert_dir }}"
+  roles:
+  - openshift_node
+  - { role: openshift_sdn_node, when: openshift.common.use_openshift_sdn | bool }
+  tasks:
+  - name: Create group for deployment type
+    group_by: key=oo_nodes_deployment_type_{{ openshift.common.deployment_type }}
+    changed_when: False
+
+
+- name: Delete temporary directory
+  hosts: localhost
+  gather_facts: no
+  tasks:
+  - file: name={{ mktemp.stdout }} state=absent
+    changed_when: False
+
+
+# Additional config for online type deployments
+- name: Additional instance config
+  hosts: oo_nodes_deployment_type_online
+  gather_facts: no
+  roles:
+  - os_env_extras
+  - os_env_extras_node

playbooks/common/openshift-node/filter_plugins

@@ -0,0 +1 @@
+../../../filter_plugins

playbooks/common/openshift-node/roles

@@ -0,0 +1 @@
+../../../roles/

playbooks/gce/openshift-cluster/config.yml

@@ -0,0 +1,37 @@
+---
+# TODO: fix firewall related bug with GCE and origin, since GCE is overriding
+# /etc/sysconfig/iptables
+- name: Populate oo_masters_to_config host group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_masters_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_config
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-master"] | default([])
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_config
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-node"] | default([])
+  - name: Evaluate oo_first_master
+    add_host:
+      name: "{{ groups['tag_env-host-type-' ~ cluster_id ~ '-openshift-master'][0] }}"
+      groups: oo_first_master
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    when: "'tag_env-host-type-{{ cluster_id }}-openshift-master' in groups"
+
+- include: ../../common/openshift-cluster/config.yml
+  vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
+    openshift_hostname: "{{ gce_private_ip }}"

playbooks/gce/openshift-cluster/launch.yml

@@ -4,59 +4,25 @@
   connection: local
   gather_facts: no
   vars_files:
-      - vars.yml
+  - vars.yml
   tasks:
-    - set_fact: k8s_type="master"
-
-    - name: Generate master instance names(s)
-      set_fact: scratch={{ cluster_id }}-{{ k8s_type }}-{{ '%05x' |format( 1048576 |random) }}
-      register: master_names_output
-      with_sequence: start=1 end={{ num_masters }}
-
-    # These set_fact's cannot be combined
-    - set_fact:
-        master_names_string: "{% for item in master_names_output.results %}{{ item.ansible_facts.scratch }} {% endfor %}"
-
-    - set_fact:
-        master_names: "{{ master_names_string.strip().split(' ') }}"
-
-    - include: launch_instances.yml
-      vars:
-        instances: "{{ master_names }}"
-        cluster: "{{ cluster_id }}"
-        type: "{{ k8s_type }}"
-
-    - set_fact: k8s_type="node"
-
-    - name: Generate node instance names(s)
-      set_fact: scratch={{ cluster_id }}-{{ k8s_type }}-{{ '%05x' |format( 1048576 |random) }}
-      register: node_names_output
-      with_sequence: start=1 end={{ num_nodes }}
-
-    # These set_fact's cannot be combined
-    - set_fact:
-        node_names_string: "{% for item in node_names_output.results %}{{ item.ansible_facts.scratch }} {% endfor %}"
-
-    - set_fact:
-        node_names: "{{ node_names_string.strip().split(' ') }}"
-
-    - include: launch_instances.yml
-      vars:
-        instances: "{{ node_names }}"
-        cluster: "{{ cluster_id }}"
-        type: "{{ k8s_type }}"
-
-- hosts: "tag_env-{{ cluster_id }}"
-  roles:
-  - openshift_repos
-  - os_update_latest
-
-- include: ../openshift-master/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type-{{ cluster_id }}-openshift-master\"]"
-
-- include: ../openshift-node/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type-{{ cluster_id }}-openshift-node\"]"
+  - fail: msg="Deployment type not supported for libvirt provider yet"
+    when: deployment_type == 'enterprise'
+
+  - include: ../../common/openshift-cluster/set_master_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ master_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
+
+  - include: ../../common/openshift-cluster/set_node_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ node_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
+
+- include: update.yml
 
 - include: list.yml

playbooks/gce/openshift-cluster/list.yml

@@ -2,16 +2,23 @@
 - name: Generate oo_list_hosts group
   hosts: localhost
   gather_facts: no
+  vars_files:
+  - vars.yml
   tasks:
   - set_fact: scratch_group=tag_env-{{ cluster_id }}
     when: cluster_id != ''
   - set_fact: scratch_group=all
-    when: scratch_group is not defined
-  - add_host: name={{ item }} groups=oo_list_hosts
-    with_items: groups[scratch_group] | difference(['localhost']) | difference(groups.status_terminated)
+    when: cluster_id == ''
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_list_hosts
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost']) | difference(groups.status_terminated)
 
 - name: List Hosts
   hosts: oo_list_hosts
   gather_facts: no
   tasks:
-  - debug: msg="public:{{hostvars[inventory_hostname].gce_public_ip}} private:{{hostvars[inventory_hostname].gce_private_ip}}"
+  - debug:
+      msg: "public ip:{{ hostvars[inventory_hostname].gce_public_ip }} private ip:{{ hostvars[inventory_hostname].gce_private_ip }} deployment-type: {{ hostvars[inventory_hostname].group_names | oo_get_deployment_type_from_groups }}"

playbooks/gce/openshift-cluster/launch_instances.yml

@@ -2,41 +2,39 @@
 # TODO: when we are ready to go to ansible 1.9+ support only, we can update to
 # the gce task to use the disk_auto_delete parameter to avoid having to delete
 # the disk as a separate step on termination
-
-- set_fact:
-    machine_type: "{{ lookup('env', 'gce_machine_type') |default('n1-standard-1', true) }}"
-    machine_image: "{{ lookup('env', 'gce_machine_image') |default('libra-rhel7', true) }}"
-
 - name: Launch instance(s)
   gce:
     instance_names: "{{ instances }}"
-    machine_type: "{{ machine_type }}"
-    image: "{{ machine_image }}"
+    machine_type: "{{ lookup('env', 'gce_machine_type') | default('n1-standard-1', true) }}"
+    image: "{{ lookup('env', 'gce_machine_image') | default(deployment_vars[deployment_type].image, true) }}"
     service_account_email: "{{ lookup('env', 'gce_service_account_email_address') }}"
     pem_file: "{{ lookup('env', 'gce_service_account_pem_file_path') }}"
     project_id: "{{ lookup('env', 'gce_project_id') }}"
     tags:
-      - "created-by-{{ lookup('env', 'LOGNAME') |default(cluster, true) }}"
-      - "env-{{ cluster }}"
-      - "host-type-{{ type }}"
-      - "env-host-type-{{ cluster }}-openshift-{{ type }}"
+      - created-by-{{ lookup('env', 'LOGNAME') |default(cluster, true) }}
+      - env-{{ cluster }}
+      - host-type-{{ type }}
+      - env-host-type-{{ cluster }}-openshift-{{ type }}
+      - deployment-type-{{ deployment_type }}
   register: gce
 
 - name: Add new instances to groups and set variables needed
   add_host:
     hostname: "{{ item.name }}"
     ansible_ssh_host: "{{ item.public_ip }}"
+    ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+    ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
     groups: "{{ item.tags | oo_prepend_strings_in_list('tag_') | join(',') }}"
     gce_public_ip: "{{ item.public_ip }}"
     gce_private_ip: "{{ item.private_ip }}"
   with_items: gce.instance_data
 
 - name: Wait for ssh
-  wait_for: "port=22 host={{ item.public_ip }}"
+  wait_for: port=22 host={{ item.public_ip }}
   with_items: gce.instance_data
 
-- name: Wait for root user setup
-  command: "ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null root@{{ item.public_ip }} echo root user is setup"
+- name: Wait for user setup
+  command: "ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null {{ hostvars[item.name].ansible_ssh_user }}@{{ item.public_ip }} echo {{ hostvars[item.name].ansible_ssh_user }} user is setup"
   register: result
   until: result.rc == 0
   retries: 20

playbooks/gce/openshift-cluster/terminate.yml

@@ -1,20 +1,34 @@
 ---
 - name: Terminate instance(s)
   hosts: localhost
-
+  gather_facts: no
   vars_files:
-    - vars.yml
+  - vars.yml
+  tasks:
+  - set_fact: scratch_group=tag_env-host-type-{{ cluster_id }}-openshift-node
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_terminate
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost']) | difference(groups.status_terminated)
+
+  - set_fact: scratch_group=tag_env-host-type-{{ cluster_id }}-openshift-master
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_terminate
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost']) | difference(groups.status_terminated)
 
 - include: ../openshift-node/terminate.yml
   vars:
-    oo_host_group_exp: 'groups["tag_env-host-type-{{ cluster_id }}-openshift-node"]'
     gce_service_account_email: "{{ lookup('env', 'gce_service_account_email_address') }}"
     gce_pem_file: "{{ lookup('env', 'gce_service_account_pem_file_path') }}"
     gce_project_id: "{{ lookup('env', 'gce_project_id') }}"
 
 - include: ../openshift-master/terminate.yml
   vars:
-    oo_host_group_exp: 'groups["tag_env-host-type-{{ cluster_id }}-openshift-master"]'
     gce_service_account_email: "{{ lookup('env', 'gce_service_account_email_address') }}"
     gce_pem_file: "{{ lookup('env', 'gce_service_account_pem_file_path') }}"
     gce_project_id: "{{ lookup('env', 'gce_project_id') }}"

playbooks/gce/openshift-cluster/update.yml

@@ -1,13 +1,18 @@
 ---
-- hosts: "tag_env-{{ cluster_id }}"
-  roles:
-  - openshift_repos
-  - os_update_latest
+- name: Populate oo_hosts_to_update group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_hosts_to_update
+    add_host:
+      name: "{{ item }}"
+      groups: oo_hosts_to_update
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user | default(ansible_ssh_user, true) }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-master"] | union(groups["tag_env-host-type-{{ cluster_id }}-openshift-node"]) | default([])
 
-- include: ../openshift-master/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type-{{ cluster_id }}-openshift-master\"]"
+- include: ../../common/openshift-cluster/update_repos_and_packages.yml
 
-- include: ../openshift-node/config.yml
-  vars:
-    oo_host_group_exp: "groups[\"tag_env-host-type-{{ cluster_id }}-openshift-node\"]"
+- include: config.yml

playbooks/gce/openshift-cluster/vars.yml

@@ -1 +1,15 @@
 ---
+deployment_vars:
+  origin:
+    image: centos-7
+    ssh_user:
+    sudo: yes
+  online:
+    image: libra-rhel7
+    ssh_user: root
+    sudo: no
+  enterprise:
+    image: rhel-7
+    ssh_user:
+    sudo: yes
+

playbooks/gce/openshift-master/config.yml

@@ -1,20 +1,18 @@
 ---
-- name: master/config.yml, populate oo_masters_to_config host group if needed
+- name: Populate oo_masters_to_config host group
   hosts: localhost
   gather_facts: no
   tasks:
-  - name: "Evaluate oo_host_group_exp if it's set"
-    add_host: "name={{ item }} groups=oo_masters_to_config"
-    with_items: "{{ oo_host_group_exp | default('') }}"
-    when: oo_host_group_exp is defined
+  - name: Evaluate oo_masters_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_masters_to_config
+      ansible_ssh_user: root
+    with_items: oo_host_group_exp | default([])
 
-- name: "Configure instances"
-  hosts: oo_masters_to_config
+- include: ../../common/openshift-master/config.yml
   vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
     openshift_hostname: "{{ gce_private_ip }}"
-  vars_files:
-  - vars.yml
-  roles:
-    - openshift_master
-    - pods
-    - os_env_extras

playbooks/gce/openshift-master/launch.yml

@@ -8,14 +8,12 @@
   connection: local
   gather_facts: no
 
+# TODO: modify image based on deployment_type
   vars:
     inst_names: "{{ oo_new_inst_names }}"
     machine_type: n1-standard-1
     image: libra-rhel7
 
-  vars_files:
-      - vars.yml
-
   tasks:
     - name: Launch instances
       gce:
@@ -37,7 +35,7 @@
       with_items: gce.instance_data
 
     - name: Wait for ssh
-      wait_for: "port=22 host={{ item.public_ip }}"
+      wait_for: port=22 host={{ item.public_ip }}
       with_items: gce.instance_data
 
     - name: Wait for root user setup

playbooks/gce/openshift-master/terminate.yml

@@ -3,10 +3,9 @@
   hosts: localhost
   gather_facts: no
   tasks:
-    - name: Evaluate oo_host_group_exp if it's set
-      add_host: "name={{ item }} groups=oo_masters_to_terminate"
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
+    - name: Evaluate oo_masters_to_terminate
+      add_host: name={{ item }} groups=oo_masters_to_terminate
+      with_items: oo_host_group_exp | default([])
 
 - name: Terminate master instances
   hosts: localhost
@@ -22,6 +21,7 @@
         instance_names: "{{ groups['oo_masters_to_terminate'] }}"
         disks: "{{ groups['oo_masters_to_terminate'] }}"
       register: gce
+      when: "'oo_masters_to_terminate' in groups"
 
     - name: Remove disks of instances
       gce_pd:
@@ -32,5 +32,4 @@
         zone: "{{ gce.zone }}"
         state: absent
       with_items: gce.instance_names
-
-
+      when: "'oo_masters_to_terminate' in groups"

playbooks/gce/openshift-master/vars.yml

@@ -1,3 +0,0 @@
----
-openshift_debug_level: 4
-openshift_cluster_id: "{{ cluster_id }}"

playbooks/gce/openshift-node/config.yml

@@ -1,100 +1,24 @@
 ---
-- name: node/config.yml, populate oo_nodes_to_config host group if needed
+- name: Populate oo_nodes_to_config and oo_first_master host groups
   hosts: localhost
   gather_facts: no
   tasks:
-  - name: Evaluate oo_host_group_exp
-    add_host: "name={{ item }} groups=oo_nodes_to_config"
-    with_items: "{{ oo_host_group_exp | default('') }}"
-    when: oo_host_group_exp is defined
-  - add_host:
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      groups: oo_nodes_to_config
+      ansible_ssh_user: root
+    with_items: oo_host_group_exp | default([])
+  - name: Evaluate oo_first_master
+    add_host:
       name: "{{ groups['tag_env-host-type-' ~ cluster_id ~ '-openshift-master'][0] }}"
       groups: oo_first_master
-    when: oo_host_group_exp is defined
+      ansible_ssh_user: root
 
 
-- name: Gather and set facts for hosts to configure
-  hosts: oo_nodes_to_config
-  roles:
-  - openshift_facts
-  tasks:
-  # Since the master is registering the nodes before they are configured, we
-  # need to make sure to set the node properties beforehand if we do not want
-  # the defaults
-  - openshift_facts:
-      role: "{{ item.role }}"
-      local_facts: "{{ item.local_facts }}"
-    with_items:
-    - role: common
-      local_facts:
-        hostname: "{{ gce_private_ip }}"
-    - role: node
-      local_facts:
-        external_id: "{{ openshift_node_external_id | default(None) }}"
-        resources_cpu: "{{ openshfit_node_resources_cpu | default(None) }}"
-        resources_memory: "{{ openshfit_node_resources_memory | default(None) }}"
-        pod_cidr: "{{ openshfit_node_pod_cidr | default(None) }}"
-        labels: "{{ openshfit_node_labels | default(None) }}"
-        annotations: "{{ openshfit_node_annotations | default(None) }}"
-
-
-- name: Register nodes
-  hosts: oo_first_master
-  vars:
-    openshift_nodes: "{{ hostvars
-          | oo_select_keys(groups['oo_nodes_to_config']) }}"
-  roles:
-  - openshift_register_nodes
-  tasks:
-  - name: Create local temp directory for syncing certs
-    local_action: command /usr/bin/mktemp -d /tmp/openshift-ansible-XXXXXXX
-    register: mktemp
-
-  - name: Sync master certs to localhost
-    synchronize:
-      mode: pull
-      checksum: yes
-      src: /var/lib/openshift/openshift.local.certificates
-      dest: "{{ mktemp.stdout }}"
-
-- name: Configure instances
-  hosts: oo_nodes_to_config
-  vars_files:
-  - vars.yml
+- include: ../../common/openshift-node/config.yml
   vars:
-    sync_tmpdir: "{{ hostvars[groups['oo_first_master'][0]].mktemp.stdout }}"
-    cert_parent_rel_path: openshift.local.certificates
-    cert_rel_path: "{{ cert_parent_rel_path }}/node-{{ openshift.common.hostname }}"
-    cert_base_path: /var/lib/openshift
-    cert_parent_path: "{{ cert_base_path }}/{{ cert_parent_rel_path }}"
-    cert_path: "{{ cert_base_path }}/{{ cert_rel_path }}"
-  pre_tasks:
-  - name: Ensure certificate directories exists
-    file:
-      path: "{{ item }}"
-      state: directory
-    with_items:
-    - "{{ cert_path }}"
-    - "{{ cert_parent_path }}/ca"
-
-  # TODO: notify restart openshift-node and/or restart openshift-sdn-node,
-  # possibly test service started time against certificate/config file
-  # timestamps in openshift-node or openshift-sdn-node to trigger notify
-  - name: Sync certs to nodes
-    synchronize:
-      checksum: yes
-      src: "{{ item.src }}"
-      dest: "{{ item.dest }}"
-      owner: no
-      group: no
-    with_items:
-    - src: "{{ sync_tmpdir }}/{{ cert_rel_path }}"
-      dest: "{{ cert_parent_path }}"
-    - src: "{{ sync_tmpdir }}/{{ cert_parent_rel_path }}/ca/cert.crt"
-      dest: "{{ cert_parent_path }}/ca/cert.crt"
-  - local_action: file name={{ sync_tmpdir }} state=absent
-    run_once: true
-  roles:
-    - openshift_node
-    - os_env_extras
-    - os_env_extras_node
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"
+    openshift_hostname: "{{ gce_private_ip }}"

playbooks/gce/openshift-node/launch.yml

@@ -8,14 +8,12 @@
   connection: local
   gather_facts: no
 
+# TODO: modify image based on deployment_type
   vars:
     inst_names: "{{ oo_new_inst_names }}"
     machine_type: n1-standard-1
     image: libra-rhel7
 
-  vars_files:
-      - vars.yml
-
   tasks:
     - name: Launch instances
       gce:
@@ -37,7 +35,7 @@
       with_items: gce.instance_data
 
     - name: Wait for ssh
-      wait_for: "port=22 host={{ item.public_ip }}"
+      wait_for: port=22 host={{ item.public_ip }}
       with_items: gce.instance_data
 
     - name: Wait for root user setup

playbooks/gce/openshift-node/terminate.yml

@@ -3,10 +3,9 @@
   hosts: localhost
   gather_facts: no
   tasks:
-    - name: Evaluate oo_host_group_exp if it's set
-      add_host: "name={{ item }} groups=oo_nodes_to_terminate"
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
+    - name: Evaluate oo_nodes_to_terminate
+      add_host: name={{ item }} groups=oo_nodes_to_terminate
+      with_items: oo_host_group_exp | default([])
 
 - name: Terminate node instances
   hosts: localhost
@@ -22,6 +21,7 @@
         instance_names: "{{ groups['oo_nodes_to_terminate'] }}"
         disks: "{{ groups['oo_nodes_to_terminate'] }}"
       register: gce
+      when: "'oo_nodes_to_terminate' in groups"
 
     - name: Remove disks of instances
       gce_pd:
@@ -32,5 +32,4 @@
         zone: "{{ gce.zone }}"
         state: absent
       with_items: gce.instance_names
-
-
+      when: "'oo_nodes_to_terminate' in groups"

playbooks/gce/openshift-node/vars.yml

@@ -1,3 +0,0 @@
----
-openshift_debug_level: 4
-openshift_cluster_id: "{{ cluster_id }}"

playbooks/libvirt/openshift-cluster/config.yml

@@ -0,0 +1,38 @@
+---
+# TODO: need to figure out a plan for setting hostname, currently the default
+# is localhost, so no hostname value (or public_hostname) value is getting
+# assigned
+
+- name: Populate oo_masters_to_config host group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_masters_to_config
+    add_host:
+      name: "{{ item }}"
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+      groups: oo_masters_to_config
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-master"] | default([])
+  - name: Evaluate oo_nodes_to_config
+    add_host:
+      name: "{{ item }}"
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+      groups: oo_nodes_to_config
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-node"] | default([])
+  - name: Evaluate oo_first_master
+    add_host:
+      name: "{{ groups['tag_env-host-type-' ~ cluster_id ~ '-openshift-master'][0] }}"
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+      groups: oo_first_master
+    when: "'tag_env-host-type-{{ cluster_id }}-openshift-master' in groups"
+
+- include: ../../common/openshift-cluster/config.yml
+  vars:
+    openshift_cluster_id: "{{ cluster_id }}"
+    openshift_debug_level: 4
+    openshift_deployment_type: "{{ deployment_type }}"

playbooks/libvirt/openshift-cluster/launch.yml

@@ -1,65 +1,36 @@
+---
 - name: Launch instance(s)
   hosts: localhost
-  connection: local
   gather_facts: no
-
-  vars:
-    libvirt_storage_pool_path: "{{ lookup('env','HOME') }}/libvirt-storage-pool-openshift"
-    libvirt_storage_pool: 'openshift'
-    libvirt_uri: 'qemu:///system'
-
   vars_files:
-    - vars.yml
-
+  - vars.yml
+  vars:
+    os_libvirt_storage_pool: "{{ libvirt_storage_pool | default('images') }}"
+    os_libvirt_storage_pool_path: "{{ libvirt_storage_pool_path | default('/var/lib/libvirt/images') }}"
+    os_libvirt_network: "{{ libvirt_network | default('default') }}"
+    image_url: "{{ deployment_vars[deployment_type].image.url }}"
+    image_sha256: "{{ deployment_vars[deployment_type].image.sha256 }}"
+    image_name: "{{ deployment_vars[deployment_type].image.name }}"
   tasks:
-    - set_fact:
-        k8s_type: master
-
-    - name: Generate master instance name(s)
-      set_fact:
-        scratch_name: "{{ cluster_id }}-{{ k8s_type }}-{{ '%05x' | format( 1048576 | random ) }}"
-      register: master_names_output
-      with_sequence: start=1 end='{{ num_masters }}'
+  - fail: msg="Deployment type not supported for libvirt provider yet"
+    when: deployment_type in ['online', 'enterprise']
 
-    - set_fact:
-        master_names: "{{ master_names_output.results | oo_collect('ansible_facts') | oo_collect('scratch_name') }}"
+  - include: tasks/configure_libvirt.yml
 
-    - include: launch_instances.yml
-      vars:
-        instances: '{{ master_names }}'
-        cluster: '{{ cluster_id }}'
-        type: '{{ k8s_type }}'
-        group_name: 'tag_env-host-type-{{ cluster_id }}-openshift-master'
+  - include: ../../common/openshift-cluster/set_master_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ master_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
 
-    - set_fact:
-        k8s_type: node
+  - include: ../../common/openshift-cluster/set_node_launch_facts_tasks.yml
+  - include: tasks/launch_instances.yml
+    vars:
+      instances: "{{ node_names }}"
+      cluster: "{{ cluster_id }}"
+      type: "{{ k8s_type }}"
 
-    - name: Generate node instance name(s)
-      set_fact:
-        scratch_name: "{{ cluster_id }}-{{ k8s_type }}-{{ '%05x' | format( 1048576 | random ) }}"
-      register: node_names_output
-      with_sequence: start=1 end='{{ num_nodes }}'
+- include: update.yml
 
-    - set_fact:
-        node_names: "{{ node_names_output.results | oo_collect('ansible_facts') | oo_collect('scratch_name') }}"
-
-    - include: launch_instances.yml
-      vars:
-        instances: '{{ node_names }}'
-        cluster: '{{ cluster_id }}'
-        type: '{{ k8s_type }}'
-
-- hosts: 'tag_env-{{ cluster_id }}'
-  roles:
-    - openshift_repos
-    - os_update_latest
-
-- include: ../openshift-master/config.yml
-  vars:
-    oo_host_group_exp: 'groups["tag_env-host-type-{{ cluster_id }}-openshift-master"]'
-    oo_env: '{{ cluster_id }}'
-
-- include: ../openshift-node/config.yml
-  vars:
-    oo_host_group_exp: 'groups["tag_env-host-type-{{ cluster_id }}-openshift-node"]'
-    oo_env: '{{ cluster_id }}'
+- include: list.yml

playbooks/libvirt/openshift-cluster/launch_instances.yml

@@ -1,102 +0,0 @@
-- name: Create the libvirt storage directory for openshift
-  file:
-    dest: '{{ libvirt_storage_pool_path }}'
-    state: directory
-
-- name: Download Base Cloud image
-  get_url:
-    url: '{{ base_image_url }}'
-    sha256sum: '{{ base_image_sha256 }}'
-    dest: '{{ libvirt_storage_pool_path }}/{{ base_image_name }}'
-
-- name: Create the cloud-init config drive path
-  file:
-    dest: '{{ libvirt_storage_pool_path }}/{{ item }}_configdrive/openstack/latest'
-    state: directory
-  with_items: '{{ instances }}'
-
-- name: Create the cloud-init config drive files
-  template:
-    src: '{{ item[1] }}'
-    dest: '{{ libvirt_storage_pool_path }}/{{ item[0] }}_configdrive/openstack/latest/{{ item[1] }}'
-  with_nested:
-    - '{{ instances }}'
-    - [ user-data, meta-data ]
-
-- name: Create the cloud-init config drive
-  command: 'genisoimage -output {{ libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso -volid cidata -joliet -rock user-data meta-data'
-  args:
-    chdir: '{{ libvirt_storage_pool_path }}/{{ item }}_configdrive/openstack/latest'
-    creates: '{{ libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso'
-  with_items: '{{ instances }}'
-
-- name: Create the libvirt storage pool for openshift
-  command: 'virsh -c {{ libvirt_uri }} pool-create-as {{ libvirt_storage_pool }} dir --target {{ libvirt_storage_pool_path }}'
-  ignore_errors: yes
-
-- name: Refresh the libvirt storage pool for openshift
-  command: 'virsh -c {{ libvirt_uri }} pool-refresh {{ libvirt_storage_pool }}'
-
-- name: Create VMs drives
-  command: 'virsh -c {{ libvirt_uri }} vol-create-as {{ libvirt_storage_pool }} {{ item }}.qcow2 10G --format qcow2 --backing-vol {{ base_image_name }} --backing-vol-format qcow2'
-  with_items: '{{ instances }}'
-
-- name: Create VMs
-  virt:
-    name: '{{ item }}'
-    command: define
-    xml: "{{ lookup('template', '../templates/domain.xml') }}"
-    uri: '{{ libvirt_uri }}'
-  with_items: '{{ instances }}'
-
-- name: Start VMs
-  virt:
-    name: '{{ item }}'
-    state: running
-    uri: '{{ libvirt_uri }}'
-  with_items: '{{ instances }}'
-
-- name: Collect MAC addresses of the VMs
-  shell: 'virsh -c {{ libvirt_uri }} dumpxml {{ item }} | xmllint --xpath "string(//domain/devices/interface/mac/@address)" -'
-  register: scratch_mac
-  with_items: '{{ instances }}'
-
-- name: Wait for the VMs to get an IP
-  command: "egrep -c '{{ scratch_mac.results | oo_collect('stdout') | join('|') }}' /proc/net/arp"
-  ignore_errors: yes
-  register: nb_allocated_ips
-  until: nb_allocated_ips.stdout == '{{ instances | length }}'
-  retries: 30
-  delay: 1
-
-- name: Collect IP addresses of the VMs
-  shell: "awk '/{{ item.stdout }}/ {print $1}' /proc/net/arp"
-  register: scratch_ip
-  with_items: '{{ scratch_mac.results }}'
-
-- set_fact:
-    ips: "{{ scratch_ip.results | oo_collect('stdout') }}"
-
-- name: Add new instances
-  add_host:
-    hostname: '{{ item.0 }}'
-    ansible_ssh_host: '{{ item.1 }}'
-    ansible_ssh_user: root
-    groups: 'tag_env-{{ cluster }}, tag_host-type-{{ type }}, tag_env-host-type-{{ cluster }}-openshift-{{ type }}'
-  with_together:
-    - instances
-    - ips
-
-- name: Wait for ssh
-  wait_for:
-    host: '{{ item }}'
-    port: 22
-  with_items: ips
-
-- name: Wait for root user setup
-  command: 'ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null root@{{ item }} echo root user is setup'
-  register: result
-  until: result.rc == 0
-  retries: 30
-  delay: 1
-  with_items: ips

playbooks/libvirt/openshift-cluster/list.yml

@@ -1,43 +1,23 @@
+---
 - name: Generate oo_list_hosts group
   hosts: localhost
-  connection: local
   gather_facts: no
-
-  vars:
-    libvirt_uri: 'qemu:///system'
-
+  vars_files:
+  - vars.yml
   tasks:
-    - name: List VMs
-      virt:
-        command: list_vms
-      register: list_vms
-
-    - name: Collect MAC addresses of the VMs
-      shell: 'virsh -c {{ libvirt_uri }} dumpxml {{ item }} | xmllint --xpath "string(//domain/devices/interface/mac/@address)" -'
-      register: scratch_mac
-      with_items: '{{ list_vms.list_vms }}'
-      when: item|truncate(cluster_id|length+1, True) == '{{ cluster_id }}-...'
-
-    - name: Collect IP addresses of the VMs
-      shell: "awk '/{{ item.stdout }}/ {print $1}' /proc/net/arp"
-      register: scratch_ip
-      with_items: '{{ scratch_mac.results }}'
-      when: item.skipped is not defined
-
-    - name: Add hosts
-      add_host:
-        hostname: '{{ item[0] }}'
-        ansible_ssh_host: '{{ item[1].stdout }}'
-        ansible_ssh_user: root
-        groups: oo_list_hosts
-      with_together:
-        - '{{ list_vms.list_vms }}'
-        - '{{ scratch_ip.results }}'
-      when: item[1].skipped is not defined
+  - set_fact: scratch_group=tag_env-{{ cluster_id }}
+    when: cluster_id != ''
+  - set_fact: scratch_group=all
+    when: cluster_id == ''
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_list_hosts
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[scratch_group] | default([]) | difference(['localhost'])
 
 - name: List Hosts
   hosts: oo_list_hosts
-
   tasks:
-    - debug:
-        msg: 'public:{{ansible_default_ipv4.address}} private:{{ansible_default_ipv4.address}}'
+  - debug:
+      msg: 'public:{{ansible_default_ipv4.address}} private:{{ansible_default_ipv4.address}} deployment-type: {{ hostvars[inventory_hostname].group_names | oo_get_deployment_type_from_groups }}'

playbooks/libvirt/openshift-cluster/tasks/configure_libvirt.yml

@@ -0,0 +1,6 @@
+---
+- include: configure_libvirt_storage_pool.yml
+  when: libvirt_storage_pool is defined and libvirt_storage_pool_path is defined
+
+- include: configure_libvirt_network.yml
+  when: libvirt_network is defined

playbooks/libvirt/openshift-cluster/tasks/configure_libvirt_network.yml

@@ -0,0 +1,27 @@
+---
+- name: Test if libvirt network for openshift already exists
+  command: "virsh -c {{ libvirt_uri }} net-info {{ libvirt_network }}"
+  register: net_info_result
+  changed_when: False
+  failed_when: "net_info_result.rc != 0 and 'error: Network not found:' not in net_info_result.stderr"
+
+- name: Create a temp directory for the template xml file
+  command: "/usr/bin/mktemp -d /tmp/openshift-ansible-XXXXXXX"
+  register: mktemp
+  when: net_info_result.rc == 1
+
+- name: Create network xml file
+  template:
+    src: templates/network.xml
+    dest: "{{ mktemp.stdout }}/network.xml"
+  when: net_info_result.rc == 1
+
+- name: Create libvirt network for openshift
+  command: "virsh -c {{ libvirt_uri }} net-create {{ mktemp.stdout }}/network.xml"
+  when: net_info_result.rc == 1
+
+- name: Remove the temp directory
+  file:
+    path: "{{ mktemp.stdout }}"
+    state: absent
+  when: net_info_result.rc == 1

playbooks/libvirt/openshift-cluster/tasks/configure_libvirt_storage_pool.yml

@@ -0,0 +1,27 @@
+---
+- name: Create libvirt storage directory for openshift
+  file:
+    dest: "{{ libvirt_storage_pool_path }}"
+    state: directory
+
+- acl:
+    default: yes
+    entity: kvm
+    etype: group
+    name: "{{ libvirt_storage_pool_path }}"
+    permissions: rwx
+    state: present
+
+- name: Test if libvirt storage pool for openshift already exists
+  command: "virsh -c {{ libvirt_uri }} pool-info {{ libvirt_storage_pool }}"
+  register: pool_info_result
+  changed_when: False
+  failed_when: "pool_info_result.rc != 0 and 'error: Storage pool not found:' not in pool_info_result.stderr"
+
+- name: Create the libvirt storage pool for openshift
+  command: 'virsh -c {{ libvirt_uri }} pool-create-as {{ libvirt_storage_pool }} dir --target {{ libvirt_storage_pool_path }}'
+  when: pool_info_result.rc == 1
+
+- name: Refresh the libvirt storage pool for openshift
+  command: 'virsh -c {{ libvirt_uri }} pool-refresh {{ libvirt_storage_pool }}'
+  when: pool_info_result.rc == 1

playbooks/libvirt/openshift-cluster/tasks/launch_instances.yml

@@ -0,0 +1,104 @@
+---
+# TODO: Add support for choosing base image based on deployment_type and os
+# wanted (os wanted needs support added in bin/cluster with sane defaults:
+# fedora/centos for origin, rhel for online/enterprise)
+
+# TODO: create a role to encapsulate some of this complexity, possibly also
+# create a module to manage the storage tasks, network tasks, and possibly
+# even handle the libvirt tasks to set metadata in the domain xml and be able
+# to create/query data about vms without having to use xml the python libvirt
+# bindings look like a good candidate for this
+
+- name: Download Base Cloud image
+  get_url:
+    url: '{{ image_url }}'
+    sha256sum: '{{ image_sha256 }}'
+    dest: '{{ os_libvirt_storage_pool_path }}/{{ image_name }}'
+
+- name: Create the cloud-init config drive path
+  file:
+    dest: '{{ os_libvirt_storage_pool_path }}/{{ item }}_configdrive/'
+    state: directory
+  with_items: instances
+
+- name: Create the cloud-init config drive files
+  template:
+    src: '{{ item[1] }}'
+    dest: '{{ os_libvirt_storage_pool_path }}/{{ item[0] }}_configdrive/{{ item[1] }}'
+  with_nested:
+    - instances
+    - [ user-data, meta-data ]
+
+- name: Create the cloud-init config drive
+  command: 'genisoimage -output {{ os_libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso -volid cidata -joliet -rock user-data meta-data'
+  args:
+    chdir: '{{ os_libvirt_storage_pool_path }}/{{ item }}_configdrive/'
+    creates: '{{ os_libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso'
+  with_items: instances
+
+- name: Create VMs drives
+  command: 'virsh -c {{ libvirt_uri }} vol-create-as {{ os_libvirt_storage_pool }} {{ item }}.qcow2 10G --format qcow2 --backing-vol {{ image_name }} --backing-vol-format qcow2'
+  with_items: instances
+
+- name: Create VMs
+  virt:
+    name: '{{ item }}'
+    command: define
+    xml: "{{ lookup('template', '../templates/domain.xml') }}"
+    uri: '{{ libvirt_uri }}'
+  with_items: instances
+
+- name: Start VMs
+  virt:
+    name: '{{ item }}'
+    state: running
+    uri: '{{ libvirt_uri }}'
+  with_items: instances
+
+- name: Collect MAC addresses of the VMs
+  shell: 'virsh -c {{ libvirt_uri }} dumpxml {{ item }} | xmllint --xpath "string(//domain/devices/interface/mac/@address)" -'
+  register: scratch_mac
+  with_items: instances
+
+- name: Wait for the VMs to get an IP
+  command: "egrep -c '{{ scratch_mac.results | oo_collect('stdout') | join('|') }}' /proc/net/arp"
+  ignore_errors: yes
+  register: nb_allocated_ips
+  until: nb_allocated_ips.stdout == '{{ instances | length }}'
+  retries: 30
+  delay: 1
+
+- name: Collect IP addresses of the VMs
+  shell: "awk '/{{ item.stdout }}/ {print $1}' /proc/net/arp"
+  register: scratch_ip
+  with_items: scratch_mac.results
+
+- set_fact:
+    ips: "{{ scratch_ip.results | oo_collect('stdout') }}"
+
+- name: Add new instances
+  add_host:
+    hostname: '{{ item.0 }}'
+    ansible_ssh_host: '{{ item.1 }}'
+    ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+    ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    groups: 'tag_env-{{ cluster }}, tag_host-type-{{ type }}, tag_env-host-type-{{ cluster }}-openshift-{{ type }}'
+  with_together:
+    - instances
+    - ips
+
+- name: Wait for ssh
+  wait_for:
+    host: '{{ item }}'
+    port: 22
+  with_items: ips
+
+- name: Wait for openshift user setup
+  command: 'ssh -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null openshift@{{ item.1 }} echo openshift user is setup'
+  register: result
+  until: result.rc == 0
+  retries: 30
+  delay: 1
+  with_together:
+  - instances
+  - ips

playbooks/libvirt/templates/domain.xml

@@ -1,6 +1,12 @@
 <domain type='kvm' id='8'>
   <name>{{ item }}</name>
   <memory unit='GiB'>1</memory>
+  <metadata xmlns:ansible="https://github.com/ansible/ansible">
+    <ansible:tag>deployment-type-{{ deployment_type }}</ansible:tag>
+    <ansible:tag>env-{{ cluster }}</ansible:tag>
+    <ansible:tag>env-host-type-{{ cluster }}-openshift-{{ type }}</ansible:tag>
+    <ansible:tag>host-type-{{ type }}</ansible:tag>
+  </metadata>
   <currentMemory unit='GiB'>1</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <os>
@@ -24,18 +30,18 @@
     <emulator>/usr/bin/qemu-system-x86_64</emulator>
     <disk type='file' device='disk'>
       <driver name='qemu' type='qcow2'/>
-      <source file='{{ libvirt_storage_pool_path }}/{{ item }}.qcow2'/>
+      <source file='{{ os_libvirt_storage_pool_path }}/{{ item }}.qcow2'/>
       <target dev='vda' bus='virtio'/>
     </disk>
     <disk type='file' device='cdrom'>
       <driver name='qemu' type='raw'/>
-      <source file='{{ libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso'/>
+      <source file='{{ os_libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso'/>
       <target dev='vdb' bus='virtio'/>
       <readonly/>
     </disk>
     <controller type='usb' index='0' />
     <interface type='network'>
-      <source network='default'/>
+      <source network='{{ os_libvirt_network }}'/>
       <model type='virtio'/>
     </interface>
     <serial type='pty'>
@@ -49,7 +55,6 @@
     </channel>
     <input type='tablet' bus='usb' />
     <input type='mouse' bus='ps2'/>
-    <input type='keyboard' bus='ps2'/>
     <graphics type='spice' autoport='yes' />
     <video>
       <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>

playbooks/libvirt/openshift-cluster/templates/meta-data

@@ -0,0 +1,3 @@
+instance-id: {{ item[0] }}
+hostname: {{ item[0] }}
+local-hostname: {{ item[0] }}.example.com

playbooks/libvirt/openshift-cluster/templates/network.xml

@@ -0,0 +1,23 @@
+<network>
+  <name>openshift-ansible</name>
+  <forward mode='nat'>
+    <nat>
+      <port start='1024' end='65535'/>
+    </nat>
+  </forward>
+  <!-- TODO: query for first available virbr interface available -->
+  <bridge name='virbr3' stp='on' delay='0'/>
+  <!-- TODO: make overridable -->
+  <domain name='example.com'/>
+  <dns>
+    <!-- TODO: automatically add host entries -->
+  </dns>
+  <!-- TODO: query for available address space -->
+  <ip address='192.168.55.1' netmask='255.255.255.0'>
+    <dhcp>
+      <range start='192.168.55.2' end='192.168.55.254'/>
+      <!-- TODO: add static entries addresses for the hosts to be created -->
+    </dhcp>
+  </ip>
+</network>
+

playbooks/libvirt/openshift-cluster/templates/user-data

@@ -0,0 +1,23 @@
+#cloud-config
+disable_root: true
+
+hostname: {{ item[0] }}
+fqdn: {{ item[0] }}.example.com
+manage_etc_hosts: true
+
+users:
+  - default
+  - name: root
+    ssh_authorized_keys:
+    - {{ lookup('file', '~/.ssh/id_rsa.pub') }}
+
+system_info:
+  default_user:
+    name: openshift
+    sudo: ["ALL=(ALL) NOPASSWD: ALL"]
+
+ssh_authorized_keys:
+  - {{ lookup('file', '~/.ssh/id_rsa.pub') }}
+
+bootcmd:
+  - NETWORK_CONFIG=/etc/sysconfig/network-scripts/ifcfg-eth0; if ! grep DHCP_HOSTNAME ${NETWORK_CONFIG}; then echo 'DHCP_HOSTNAME="{{ item[0] }}.example.com"' >> ${NETWORK_CONFIG}; fi; pkill -9 dhclient; service network restart

playbooks/libvirt/openshift-cluster/terminate.yml

@@ -1,41 +1,44 @@
+---
+# TODO: does not handle a non-existant cluster gracefully
+
 - name: Terminate instance(s)
   hosts: localhost
-  connection: local
   gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - set_fact: cluster_group=tag_env-{{ cluster_id }}
+  - add_host:
+      name: "{{ item }}"
+      groups: oo_hosts_to_terminate
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups[cluster_group] | default([])
 
-  vars:
-    libvirt_storage_pool_path: "{{ lookup('env','HOME') }}/libvirt-storage-pool-openshift"
-    libvirt_storage_pool: 'openshift'
-    libvirt_uri: 'qemu:///system'
+  - name: Destroy VMs
+    virt:
+      name: '{{ item[0] }}'
+      command: '{{ item[1] }}'
+      uri: '{{ libvirt_uri }}'
+    with_nested:
+    - groups['oo_hosts_to_terminate']
+    - [ destroy, undefine ]
 
-  tasks:
-    - name: List VMs
-      virt:
-        command: list_vms
-      register: list_vms
+  - name: Delete VMs drives
+    command: 'virsh -c {{ libvirt_uri }} vol-delete --pool {{ libvirt_storage_pool }} {{ item }}.qcow2'
+    args:
+      removes: '{{ libvirt_storage_pool_path }}/{{ item }}.qcow2'
+    with_items: groups['oo_hosts_to_terminate']
 
-    - name: Destroy VMs
-      virt:
-        name: '{{ item[0] }}'
-        command: '{{ item[1] }}'
-        uri: '{{ libvirt_uri }}'
-      with_nested:
-        - '{{ list_vms.list_vms }}'
-        - [ destroy, undefine ]
-      when: item[0]|truncate(cluster_id|length+1, True) == '{{ cluster_id }}-...'
+  - name: Delete the VM cloud-init image
+    file:
+      path: '{{ libvirt_storage_pool_path }}/{{ item }}_cloud-init.iso'
+      state: absent
+    with_items: groups['oo_hosts_to_terminate']
 
-    - name: Delete VMs config drive
-      file:
-        path: '{{ libvirt_storage_pool_path }}/{{ item }}_configdrive/openstack'
-        state: absent
-      with_items: '{{ list_vms.list_vms }}'
-      when: item|truncate(cluster_id|length+1, True) == '{{ cluster_id }}-...'
+  - name: Remove the cloud-init config directory
+    file:
+      path: '{{ libvirt_storage_pool_path }}/{{ item }}_configdrive/'
+      state: absent
+    with_items: groups['oo_hosts_to_terminate']
 
-    - name: Delete VMs drives
-      command: 'virsh -c {{ libvirt_uri }} vol-delete --pool {{ libvirt_storage_pool }} {{ item[0] }}{{ item[1] }}'
-      args:
-        removes: '{{ libvirt_storage_pool_path }}/{{ item[0] }}{{ item[1] }}'
-      with_nested:
-        - '{{ list_vms.list_vms }}'
-        - [ '_configdrive', '_cloud-init.iso', '.qcow2' ]
-      when: item[0]|truncate(cluster_id|length+1, True) == '{{ cluster_id }}-...'

playbooks/libvirt/openshift-cluster/update.yml

@@ -0,0 +1,18 @@
+---
+- name: Populate oo_hosts_to_update group
+  hosts: localhost
+  gather_facts: no
+  vars_files:
+  - vars.yml
+  tasks:
+  - name: Evaluate oo_hosts_to_update
+    add_host:
+      name: "{{ item }}"
+      groups: oo_hosts_to_update
+      ansible_ssh_user: "{{ deployment_vars[deployment_type].ssh_user }}"
+      ansible_sudo: "{{ deployment_vars[deployment_type].sudo }}"
+    with_items: groups["tag_env-host-type-{{ cluster_id }}-openshift-master"] | union(groups["tag_env-host-type-{{ cluster_id }}-openshift-node"]) | default([])
+
+- include: ../../common/openshift-cluster/update_repos_and_packages.yml
+
+- include: config.yml

playbooks/libvirt/openshift-cluster/vars.yml

@@ -1,7 +1,33 @@
-# base_image_url: http://download.fedoraproject.org/pub/fedora/linux/releases/21/Cloud/Images/x86_64/Fedora-Cloud-Base-20141203-21.x86_64.qcow2
-# base_image_name: Fedora-Cloud-Base-20141203-21.x86_64.qcow2
-# base_image_sha256: 3a99bb89f33e3d4ee826c8160053cdb8a72c80cd23350b776ce73cd244467d86
+---
+libvirt_storage_pool_path: "{{ lookup('env','HOME') }}/libvirt-storage-pool-openshift-ansible"
+libvirt_storage_pool: 'openshift-ansible'
+libvirt_network: openshift-ansible
+libvirt_uri: 'qemu:///system'
 
-base_image_url: http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
-base_image_name: CentOS-7-x86_64-GenericCloud.qcow2
-base_image_sha256: e324e3ab1d24a1bbf035ddb365e7f9058c0b454acf48d7aa15c5519fae5998ab
+deployment_vars:
+  origin:
+    image:
+      url: "http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2"
+      name: CentOS-7-x86_64-GenericCloud.qcow2
+      sha256: e324e3ab1d24a1bbf035ddb365e7f9058c0b454acf48d7aa15c5519fae5998ab
+    ssh_user: openshift
+    sudo: yes
+  online:
+    image:
+      url:
+      name:
+      sha256:
+    ssh_user: root
+    sudo: no
+  enterprise:
+    image:
+      url:
+      name:
+      sha256:
+    ssh_user: openshift
+    sudo: yes
+#  origin:
+#    fedora:
+#      url: "http://download.fedoraproject.org/pub/fedora/linux/releases/21/Cloud/Images/x86_64/Fedora-Cloud-Base-20141203-21.x86_64.qcow2"
+#      name: Fedora-Cloud-Base-20141203-21.x86_64.qcow2
+#      sha256: 3a99bb89f33e3d4ee826c8160053cdb8a72c80cd23350b776ce73cd244467d86

playbooks/libvirt/openshift-master/config.yml

@@ -1,21 +0,0 @@
-- name: master/config.yml, populate oo_masters_to_config host group if needed
-  hosts: localhost
-  gather_facts: no
-  tasks:
-    - name: "Evaluate oo_host_group_exp if it's set"
-      add_host:
-        name: '{{ item }}'
-        groups: oo_masters_to_config
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
-
-- name: Configure instances
-  hosts: oo_masters_to_config
-  vars:
-    openshift_hostname: '{{ ansible_default_ipv4.address }}'
-  vars_files:
-    - vars.yml
-  roles:
-    - openshift_master
-    - pods
-    - os_env_extras

playbooks/libvirt/openshift-master/roles

@@ -1 +0,0 @@
-../../../roles

playbooks/libvirt/openshift-master/vars.yml

@@ -1 +0,0 @@
-openshift_debug_level: 4

playbooks/libvirt/openshift-node/config.yml

@@ -1,102 +0,0 @@
-- name: node/config.yml, populate oo_nodes_to_config host group if needed
-  hosts: localhost
-  gather_facts: no
-  tasks:
-    - name: "Evaluate oo_host_group_exp if it's set"
-      add_host:
-        name: '{{ item }}'
-        groups: oo_nodes_to_config
-      with_items: "{{ oo_host_group_exp | default('') }}"
-      when: oo_host_group_exp is defined
-
-    - add_host:
-        name: "{{ groups['tag_env-host-type-' ~ cluster_id ~ '-openshift-master'][0] }}"
-        groups: oo_first_master
-      when: oo_host_group_exp is defined
-
-
-- name: Gather and set facts for hosts to configure
-  hosts: oo_nodes_to_config
-  roles:
-  - openshift_facts
-  tasks:
-  # Since the master is registering the nodes before they are configured, we
-  # need to make sure to set the node properties beforehand if we do not want
-  # the defaults
-  - openshift_facts:
-      role: "{{ item.role }}"
-      local_facts: "{{ item.local_facts }}"
-    with_items:
-    - role: common
-      local_facts:
-        hostname: "{{ ansible_default_ipv4.address }}"
-    - role: node
-      local_facts:
-        external_id: "{{ openshift_node_external_id | default(None) }}"
-        resources_cpu: "{{ openshfit_node_resources_cpu | default(None) }}"
-        resources_memory: "{{ openshfit_node_resources_memory | default(None) }}"
-        pod_cidr: "{{ openshfit_node_pod_cidr | default(None) }}"
-        labels: "{{ openshfit_node_labels | default(None) }}"
-        annotations: "{{ openshfit_node_annotations | default(None) }}"
-
-
-- name: Register nodes
-  hosts: oo_first_master
-  vars:
-    openshift_nodes: "{{ hostvars
-          | oo_select_keys(groups['oo_nodes_to_config']) }}"
-  roles:
-  - openshift_register_nodes
-  tasks:
-  - name: Create local temp directory for syncing certs
-    local_action: command /usr/bin/mktemp -d /tmp/openshift-ansible-XXXXXXX
-    register: mktemp
-
-  - name: Sync master certs to localhost
-    synchronize:
-      mode: pull
-      checksum: yes
-      src: /var/lib/openshift/openshift.local.certificates
-      dest: "{{ mktemp.stdout }}"
-
-- name: Configure instances
-  hosts: oo_nodes_to_config
-  vars_files:
-  - vars.yml
-  vars:
-    sync_tmpdir: "{{ hostvars[groups['oo_first_master'][0]].mktemp.stdout }}"
-    cert_parent_rel_path: openshift.local.certificates
-    cert_rel_path: "{{ cert_parent_rel_path }}/node-{{ openshift.common.hostname }}"
-    cert_base_path: /var/lib/openshift
-    cert_parent_path: "{{ cert_base_path }}/{{ cert_parent_rel_path }}"
-    cert_path: "{{ cert_base_path }}/{{ cert_rel_path }}"
-  pre_tasks:
-  - name: Ensure certificate directories exists
-    file:
-      path: "{{ item }}"
-      state: directory
-    with_items:
-    - "{{ cert_path }}"
-    - "{{ cert_parent_path }}/ca"
-
-  # TODO: notify restart openshift-node and/or restart openshift-sdn-node,
-  # possibly test service started time against certificate/config file
-  # timestamps in openshift-node or openshift-sdn-node to trigger notify
-  - name: Sync certs to nodes
-    synchronize:
-      checksum: yes
-      src: "{{ item.src }}"
-      dest: "{{ item.dest }}"
-      owner: no
-      group: no
-    with_items:
-    - src: "{{ sync_tmpdir }}/{{ cert_rel_path }}"
-      dest: "{{ cert_parent_path }}"
-    - src: "{{ sync_tmpdir }}/{{ cert_parent_rel_path }}/ca/cert.crt"
-      dest: "{{ cert_parent_path }}/ca/cert.crt"
-  - local_action: file name={{ sync_tmpdir }} state=absent
-    run_once: true
-  roles:
-    - openshift_node
-    - os_env_extras
-    - os_env_extras_node

playbooks/libvirt/openshift-node/vars.yml

@@ -1 +0,0 @@
-openshift_debug_level: 4

playbooks/libvirt/templates/meta-data

@@ -1,2 +0,0 @@
-instance-id: {{ item[0] }}
-local-hostname: {{ item[0] }}

playbooks/libvirt/templates/user-data

@@ -1,10 +0,0 @@
-#cloud-config
-
-disable_root: 0
-
-system_info:
-  default_user:
-    name: root
-
-ssh_authorized_keys:
-  - {{ lookup('file', '~/.ssh/id_rsa.pub') }}

roles/openshift_common/tasks/main.yml

@@ -1,7 +1,7 @@
 ---
 - name: Set common OpenShift facts
   openshift_facts:
-    role: 'common'
+    role: common
     local_facts:
       cluster_id: "{{ openshift_cluster_id | default('default') }}"
       debug_level: "{{ openshift_debug_level | default(0) }}"
@@ -10,7 +10,7 @@
       public_hostname: "{{ openshift_public_hostname | default(None) }}"
       public_ip: "{{ openshift_public_ip | default(None) }}"
       use_openshift_sdn: "{{ openshift_use_openshift_sdn | default(None) }}"
-
+      deployment_type: "{{ openshift_deployment_type }}"
 - name: Set hostname
   hostname: name={{ openshift.common.hostname }}
 

roles/openshift_common/vars/main.yml

@@ -5,3 +5,7 @@
 # chains with the public zone (or the zone associated with the correct
 # interfaces)
 os_firewall_use_firewalld: False
+
+openshift_cert_parent_dir: /var/lib/openshift
+openshift_cert_relative_dir: openshift.local.certificates
+openshift_cert_dir: "{{ openshift_cert_parent_dir }}/{{ openshift_cert_relative_dir }}"

roles/openshift_facts/library/openshift_facts.py

@@ -21,8 +21,11 @@ class OpenShiftFactsUnsupportedRoleError(Exception):
 class OpenShiftFactsFileWriteError(Exception):
     pass
 
+class OpenShiftFactsMetadataUnavailableError(Exception):
+    pass
+
 class OpenShiftFacts():
-    known_roles = ['common', 'master', 'node', 'master_sdn', 'node_sdn']
+    known_roles = ['common', 'master', 'node', 'master_sdn', 'node_sdn', 'dns']
 
     def __init__(self, role, filename, local_facts):
         self.changed = False
@@ -169,20 +172,18 @@ class OpenShiftFacts():
         return hostname
 
     def get_defaults(self, roles):
-        hardware_facts = self.get_hardware_facts()
-        net_facts = self.get_net_facts()
-        base_facts = self.get_base_facts()
+        ansible_facts = self.get_ansible_facts()
 
         defaults = dict()
 
         common = dict(use_openshift_sdn=True)
-        ip = net_facts['default_ipv4']['address']
+        ip = ansible_facts['default_ipv4']['address']
         common['ip'] = ip
         common['public_ip'] = ip
 
         rc, output, error = module.run_command(['hostname', '-f'])
         hostname_f = output.strip() if rc == 0 else ''
-        hostname_values = [hostname_f, base_facts['nodename'], base_facts['fqdn']]
+        hostname_values = [hostname_f, ansible_facts['nodename'], ansible_facts['fqdn']]
         hostname = self.choose_hostname(hostname_values)
 
         common['hostname'] = hostname
@@ -196,14 +197,14 @@ class OpenShiftFacts():
             master = dict(api_use_ssl=True, api_port='8443',
                     console_use_ssl=True, console_path='/console',
                     console_port='8443', etcd_use_ssl=False,
-                    etcd_port='4001')
+                    etcd_port='4001', portal_net='172.30.17.0/24')
             defaults['master'] = master
 
         if 'node' in roles:
             node = dict(external_id=common['hostname'], pod_cidr='',
                         labels={}, annotations={})
-            node['resources_cpu'] = hardware_facts['processor_cores']
-            node['resources_memory'] = int(int(hardware_facts['memtotal_mb']) * 1024 * 1024 * 0.75)
+            node['resources_cpu'] = ansible_facts['processor_cores']
+            node['resources_memory'] = int(int(ansible_facts['memtotal_mb']) * 1024 * 1024 * 0.75)
             defaults['node'] = node
 
         return defaults
@@ -226,8 +227,7 @@ class OpenShiftFacts():
     def query_metadata(self, metadata_url, headers=None, expect_json=False):
         r, info = fetch_url(module, metadata_url, headers=headers)
         if info['status'] != 200:
-            module.fail_json(msg='Failed to query metadata', result=r,
-                             info=info)
+            raise OpenShiftFactsMetadataUnavailableError("Metadata unavailable")
         if expect_json:
             return module.from_json(r.read())
         else:
@@ -252,40 +252,27 @@ class OpenShiftFacts():
 
     def get_provider_metadata(self, metadata_url, supports_recursive=False,
                           headers=None, expect_json=False):
-        if supports_recursive:
-            metadata = self.query_metadata(metadata_url, headers, expect_json)
-        else:
-            metadata = self.walk_metadata(metadata_url, headers, expect_json)
+        try:
+            if supports_recursive:
+                metadata = self.query_metadata(metadata_url, headers, expect_json)
+            else:
+                metadata = self.walk_metadata(metadata_url, headers, expect_json)
+        except OpenShiftFactsMetadataUnavailableError as e:
+            metadata = None
         return metadata
 
-    def get_hardware_facts(self):
-        if not hasattr(self, 'hardware_facts'):
-            self.hardware_facts = Hardware().populate()
-        return self.hardware_facts
-
-    def get_base_facts(self):
-        if not hasattr(self, 'base_facts'):
-            self.base_facts = Facts().populate()
-        return self.base_facts
-
-    def get_virt_facts(self):
-        if not hasattr(self, 'virt_facts'):
-            self.virt_facts = Virtual().populate()
-        return self.virt_facts
-
-    def get_net_facts(self):
-        if not hasattr(self, 'net_facts'):
-            self.net_facts = Network(module).populate()
-        return self.net_facts
+    def get_ansible_facts(self):
+        if not hasattr(self, 'ansible_facts'):
+            self.ansible_facts = ansible_facts(module)
+        return self.ansible_facts
 
     def guess_host_provider(self):
         # TODO: cloud provider facts should probably be submitted upstream
-        virt_facts = self.get_virt_facts()
-        hardware_facts = self.get_hardware_facts()
-        product_name = hardware_facts['product_name']
-        product_version = hardware_facts['product_version']
-        virt_type = virt_facts['virtualization_type']
-        virt_role = virt_facts['virtualization_role']
+        ansible_facts = self.get_ansible_facts()
+        product_name = ansible_facts['product_name']
+        product_version = ansible_facts['product_version']
+        virt_type = ansible_facts['virtualization_type']
+        virt_role = ansible_facts['virtualization_role']
         provider = None
         metadata = None
 
@@ -300,8 +287,9 @@ class OpenShiftFacts():
                                                   True)
 
             # Filter sshKeys and serviceAccounts from gce metadata
-            metadata['project']['attributes'].pop('sshKeys', None)
-            metadata['instance'].pop('serviceAccounts', None)
+            if metadata:
+                metadata['project']['attributes'].pop('sshKeys', None)
+                metadata['instance'].pop('serviceAccounts', None)
         elif virt_type == 'xen' and virt_role == 'guest' and re.match(r'.*\.amazon$', product_version):
             provider = 'ec2'
             metadata_url = 'http://169.254.169.254/latest/meta-data/'
@@ -310,12 +298,18 @@ class OpenShiftFacts():
             provider = 'openstack'
             metadata_url = 'http://169.254.169.254/openstack/latest/meta_data.json'
             metadata = self.get_provider_metadata(metadata_url, True, None, True)
-            ec2_compat_url = 'http://169.254.169.254/latest/meta-data/'
-            metadata['ec2_compat'] = self.get_provider_metadata(ec2_compat_url)
 
-            # Filter public_keys  and random_seed from openstack metadata
-            metadata.pop('public_keys', None)
-            metadata.pop('random_seed', None)
+            if metadata:
+                ec2_compat_url = 'http://169.254.169.254/latest/meta-data/'
+                metadata['ec2_compat'] = self.get_provider_metadata(ec2_compat_url)
+
+                # Filter public_keys  and random_seed from openstack metadata
+                metadata.pop('public_keys', None)
+                metadata.pop('random_seed', None)
+
+                if not metadata['ec2_compat']:
+                    metadata = None
+
         return dict(name=provider, metadata=metadata)
 
     def normalize_provider_facts(self, provider, metadata):
@@ -479,4 +473,6 @@ def main():
 from ansible.module_utils.basic import *
 from ansible.module_utils.facts import *
 from ansible.module_utils.urls import *
-main()
+
+if __name__ == '__main__':
+    main()

roles/openshift_master/tasks/main.yml

@@ -11,33 +11,67 @@
       api_url: "{{ openshift_master_api_url | default(None) }}"
       api_use_ssl: "{{ openshift_master_api_use_ssl | default(None) }}"
       public_api_url: "{{ openshift_master_public_api_url | default(None) }}"
+      console_path: "{{ openshift_master_console_path | default(None) }}"
       console_port: "{{ openshift_master_console_port | default(None) }}"
       console_url: "{{ openshift_master_console_url | default(None) }}"
       console_use_ssl: "{{ openshift_master_console_use_ssl | default(None) }}"
       public_console_url: "{{ openshift_master_public_console_url | default(None) }}"
+      etcd_port: "{{ openshift_master_etcd_port | default(None) }}"
       etcd_use_ssl: "{{ openshift_master_etcd_use_ssl | default(None) }}"
+      portal_net: "{{ openshift_master_portal_net | default(None) }}"
+
+# TODO: These values need to be configurable
+- name: Set dns OpenShift facts
+  openshift_facts:
+    role: 'dns'
+    local_facts:
+      ip: "{{ openshift.common.ip }}"
+      domain: local
 
 - name: Install OpenShift Master package
   yum: pkg=openshift-master state=installed
+  register: install_result
+
+- name: Reload systemd units
+  command: systemctl daemon-reload
+  when: install_result | changed
+
+- name: Create certificate parent directory if it doesn't exist
+  file:
+    path: "{{ openshift_cert_parent_dir }}"
+    state: directory
+
+- name: Create config parent directory if it doesn't exist
+  file:
+    path: "{{ openshift_master_config | dirname }}"
+    state: directory
+
+# TODO: should probably use a template lookup for this
+# TODO: should allow for setting --etcd, --kubernetes options
+# TODO: recreate config if values change
+- name: Use enterprise default for openshift_registry_url if not set
+  set_fact:
+    openshift_registry_url: "openshift3_beta/ose-${component}:${version}"
+  when: openshift.common.deployment_type == 'enterprise' and openshift_registry_url is not defined
+- name: Create master config
+  command: >
+    /usr/bin/openshift start master --write-config
+    --config={{ openshift_master_config }}
+    --portal-net={{ openshift.master.portal_net }}
+    --master={{ openshift.master.api_url }}
+    --public-master={{ openshift.master.public_api_url }}
+    --listen={{ 'https' if openshift.master.api_use_ssl else 'http' }}://0.0.0.0:{{ openshift.master.api_port }}
+    {{ ('--images=' ~ openshift_registry_url) if openshift_registry_url is defined else '' }}
+    {{ ('--nodes=' ~ openshift_node_ips | join(',')) if openshift_node_ips is defined else '' }}
+  args:
+    chdir: "{{ openshift_cert_parent_dir }}"
+    creates: "{{ openshift_master_config }}"
 
-# TODO: We should pre-generate the master config and point to the generated
-# config rather than setting command line flags here
 - name: Configure OpenShift settings
   lineinfile:
     dest: /etc/sysconfig/openshift-master
     regexp: '^OPTIONS='
-    line: "OPTIONS=\"--master={{ openshift.common.hostname }} --public-master={{ openshift.common.public_hostname }} {% if openshift_node_ips %} --nodes={{ openshift_node_ips | join(',') }} {% endif %} --loglevel={{ openshift.master.debug_level }}\""
-  notify:
-  - restart openshift-master
-
-# TODO: should this be populated by a fact based on the deployment type
-# (origin, online, enterprise)?
-- name: Set default registry url
-  lineinfile:
-    dest: /etc/sysconfig/openshift-master
-    regexp: '^IMAGES='
-    line: "IMAGES={{ openshift_registry_url }}"
-  when: openshift_registry_url is defined
+    line: "OPTIONS=\"--config={{ openshift_master_config }} --loglevel={{ openshift.master.debug_level }}\""
   notify:
   - restart openshift-master
 
@@ -53,6 +87,6 @@
 # TODO: Update this file if the contents of the source file are not present in
 # the dest file, will need to make sure to ignore things that could be added
 - name: Configure root user kubeconfig
-  command: cp /var/lib/openshift/openshift.local.certificates/openshift-client/.kubeconfig /root/.kube/.kubeconfig
+  command: cp {{ openshift_cert_dir }}/openshift-client/.kubeconfig /root/.kube/.kubeconfig
   args:
     creates: /root/.kube/.kubeconfig

roles/openshift_master/vars/main.yml

@@ -0,0 +1,5 @@
+---
+openshift_master_config: /etc/openshift/master.yaml
+openshift_master_ca_dir: "{{ openshift_cert_dir }}/ca"
+openshift_master_ca_cert: "{{ openshift_master_ca_dir }}/cert.crt"
+openshift_master_ca_key: "{{ openshift_master_ca_dir }}/key.key"

roles/openshift_node/tasks/main.yml

@@ -13,17 +13,22 @@
   failed_when: not result.stat.exists
   register: result
   with_items:
-  - "{{ cert_path }}"
-  - "{{ cert_path }}/cert.crt"
-  - "{{ cert_path }}/key.key"
-  - "{{ cert_path }}/.kubeconfig"
-  - "{{ cert_path }}/server.crt"
-  - "{{ cert_path }}/server.key"
-  - "{{ cert_parent_path }}/ca/cert.crt"
-  #- "{{ cert_path }}/node.yaml"
+  - "{{ openshift_node_cert_dir }}"
+  - "{{ openshift_node_cert_dir }}/ca.crt"
+  - "{{ openshift_node_cert_dir }}/client.crt"
+  - "{{ openshift_node_cert_dir }}/client.key"
+  - "{{ openshift_node_cert_dir }}/.kubeconfig"
+  - "{{ openshift_node_cert_dir }}/node-config.yaml"
+  - "{{ openshift_node_cert_dir }}/server.crt"
+  - "{{ openshift_node_cert_dir }}/server.key"
 
 - name: Install OpenShift Node package
   yum: pkg=openshift-node state=installed
+  register: install_result
+
+- name: Reload systemd units
+  command: systemctl daemon-reload
+  when: install_result | changed
 
 # --create-certs=false is a temporary workaround until
 # https://github.com/openshift/origin/pull/1361 is merged upstream and it is
@@ -32,16 +37,7 @@
   lineinfile:
     dest: /etc/sysconfig/openshift-node
     regexp: '^OPTIONS='
-    line: "OPTIONS=\"--hostname={{ openshift.common.hostname }} --loglevel={{ openshift.node.debug_level }} --create-certs=false\""
-  notify:
-  - restart openshift-node
-
-- name: Set default registry url
-  lineinfile:
-    dest: /etc/sysconfig/openshift-node
-    regexp: '^IMAGES='
-    line: "IMAGES={{ openshift_registry_url }}"
-  when: openshift_registry_url is defined
+    line: "OPTIONS=\"--loglevel={{ openshift.node.debug_level }} --config={{ openshift_node_cert_dir }}/node-config.yaml\""
   notify:
   - restart openshift-node
 

roles/openshift_node/vars/main.yml

@@ -0,0 +1,2 @@
+---
+openshift_node_cert_dir: /etc/openshift/node

roles/openshift_register_nodes/defaults/main.yml

@@ -1,5 +1,2 @@
 ---
 openshift_kube_api_version: v1beta1
-openshift_cert_dir: openshift.local.certificates
-openshift_cert_dir_parent: /var/lib/openshift
-openshift_cert_dir_abs: "{{ openshift_cert_dir_parent ~ '/' ~ openshift_cert_dir }}"

roles/openshift_register_nodes/library/kubernetes_register_node.py

@@ -97,10 +97,8 @@ class ClientConfigException(Exception):
 
 class ClientConfig:
     def __init__(self, client_opts, module):
-        _, output, error = module.run_command(["/usr/bin/openshift", "ex",
-                                               "config", "view", "-o",
-                                               "json"] + client_opts,
-                                              check_rc = True)
+        kubectl = module.params['kubectl_cmd']
+        _, output, error = module.run_command(kubectl + ["config", "view", "-o", "json"] + client_opts, check_rc = True)
         self.config = json.loads(output)
 
         if not (bool(self.config['clusters']) or
@@ -146,6 +144,9 @@ class ClientConfig:
     def get_cluster_for_context(self, context):
         return self.get_value_for_context(context, 'cluster')
 
+    def get_namespace_for_context(self, context):
+        return self.get_value_for_context(context, 'namespace')
+
 class Util:
     @staticmethod
     def remove_empty_elements(mapping):
@@ -247,15 +248,15 @@ class Node:
         return Util.remove_empty_elements(node)
 
     def exists(self):
-        _, output, error = self.module.run_command(["/usr/bin/osc", "get",
-                                                    "nodes"] +  self.client_opts,
-                                                   check_rc = True)
+        kubectl = self.module.params['kubectl_cmd']
+        _, output, error = self.module.run_command(kubectl + ["get", "nodes"] +  self.client_opts, check_rc = True)
         if re.search(self.module.params['name'], output, re.MULTILINE):
             return True
         return False
 
     def create(self):
-        cmd = ['/usr/bin/osc'] + self.client_opts + ['create', 'node', '-f', '-']
+        kubectl = self.module.params['kubectl_cmd']
+        cmd = kubectl + self.client_opts + ['create', '-f', '-']
         rc, output, error = self.module.run_command(cmd,
                                                data=self.module.jsonify(self.get_node()))
         if rc != 0:
@@ -273,24 +274,26 @@ class Node:
 
 def main():
     module = AnsibleModule(
-        argument_spec      = dict(
-            name           = dict(required = True, type = 'str'),
-            host_ip        = dict(type = 'str'),
-            hostnames      = dict(type = 'list', default = []),
-            external_ips   = dict(type = 'list', default = []),
-            internal_ips   = dict(type = 'list', default = []),
-            api_version    = dict(type = 'str', default = 'v1beta1', # TODO: after kube rebase, we can default to v1beta3
-                                  choices = ['v1beta1', 'v1beta3']),
-            cpu            = dict(type = 'str'),
-            memory         = dict(type = 'str'),
-            labels         = dict(type = 'dict', default = {}), # TODO: needs documented
-            annotations    = dict(type = 'dict', default = {}), # TODO: needs documented
-            pod_cidr       = dict(type = 'str'), # TODO: needs documented
-            external_id    = dict(type = 'str'), # TODO: needs documented
-            client_config  = dict(type = 'str'), # TODO: needs documented
-            client_cluster = dict(type = 'str', default = 'master'), # TODO: needs documented
-            client_context = dict(type = 'str', default = 'master'), # TODO: needs documented
-            client_user    = dict(type = 'str', default = 'admin') # TODO: needs documented
+        argument_spec        = dict(
+            name             = dict(required = True, type = 'str'),
+            host_ip          = dict(type = 'str'),
+            hostnames        = dict(type = 'list', default = []),
+            external_ips     = dict(type = 'list', default = []),
+            internal_ips     = dict(type = 'list', default = []),
+            api_version      = dict(type = 'str', default = 'v1beta1', # TODO: after kube rebase, we can default to v1beta3
+                                    choices = ['v1beta1', 'v1beta3']),
+            cpu              = dict(type = 'str'),
+            memory           = dict(type = 'str'),
+            labels           = dict(type = 'dict', default = {}), # TODO: needs documented
+            annotations      = dict(type = 'dict', default = {}), # TODO: needs documented
+            pod_cidr         = dict(type = 'str'), # TODO: needs documented
+            external_id      = dict(type = 'str'), # TODO: needs documented
+            client_config    = dict(type = 'str'), # TODO: needs documented
+            client_cluster   = dict(type = 'str', default = 'master'), # TODO: needs documented
+            client_context   = dict(type = 'str', default = 'default'), # TODO: needs documented
+            client_namespace = dict(type = 'str', default = 'default'), # TODO: needs documented
+            client_user      = dict(type = 'str', default = 'system:openshift-client'), # TODO: needs documented
+            kubectl_cmd      = dict(type = 'list', default = ['kubectl']) # TODO: needs documented
         ),
         mutually_exclusive = [
             ['host_ip', 'external_ips'],
@@ -333,14 +336,16 @@ def main():
 
     client_cluster = module.params['client_cluster']
     if config.has_cluster(client_cluster):
-        if client_cluster != config.get_cluster_for_context(client_cluster):
+        if client_cluster != config.get_cluster_for_context(client_context):
             client_opts.append("--cluster=%s" % client_cluster)
     else:
         module.fail_json(msg="Cluster %s not found in client config" %
                          client_cluster)
 
-    # TODO: provide sane defaults for some (like hostname, externalIP,
-    # internalIP, etc)
+    client_namespace = module.params['client_namespace']
+    if client_namespace != config.get_namespace_for_context(client_context):
+        client_opts.append("--namespace=%s" % client_namespace)
+
     node = Node(module, client_opts, module.params['api_version'],
                 module.params['name'], module.params['host_ip'],
                 module.params['hostnames'], module.params['external_ips'],

roles/openshift_register_nodes/tasks/main.yml

@@ -3,53 +3,37 @@
 # TODO: recreate master/node configs if settings that affect the configs
 # change (hostname, public_hostname, ip, public_ip, etc)
 
-# TODO: create a failed_when condition
-- name: Create node server certificates
-  command: >
-    /usr/bin/openshift admin create-server-cert
-    --overwrite=false
-    --cert={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/server.crt
-    --key={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/server.key
-    --hostnames={{ [item.openshift.common.hostname,
-                    item.openshift.common.public_hostname]|unique|join(",") }}
-  args:
-    chdir: "{{ openshift_cert_dir_parent }}"
-    creates: "{{ openshift_cert_dir_abs }}/node-{{ item.openshift.common.hostname }}/server.crt"
-  with_items: openshift_nodes
-  register: server_cert_result
-
-# TODO: create a failed_when condition
-- name: Create node client certificates
-  command: >
-    /usr/bin/openshift admin create-node-cert
-    --overwrite=false
-    --cert={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/cert.crt
-    --key={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/key.key
-    --node-name={{ item.openshift.common.hostname }}
-  args:
-    chdir: "{{ openshift_cert_dir_parent }}"
-    creates: "{{ openshift_cert_dir_abs }}/node-{{ item.openshift.common.hostname }}/cert.crt"
-  with_items: openshift_nodes
-  register: node_cert_result
 
+# TODO: use a template lookup here
 # TODO: create a failed_when condition
-- name: Create kubeconfigs for nodes
+- name: Use enterprise default for openshift_registry_url if not set
+  set_fact:
+    openshift_registry_url: "openshift3_beta/ose-${component}:${version}"
+  when: openshift.common.deployment_type == 'enterprise' and openshift_registry_url is not defined
+- name: Create node config
   command: >
-    /usr/bin/openshift admin create-kubeconfig
-    --client-certificate={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/cert.crt
-    --client-key={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/key.key
-    --kubeconfig={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}/.kubeconfig
-    --master={{ openshift.master.api_url }}
-    --public-master={{ openshift.master.public_api_url }}
+    /usr/bin/openshift admin create-node-config
+      --node-dir={{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}
+      --node={{ item.openshift.common.hostname }}
+      --hostnames={{ [item.openshift.common.hostname, item.openshift.common.public_hostname]|unique|join(",") }}
+      --dns-domain={{ openshift.dns.domain }}
+      --dns-ip={{ openshift.dns.ip }}
+      --master={{ openshift.master.api_url }}
+      --signer-key={{ openshift_master_ca_key }}
+      --signer-cert={{ openshift_master_ca_cert }}
+      --certificate-authority={{ openshift_master_ca_cert }}
+      --signer-serial={{ openshift_master_ca_dir }}/serial.txt
+      --node-client-certificate-authority={{ openshift_master_ca_cert }}
+      {{ ('--images=' ~ openshift_registry_url) if openshift_registry_url is defined else '' }}
+      --listen=https://0.0.0.0:10250
   args:
-    chdir: "{{ openshift_cert_dir_parent }}"
-    creates: "{{ openshift_cert_dir_abs }}/node-{{ item.openshift.common.hostname }}/.kubeconfig"
+    chdir: "{{ openshift_cert_parent_dir }}"
+    creates: "{{ openshift_cert_dir }}/node-{{ item.openshift.common.hostname }}"
   with_items: openshift_nodes
-  register: kubeconfig_result
 
 - name: Register unregistered nodes
   kubernetes_register_node:
-    client_user: openshift-client
+    kubectl_cmd: ['openshift', 'kube']
     name: "{{ item.openshift.common.hostname }}"
     api_version: "{{ openshift_kube_api_version }}"
     cpu: "{{ item.openshift.node.resources_cpu | default(None) }}"
@@ -61,7 +45,5 @@
     external_id: "{{ item.openshift.node.external_id }}"
     # TODO: support customizing other attributes such as: client_config,
     # client_cluster, client_context, client_user
-    # TODO: update for v1beta3 changes after rebase: hostnames, external_ips,
-    # internal_ips, external_id
   with_items: openshift_nodes
   register: register_result

roles/openshift_register_nodes/vars/main.yml

@@ -0,0 +1,7 @@
+---
+openshift_cert_parent_dir: /var/lib/openshift
+openshift_cert_relative_dir: openshift.local.certificates
+openshift_cert_dir: "{{ openshift_cert_parent_dir }}/{{ openshift_cert_relative_dir }}"
+openshift_master_ca_dir: "{{ openshift_cert_dir }}/ca"
+openshift_master_ca_cert: "{{ openshift_master_ca_dir }}/cert.crt"
+openshift_master_ca_key: "{{ openshift_master_ca_dir }}/key.key"

roles/openshift_repos/README.md

@@ -14,7 +14,7 @@ Role Variables
 
 | Name                          | Default value |                                              |
 |-------------------------------|---------------|----------------------------------------------|
-| openshift_deployment_type     | online        | Possible values enterprise, origin, online   |
+| openshift_deployment_type     | None          | Possible values enterprise, origin, online   |
 | openshift_additional_repos    | {}            | TODO                                         |
 
 Dependencies

roles/openshift_repos/defaults/main.yaml

@@ -1,7 +1,2 @@
 ---
-# TODO: once we are able to configure/deploy origin using the openshift roles,
-# then we should default to origin
-
-# TODO: push the defaulting of these values to the openshift_facts module
-openshift_deployment_type: online
 openshift_additional_repos: {}

roles/openshift_repos/files/online/epel7-kubernetes.repo

@@ -1,6 +0,0 @@
-[maxamillion-epel7-kubernetes]
-name=Copr repo for epel7-kubernetes owned by maxamillion
-baseurl=http://copr-be.cloud.fedoraproject.org/results/maxamillion/epel7-kubernetes/epel-7-$basearch/
-skip_if_unavailable=True
-gpgcheck=0
-enabled=1