Merge pull request #9862 from tomassedovic/openstack-no-floating-ip

OpenStack deployments without Floating IPs

playbooks/openstack/configuration.md

@@ -16,6 +16,7 @@ Environment variables may also be used.
 * [OpenStack With SSL Configuration](#openstack-with-ssl-configuration)
 * [Stack Name Configuration](#stack-name-configuration)
 * [DNS Configuration](#dns-configuration)
+* [Floating IP Address Configuration](#floating-ip-address-configuration)
 * [All-in-one Deployment Configuration](#all-in-one-deployment-configuration)
 * [Building Node Images](#building-node-images)
 * [Kuryr Networking Configuration](#kuryr-networking-configuration)
@@ -39,6 +40,10 @@ In `inventory/group_vars/all.yml`:
   * `openshift_openstack_num_masters` Number of master nodes to create.
   * `openshift_openstack_num_infra` Number of infra nodes to create.
   * `openshift_openstack_num_nodes` Number of app nodes to create.
+* Role Node Floating IP Allocation
+  * `openshift_openstack_master_floating_ip` Assign floating IP to master nodes. Defaults to `True`.
+  * `openshift_openstack_infra_floating_ip` Assign floating IP to infra nodes. Defaults to `True`.
+  * `openshift_openstack_compute_floating_ip` Assign floating IP to app nodes. Defaults to `True`.
 * Role Images
   * `openshift_openstack_default_image_name` OpenStack image used by all VMs, unless a particular role image name is specified.
   * `openshift_openstack_master_image_name`
@@ -410,6 +415,86 @@ These must point to the publicly-accessible IP addresses of your
 master and infra nodes or preferably to the load balancers.
 
 
+## Floating IP Address Configuration
+
+Every OpenShift node as well as the API and Router load balancer will receive a
+floating IP address by default. This is to make the deployment and debugging
+experience easier.
+
+You may want to change that behaviour, for example to prevent any possibility
+of external access to the nodes (defense in depth) or if your floating IP pool
+is not large enough.
+
+### Overview
+
+It possible to configure the playbooks to not asssign floating IP addresses.
+However, the Ansible playbooks will then not be able to SSH and install
+OpenShift.
+
+The nodes will only be accessible from the subnet they are assigned to.
+
+To solve this, we need to create the network the nodes will be placed in
+beforehnd, then boot up a bastion host in the same network and run the
+playbooks from there.
+
+### Node Network
+
+We will have to create a Neutron Network, Subnet and a Router for external
+connectivity. Take note of any DNS servers you would normally put under
+`openshift_openstack_dns_nameservers` -- they must be added to the subnet.
+
+In this example, we will call the network and its subnet `openshift` and configure
+a DNS server with IP address `10.20.30.40`. The external network will be called `public`.
+
+```
+$ openstack network create openshift
+$ openstack subnet create --subnet-range 192.168.0.0/24 --dns-nameserver 10.20.30.40 --network openshift openshift
+$ openstack router create openshift-router
+$ openstack router set --external-gateway public openshift-router
+$ openstack router add subnet openshift-router openshift
+```
+
+### Bastion host
+
+To provide SSH connectivity (that Ansible requires) to the OpenShift nodes
+without using floating IP addresses, the playbooks must be running on a server
+inside the same subnet.
+
+This will create such server and place it into the subnet created above.
+
+We will use an image called `CentOS-7-x86_64-GenericCloud`, and assume that the
+created floating IP address will be `172.24.4.10`.
+
+```
+$ openstack server create --wait --image CentOS-7-x86_64-GenericCloud --flavor m1.medium --key-name openshift --network openshift bastion
+$ openstack floating ip create public
+$ openstack server add floating ip bastion 172.24.4.10
+$ ping 172.24.4.10
+$ ssh centos@172.24.4.10
+```
+
+### openshift-ansible Configuration
+
+In addition to the rest of openshift-ansible configuration, we will need to
+specify the node netwok, subnet and that we do not want any floating IP
+addresses.
+
+You must do this from inside the "bastion" host created in the previous step.
+
+Put the following to `inventory/group_vars/all.yml`:
+
+```yaml
+openshift_openstack_node_network_name: openshift
+openshift_openstack_node_subnet_name: openshift
+openshift_openstack_master_floating_ip: false
+openshift_openstack_infra_floating_ip: false
+openshift_openstack_compute_floating_ip: false
+openshift_openstack_load_balancer_floating_ip: false
+```
+
+And then run the `playbooks/openstack/openshift-cluster/*.yml` as usual.
+
+
 ## All-in-one Deployment Configuration
 
 If you want to deploy OpenShift on a single node (e.g. for quick evaluation),

playbooks/openstack/sample-inventory/group_vars/all.yml

@@ -106,6 +106,15 @@ openshift_openstack_num_infra: 1
 openshift_openstack_num_cns: 0
 openshift_openstack_num_nodes: 2
 
+# # Public IP Allocation
+# # - manage which node roles are allocated public IP addresses
+# # - by default, all roles are given Public IP addresses
+#openshift_openstack_master_floating_ip: true
+#openshift_openstack_infra_floating_ip: true
+#openshift_openstack_etcd_floating_ip: true
+#openshift_openstack_load_balancer_floating_ip: true
+#openshift_openstack_compute_floating_ip: true
+
 # # Used Flavors
 # # - set specific flavors for roles by uncommenting corresponding lines
 # # - note: do note remove openshift_openstack_default_flavor definition

roles/openshift_openstack/defaults/main.yml

@@ -38,6 +38,11 @@ openshift_openstack_nsupdate_zone: "{{ openshift_openstack_full_dns_domain }}"
 
 
 # heat vars
+openshift_openstack_master_floating_ip: true
+openshift_openstack_infra_floating_ip: true
+openshift_openstack_compute_floating_ip: true
+openshift_openstack_etcd_floating_ip: true
+openshift_openstack_load_balancer_floating_ip: true
 openshift_openstack_heat_template_version: pike
 openshift_openstack_clusterid: openshift
 openshift_openstack_stack_name: "openshift-cluster"
@@ -71,6 +76,8 @@ openshift_openstack_lb_image: "{{ openshift_openstack_default_image_name }}"
 openshift_openstack_etcd_image: "{{ openshift_openstack_default_image_name }}"
 openshift_openstack_provider_network_name: null
 openshift_openstack_external_network_name: null
+openshift_openstack_node_network_name: null
+openshift_openstack_node_subnet_name: null
 openshift_openstack_private_network: >-
   {% if openshift_openstack_provider_network_name | default(None) -%}
   {{ openshift_openstack_provider_network_name }}

roles/openshift_openstack/templates/heat_stack.yaml.j2

@@ -15,9 +15,11 @@ outputs:
     description: IPs of the etcds
     value: { get_attr: [ etcd, private_ip ] }
 
+{% if openshift_openstack_etcd_floating_ip | default(True) | bool %}
   etcd_floating_ips:
     description: Floating IPs of the etcds
     value: { get_attr: [ etcd, floating_ip ] }
+{% endif %}
 
   master_names:
     description: Name of the masters
@@ -27,9 +29,11 @@ outputs:
     description: IPs of the masters
     value: { get_attr: [ masters, private_ip ] }
 
+{% if openshift_openstack_master_floating_ip | default(True) | bool %}
   master_floating_ips:
     description: Floating IPs of the masters
     value: { get_attr: [ masters, floating_ip ] }
+{% endif %}
 
   node_names:
     description: Name of the nodes
@@ -39,9 +43,11 @@ outputs:
     description: IPs of the nodes
     value: { get_attr: [ compute_nodes, private_ip ] }
 
+{% if openshift_openstack_compute_floating_ip | default(True) | bool %}
   node_floating_ips:
     description: Floating IPs of the nodes
     value: { get_attr: [ compute_nodes, floating_ip ] }
+{% endif %}
 
   infra_names:
     description: Name of the nodes
@@ -51,16 +57,19 @@ outputs:
     description: IPs of the nodes
     value: { get_attr: [ infra_nodes, private_ip ] }
 
+{% if openshift_openstack_infra_floating_ip | default(True) | bool %}
   infra_floating_ips:
     description: Floating IPs of the nodes
     value: { get_attr: [ infra_nodes, floating_ip ] }
 {% endif %}
+{% endif %}
 
   public_api_ip:
     description: IP address for the API/UI endpoint
-{% if openshift_openstack_use_lbaas_load_balancer %}
-    # TODO(shadower): Handle setups without floating IPs
+{% if openshift_openstack_use_lbaas_load_balancer and openshift_openstack_load_balancer_floating_ip %}
     value: { get_attr: [api_lb_floating_ip, floating_ip_address] }
+{% elif openshift_openstack_use_lbaas_load_balancer and not openshift_openstack_load_balancer_floating_ip %}
+    value: { get_attr: [api_lb, vip_address] }
 {% elif openshift_openstack_use_vm_load_balancer %}
     value: { get_attr: [loadbalancer, resource.0, floating_ip] }
 {% else %}
@@ -69,8 +78,10 @@ outputs:
 
   public_router_ip:
     description: IP address of the apps/router endpoint
-{% if openshift_openstack_use_lbaas_load_balancer %}
+{% if openshift_openstack_use_lbaas_load_balancer and openshift_openstack_load_balancer_floating_ip %}
     value: { get_attr: [router_lb_floating_ip, floating_ip_address] }
+{% elif openshift_openstack_use_lbaas_load_balancer and not openshift_openstack_load_balancer_floating_ip %}
+    value: { get_attr: [router_lb, vip_address] }
 {% else %}
     # NOTE(shadower): The VM-based loadbalancer only supports master nodes
     value: { get_attr: [infra_nodes, resource.0, floating_ip] }
@@ -156,6 +167,8 @@ resources:
 {% if openshift_use_kuryr|default(false)|bool %}
       vip_address: {{ openshift_openstack_kuryr_service_subnet_cidr | ipaddr('1') | ipaddr('address') }}
       vip_subnet: { get_resource: service_subnet }
+{% elif openshift_openstack_node_subnet_name %}
+      vip_subnet: {{ openshift_openstack_node_subnet_name }}
 {% else %}
       vip_subnet: { get_resource: subnet }
 {% endif %}
@@ -322,6 +335,7 @@ resources:
 
 {% endif %}
 
+{% if not openshift_openstack_node_network_name %}
   net:
     type: OS::Neutron::Net
     properties:
@@ -330,7 +344,9 @@ resources:
           template: openshift-ansible-cluster_id-net
           params:
             cluster_id: {{ openshift_openstack_full_dns_domain }}
+{% endif %}
 
+{% if not openshift_openstack_node_subnet_name %}
   subnet:
     type: OS::Neutron::Subnet
     properties:
@@ -348,6 +364,7 @@ resources:
 {% for nameserver in openshift_openstack_dns_nameservers %}
         - {{ nameserver }}
 {% endfor %}
+{% endif %}
 
 {% if openshift_use_flannel|default(False)|bool %}
   data_net:
@@ -376,11 +393,13 @@ resources:
       external_gateway_info:
         network: {{ openshift_openstack_external_network_name }}
 
+{% if not openshift_openstack_node_subnet_name %}
   interface:
     type: OS::Neutron::RouterInterface
     properties:
       router_id: { get_resource: router }
       subnet_id: { get_resource: subnet }
+{% endif %}
 
 {% if openshift_use_kuryr|default(false)|bool %}
   pod_subnet_interface:
@@ -607,8 +626,17 @@ resources:
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:         {{ openshift_openstack_provider_network_name }}
 {% else %}
+{% if openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
+          net_name:    {{ openshift_openstack_node_network_name }}
+{% else %}
           net:         { get_resource: net }
+{% endif %}
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -631,11 +659,11 @@ resources:
               - no_floating
               - ''
               - {{ openshift_openstack_external_network_name }}
-{% if openshift_openstack_provider_network_name %}
+{% if openshift_openstack_provider_network_name or not openshift_openstack_etcd_floating_ip | default(False) | bool %}
           attach_float_net: false
 {% endif %}
           volume_size: {{ openshift_openstack_etcd_volume_size }}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
@@ -683,9 +711,15 @@ resources:
 {% if openshift_openstack_provider_network_name %}
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:         {{ openshift_openstack_provider_network_name }}
+{% elif openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
 {% else %}
           net:         { get_resource: net }
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -708,11 +742,11 @@ resources:
               - no_floating
               - ''
               - {{ openshift_openstack_external_network_name }}
-{% if openshift_openstack_provider_network_name %}
+{% if openshift_openstack_provider_network_name or not openshift_openstack_load_balancer_floating_ip | default(True) | bool %}
           attach_float_net: false
 {% endif %}
           volume_size: {{ openshift_openstack_lb_volume_size }}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
@@ -751,8 +785,17 @@ resources:
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:         {{ openshift_openstack_provider_network_name }}
 {% else %}
+{% if openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
+          net_name:    {{ openshift_openstack_node_network_name }}
+{% else %}
           net:         { get_resource: net }
+{% endif %}
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -791,7 +834,7 @@ resources:
               - no_floating
               - ''
               - {{ openshift_openstack_external_network_name }}
-{% if openshift_openstack_provider_network_name %}
+{% if openshift_openstack_provider_network_name or not openshift_openstack_master_floating_ip | default(False) | bool %}
           attach_float_net: false
 {% endif %}
           volume_size: {{ openshift_openstack_master_volume_size }}
@@ -799,7 +842,7 @@ resources:
           scheduler_hints:
             group: { get_resource: master_server_group }
 {% endif %}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
@@ -837,8 +880,17 @@ resources:
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:         {{ openshift_openstack_provider_network_name }}
 {% else %}
+{% if openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
+          net_name:    {{ openshift_openstack_node_network_name }}
+{% else %}
           net:         { get_resource: net }
+{% endif %}
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -866,11 +918,11 @@ resources:
               - no_floating
               - ''
               - {{ openshift_openstack_external_network_name }}
-{% if openshift_openstack_provider_network_name %}
+{% if openshift_openstack_provider_network_name or not openshift_openstack_compute_floating_ip | default(False) | bool %}
           attach_float_net: false
 {% endif %}
           volume_size: {{ openshift_openstack_node_volume_size }}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
@@ -910,8 +962,17 @@ resources:
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:         {{ openshift_openstack_provider_network_name }}
 {% else %}
+{% if openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
+          net_name:    {{ openshift_openstack_node_network_name }}
+{% else %}
           net:         { get_resource: net }
+{% endif %}
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -945,7 +1006,7 @@ resources:
               - no_floating
               - ''
               - {{ openshift_openstack_external_network_name }}
-{% if openshift_openstack_provider_network_name %}
+{% if openshift_openstack_provider_network_name or not openshift_openstack_infra_floating_ip | default(True) | bool %}
           attach_float_net: false
 {% endif %}
           volume_size: {{ openshift_openstack_infra_volume_size }}
@@ -953,7 +1014,7 @@ resources:
           scheduler_hints:
             group: { get_resource: infra_server_group }
 {% endif %}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
@@ -988,8 +1049,17 @@ resources:
           net:         {{ openshift_openstack_provider_network_name }}
           net_name:    {{ openshift_openstack_provider_network_name }}
 {% else %}
+{% if openshift_openstack_node_network_name %}
+          net:         {{ openshift_openstack_node_network_name }}
+          net_name:    {{ openshift_openstack_node_network_name }}
+{% else %}
           net:         { get_resource: net }
+{% endif %}
+{% if openshift_openstack_node_subnet_name %}
+          subnet:      {{ openshift_openstack_node_subnet_name }}
+{% else %}
           subnet:      { get_resource: subnet }
+{% endif %}
 {% if openshift_use_kuryr|default(false)|bool %}
           pod_net:     { get_resource: pod_net }
           pod_subnet:  { get_resource: pod_subnet }
@@ -1021,13 +1091,14 @@ resources:
           floating_network: {{ openshift_openstack_external_network_name }}
 {% endif %}
           volume_size: {{ openshift_openstack_cns_volume_size }}
-{% if not openshift_openstack_provider_network_name %}
+{% if not openshift_openstack_provider_network_name and not openshift_openstack_node_subnet_name  %}
     depends_on:
       - interface
 {% endif %}
 
 
 {% if openshift_openstack_use_lbaas_load_balancer %}
+{% if openshift_openstack_load_balancer_floating_ip | default(True) | bool %}
   api_lb_floating_ip:
     condition: { not: no_floating }
     depends_on:
@@ -1038,13 +1109,18 @@ resources:
     properties:
       floating_network: {{ openshift_openstack_external_network_name }}
       port_id: { get_attr: [api_lb, vip_port_id] }
-
+{% endif %}
 
   router_lb:
     type: OS::{{ openshift_openstack_lbaasv2_provider }}::LoadBalancer
     properties:
+{% if openshift_openstack_node_subnet_name %}
+      vip_subnet: {{ openshift_openstack_node_subnet_name }}
+{% else %}
       vip_subnet: { get_resource: subnet }
+{% endif %}
 
+{% if openshift_openstack_load_balancer_floating_ip | default(True) | bool %}
   router_lb_floating_ip:
     condition: { not: no_floating }
     depends_on:
@@ -1057,6 +1133,7 @@ resources:
     properties:
       floating_network: {{ openshift_openstack_external_network_name }}
       port_id: { get_attr: [router_lb, vip_port_id] }
+{% endif %}
 
   router_lb_listener_http:
     type: OS::{{ openshift_openstack_lbaasv2_provider }}::Listener