Merge pull request #4973 from wozniakjan/logging_eventrouter

Automatic merge from submit-queue

logging: introducing event router

Initial concept integrating kubernetes events to EFK stack using [eventrouter](https://github.com/heptio/eventrouter)
- **eventrouter** is deployed to logging project, has a service account and its own role to read events
- **eventrouter** watches kubernetes events, marshalls them to JSON and outputs to its STDOUT
- **fluentd** picks them up and inserts to elastic search logging project index

Kubernetes events could be easily stored into different index. Among other ways, it could be achieved either by a fluentd filter plugin or custom eventrouter sink.

Pending actions:
- [x] - get our downstream and upstream images for eventrouter, don't use heptio's

cc: @josefkarasek , @jcantrill , @richm

roles/openshift_logging/README.md

@@ -12,13 +12,13 @@ generation for Elasticsearch (it uses JKS) as well as openssl to sign certificat
 As part of the installation, it is recommended that you add the Fluentd node selector label
 to the list of persisted [node labels](https://docs.openshift.org/latest/install_config/install/advanced_install.html#configuring-node-host-labels).
 
-###Required vars:
+### Required vars:
 
 - `openshift_logging_install_logging`: When `True` the `openshift_logging` role will install Aggregated Logging.
 
 When `openshift_logging_install_logging` is set to `False` the `openshift_logging` role will uninstall Aggregated Logging.
 
-###Optional vars:
+### Optional vars:
 - `openshift_logging_purge_logging`: When `openshift_logging_install_logging` is set to 'False' to trigger uninstalation and `openshift_logging_purge_logging` is set to 'True', it will completely and irreversibly remove all logging persistent data including PVC. Defaults to 'False'.
 - `openshift_logging_image_prefix`: The prefix for the logging images to use. Defaults to 'docker.io/openshift/origin-'.
 - `openshift_logging_curator_image_prefix`: Setting the image prefix for Curator image. Defaults to `openshift_logging_image_prefix`.
@@ -90,6 +90,12 @@ When `openshift_logging_install_logging` is set to `False` the `openshift_loggin
 - `openshift_logging_es_number_of_shards`: The number of primary shards for every new index created in ES. Defaults to '1'.
 - `openshift_logging_es_number_of_replicas`: The number of replica shards per primary shard for every new index. Defaults to '0'.
 
+- `openshift_logging_install_eventrouter`: Coupled with `openshift_logging_install_logging`. When both are 'True', eventrouter will be installed. When both are 'False', eventrouter will be uninstalled.
+Other combinations will keep the eventrouter untouched.
+
+Detailed eventrouter configuration can be found in
+- `roles/openshift_logging_eventrouter/README.md`
+
 When `openshift_logging_use_ops` is `True`, there are some additional vars. These work the
 same as above for their non-ops counterparts, but apply to the OPS cluster instance:
 - `openshift_logging_es_ops_host`: logging-es-ops

roles/openshift_logging/tasks/delete_logging.yaml

@@ -105,3 +105,9 @@
     - logging-elasticsearch
     - logging-fluentd
     - logging-mux
+
+## EventRouter
+- include_role:
+    name: openshift_logging_eventrouter
+  when:
+    not openshift_logging_install_eventrouter | default(false) | bool

roles/openshift_logging/tasks/install_logging.yaml

@@ -270,4 +270,12 @@
     openshift_logging_fluentd_master_url: "{{ openshift_logging_master_url }}"
     openshift_logging_fluentd_namespace: "{{ openshift_logging_namespace }}"
 
+
+## EventRouter
+- include_role:
+    name: openshift_logging_eventrouter
+  when:
+    openshift_logging_install_eventrouter | default(false) | bool
+
+
 - include: update_master_config.yaml

roles/openshift_logging_eventrouter/README.md

@@ -0,0 +1,20 @@
+Event router
+------------
+
+A pod forwarding kubernetes events to EFK aggregated logging stack.
+
+- **eventrouter** is deployed to logging project, has a service account and its own role to read events
+- **eventrouter** watches kubernetes events, marshalls them to JSON and outputs to its sink, currently only various formatting to STDOUT
+- **fluentd** picks them up and inserts to elasticsearch *.operations* index
+
+- `openshift_logging_install_eventrouter`: When 'True', eventrouter will be installed. When 'False', eventrouter will be uninstalled.
+
+Configuration variables:
+
+- `openshift_logging_eventrouter_image_prefix`: The prefix for the eventrouter logging image. Defaults to `openshift_logging_image_prefix`.
+- `openshift_logging_eventrouter_image_version`: The image version for the logging eventrouter. Defaults to 'latest'.
+- `openshift_logging_eventrouter_sink`: Select a sink for eventrouter, supported 'stdout' and 'glog'. Defaults to 'stdout'.
+- `openshift_logging_eventrouter_nodeselector`: A map of labels (e.g. {"node":"infra","region":"west"} to select the nodes where the pod will land.
+- `openshift_logging_eventrouter_cpu_limit`: The amount of CPU to allocate to eventrouter. Defaults to '100m'.
+- `openshift_logging_eventrouter_memory_limit`: The memory limit for eventrouter pods. Defaults to '128Mi'.
+- `openshift_logging_eventrouter_namespace`: The namespace where eventrouter is deployed. Defaults to 'default'.

roles/openshift_logging_eventrouter/defaults/main.yaml

@@ -0,0 +1,9 @@
+---
+openshift_logging_eventrouter_image_prefix: "{{ openshift_logging_image_prefix | default(__openshift_logging_image_prefix) }}"
+openshift_logging_eventrouter_image_version: "{{ openshift_logging_image_version | default('latest') }}"
+openshift_logging_eventrouter_replicas: 1
+openshift_logging_eventrouter_sink: stdout
+openshift_logging_eventrouter_nodeselector: ""
+openshift_logging_eventrouter_cpu_limit: 100m
+openshift_logging_eventrouter_memory_limit: 128Mi
+openshift_logging_eventrouter_namespace: default

roles/openshift_logging_eventrouter/files/eventrouter-template.yaml

@@ -0,0 +1,103 @@
+# this openshift template should match (except nodeSelector) jinja2 template in
+# ../templates/eventrouter-template.j2
+kind: Template
+apiVersion: v1
+metadata:
+  name: eventrouter-template
+  annotations:
+    description: "A pod forwarding kubernetes events to EFK aggregated logging stack."
+    tags: "events,EFK,logging"
+objects:
+  - kind: ServiceAccount
+    apiVersion: v1
+    metadata:
+      name: aggregated-logging-eventrouter
+  - kind: ClusterRole
+    apiVersion: v1
+    metadata:
+      name: event-reader
+    rules:
+    - apiGroups: [""]
+      resources: ["events"]
+      verbs: ["get", "watch", "list"]
+  - kind: ConfigMap
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+    data:
+      config.json: |- 
+        {
+          "sink": "${SINK}"
+        }
+  - kind: DeploymentConfig
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+      labels:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+    spec:
+      selector:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+      replicas: ${REPLICAS}
+      template:
+        metadata:
+          labels:
+            component: eventrouter
+            logging-infra: eventrouter
+            provider: openshift
+          name: logging-eventrouter
+        spec:
+          serviceAccount: aggregated-logging-eventrouter
+          serviceAccountName: aggregated-logging-eventrouter
+          containers:
+            - name: kube-eventrouter
+              image: ${IMAGE}
+              imagePullPolicy: Always
+              resources:
+                limits:
+                  memory: ${MEMORY} 
+                  cpu: ${CPU}
+                requires:
+                  memory: ${MEMORY}
+              volumeMounts:
+              - name: config-volume
+                mountPath: /etc/eventrouter
+          volumes:
+            - name: config-volume
+              configMap:
+                name: logging-eventrouter
+  - kind: ClusterRoleBinding
+    apiVersion: v1
+    metadata:
+      name: event-reader-binding
+    subjects:
+    - kind: ServiceAccount
+      name: aggregated-logging-eventrouter
+      namespace: ${NAMESPACE}
+    roleRef:
+      kind: ClusterRole
+      name: event-reader
+
+parameters:
+  - name: SINK
+    displayName: Sink
+    value: stdout
+  - name: REPLICAS
+    displayName: Replicas
+    value: "1"
+  - name: IMAGE
+    displayName: Image
+    value: "docker.io/openshift/origin-logging-eventrouter:latest"
+  - name: MEMORY
+    displayName: Memory
+    value: "128Mi"
+  - name: CPU
+    displayName: CPU
+    value: "100m"
+  - name: NAMESPACE
+    displayName: Namespace
+    value: default

roles/openshift_logging_eventrouter/tasks/delete_eventrouter.yaml

@@ -0,0 +1,40 @@
+---
+# delete eventrouter
+- name: Delete EventRouter service account
+  oc_serviceaccount:
+    state: absent
+    name: "aggregated-logging-eventrouter"
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+
+- name: Delete event-reader cluster role
+  oc_clusterrole:
+    state: absent
+    name: event-reader
+
+- name: Unset privileged permissions for EventRouter
+  oc_adm_policy_user:
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    resource_kind: cluster-role
+    resource_name: event-reader
+    state: absent
+    user: "system:serviceaccount:{{ openshift_logging_eventrouter_namespace }}:aggregated-logging-eventrouter"
+
+- name: Delete EventRouter configmap
+  oc_configmap:
+    state: absent
+    name: logging-eventrouter
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+
+- name: Delete EventRouter DC
+  oc_obj:
+    state: absent
+    name: logging-eventrouter
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: dc
+
+- name: Delete EventRouter Template
+  oc_obj:
+    state: absent
+    name: eventrouter-template
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: template

roles/openshift_logging_eventrouter/tasks/install_eventrouter.yaml

@@ -0,0 +1,59 @@
+---
+# initial checks
+- assert:
+    msg: Invalid sink type "{{openshift_logging_eventrouter_sink}}", only one of "{{__eventrouter_sinks}}" allowed
+    that: openshift_logging_eventrouter_sink in __eventrouter_sinks
+
+# allow passing in a tempdir
+- name: Create temp directory for doing work in
+  command: mktemp -d /tmp/openshift-logging-ansible-XXXXXX
+  register: mktemp
+  changed_when: False
+
+- set_fact:
+    tempdir: "{{ mktemp.stdout }}"
+
+- name: Create templates subdirectory
+  file:
+    state: directory
+    path: "{{ tempdir }}/templates"
+    mode: 0755
+  changed_when: False
+
+# create EventRouter deployment config
+- name: Generate EventRouter template
+  template:
+    src: eventrouter-template.j2
+    dest: "{{ tempdir }}/templates/eventrouter-template.yaml"
+  vars:
+    node_selector: "{{ openshift_logging_eventrouter_nodeselector | default({}) }}"
+
+- name: Create EventRouter template
+  oc_obj:
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: template
+    name: eventrouter-template
+    state: present
+    files:
+    - "{{ tempdir }}/templates/eventrouter-template.yaml"
+
+- name: Process EventRouter template
+  oc_process:
+    state: present
+    template_name: eventrouter-template
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    params:
+      IMAGE: "{{openshift_logging_eventrouter_image_prefix}}logging-eventrouter:{{openshift_logging_eventrouter_image_version}}"
+      REPLICAS: "{{ openshift_logging_eventrouter_replicas }}"
+      CPU: "{{ openshift_logging_eventrouter_cpu_limit }}"
+      MEMORY: "{{ openshift_logging_eventrouter_memory_limit }}"
+      NAMESPACE: "{{ openshift_logging_eventrouter_namespace }}"
+      SINK: "{{ openshift_logging_eventrouter_sink }}"
+
+## Placeholder for migration when necessary ##
+
+- name: Delete temp directory
+  file:
+    name: "{{ tempdir }}"
+    state: absent
+  changed_when: False

roles/openshift_logging_eventrouter/tasks/main.yaml

@@ -0,0 +1,6 @@
+---
+- include: "{{ role_path }}/tasks/install_eventrouter.yaml"
+  when: openshift_logging_install_eventrouter | default(false) | bool
+
+- include: "{{ role_path }}/tasks/delete_eventrouter.yaml"
+  when: not openshift_logging_install_eventrouter | default(false) | bool

roles/openshift_logging_eventrouter/templates/eventrouter-template.j2

@@ -0,0 +1,109 @@
+# this jinja2 template should always match (except nodeSelector) openshift template in
+# ../files/eventrouter-template.yaml
+kind: Template
+apiVersion: v1
+metadata:
+  name: eventrouter-template
+  annotations:
+    description: "A pod forwarding kubernetes events to EFK aggregated logging stack."
+    tags: "events,EFK,logging"
+objects:
+  - kind: ServiceAccount
+    apiVersion: v1
+    metadata:
+      name: aggregated-logging-eventrouter
+  - kind: ClusterRole
+    apiVersion: v1
+    metadata:
+      name: event-reader
+    rules:
+    - apiGroups: [""]
+      resources: ["events"]
+      verbs: ["get", "watch", "list"]
+  - kind: ConfigMap
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+    data:
+      config.json: |- 
+        {
+          "sink": "${SINK}"
+        }
+  - kind: DeploymentConfig
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+      labels:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+    spec:
+      selector:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+      replicas: ${REPLICAS}
+      template:
+        metadata:
+          labels:
+            component: eventrouter
+            logging-infra: eventrouter
+            provider: openshift
+          name: logging-eventrouter
+        spec:
+          serviceAccount: aggregated-logging-eventrouter
+          serviceAccountName: aggregated-logging-eventrouter
+{% if node_selector is iterable and node_selector | length > 0 %}
+      nodeSelector:
+{% for key, value in node_selector.iteritems() %}
+        {{ key }}: "{{ value }}"
+{% endfor %}
+{% endif %}
+          containers:
+            - name: kube-eventrouter
+              image: ${IMAGE}
+              imagePullPolicy: Always
+              resources:
+                limits:
+                  memory: ${MEMORY} 
+                  cpu: ${CPU}
+                requires:
+                  memory: ${MEMORY}
+              volumeMounts:
+              - name: config-volume
+                mountPath: /etc/eventrouter
+          volumes:
+            - name: config-volume
+              configMap:
+                name: logging-eventrouter
+  - kind: ClusterRoleBinding
+    apiVersion: v1
+    metadata:
+      name: event-reader-binding
+    subjects:
+    - kind: ServiceAccount
+      name: aggregated-logging-eventrouter
+      namespace: ${NAMESPACE}
+    roleRef:
+      kind: ClusterRole
+      name: event-reader
+
+parameters:
+  - name: SINK
+    displayName: Sink
+    value: stdout
+  - name: REPLICAS
+    displayName: Replicas
+    value: "1"
+  - name: IMAGE
+    displayName: Image
+    value: "docker.io/openshift/origin-logging-eventrouter:latest"
+  - name: MEMORY
+    displayName: Memory
+    value: "128Mi"
+  - name: CPU
+    displayName: CPU
+    value: "100m"
+  - name: NAMESPACE
+    displayName: Namespace
+    value: default

roles/openshift_logging_eventrouter/vars/main.yaml

@@ -0,0 +1,2 @@
+---
+__eventrouter_sinks: ["glog", "stdout"]

roles/openshift_logging_fluentd/templates/fluentd.j2

@@ -118,6 +118,10 @@ spec:
         - name: "MUX_CLIENT_MODE"
           value: "{{ openshift_logging_mux_client_mode }}"
 {% endif %}
+{% if openshift_logging_install_eventrouter is defined and openshift_logging_install_eventrouter %}
+        - name: "TRANSFORM_EVENTS"
+          value: "true"
+{% endif %}
       volumes:
       - name: runlogjournal
         hostPath: