Browse Source

sdn: fix OOM issues with ovs-vswitchd on many-core machines

By default ovs-vswitchd creates #cores+1 pthreads, which each get
the RHEL/Fedora default 8MB of stack.  But for whatever reason,
ovs-vswitchd immediately dirties this region resulting in all 8MB
counted to RSS, leading to excessive memory consumption.

Work around that in two ways:

1) increase the OVS container memory limits
2) limit the number of threads ovs-vswitchd uses on machines
with more than 12 cores

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1571379
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1572797
Dan Williams 7 years ago
parent
commit
6d9ad9d1ac
1 changed files with 11 additions and 2 deletions
  1. 11 2
      roles/openshift_sdn/files/sdn-ovs.yaml

+ 11 - 2
roles/openshift_sdn/files/sdn-ovs.yaml

@@ -63,6 +63,15 @@ spec:
           }
           trap quit SIGTERM
           /usr/share/openvswitch/scripts/ovs-ctl start --system-id=random
+
+          # Restrict the number of pthreads ovs-vswitchd creates to reduce the
+          # amount of RSS it uses on hosts with many cores
+          # https://bugzilla.redhat.com/show_bug.cgi?id=1571379
+          # https://bugzilla.redhat.com/show_bug.cgi?id=1572797
+          if [[ `nproc` -gt 12 ]]; then
+              ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=4
+              ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=10
+          fi
           while true; do sleep 5; done
         securityContext:
           runAsUser: 0
@@ -83,10 +92,10 @@ spec:
         resources:
           requests:
             cpu: 100m
-            memory: 200Mi
+            memory: 300Mi
           limits:
             cpu: 200m
-            memory: 300Mi
+            memory: 400Mi
 
       volumes:
       - name: host-modules