ANSUPDATE-129 Patch SSH config for timeouts

Adding execution of a script to add configuration to the local system
SSH client configuration file that engages a heartbeat mechanism that
allows SSH to know that the server is still alive instead of relying
on inactivity to declare the SSH connection as failed.  These SSH
timeout issues present themselves as Broken Pipe errors.

Long running scripts, such as the database creation/upgrade scripts
can cause the SSH connection to go idle for a period of up to several
minutes with no output.

Change-Id: I1dc1b4f51cf20c1b450fb5879c0a46774384ac92

playbooks/files/patch-ssh-client-config.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+# Copyright (c) 2014 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -eux
+set -o pipefail
+
+# This turns on an internal heartbeat mechanism in the ssh client
+# to prevent the client from believing inactivity is a connection
+# failure.
+if ! grep -q '^\s*ServerAliveInterval' /etc/ssh/ssh_config; then
+    echo "    ServerAliveInterval 30" >>/etc/ssh/ssh_config
+fi
+
+# This causes the connection to wait until the defined number of
+# heartbeats are missed before terminating the connection.
+if ! grep -q '^\s*ServerAliveCountMax' /etc/ssh/ssh_config; then
+    echo "    ServerAliveCountMax 6" >>/etc/ssh/ssh_config
+fi
+

playbooks/update_cloud.yml

@@ -14,6 +14,12 @@
 # limitations under the License.
 ---
 - include: step_ping.yml
+- hosts: localhost
+  name: "Setup local environment for upgrade processes to run"
+  gather_facts: no
+  max_fail_percentage: 0
+  tasks:
+    - include: update_local_ssh_config.yml
 - hosts: undercloud
   name: Disable Undercloud
   sudo: yes

playbooks/update_local_ssh_config.yml

@@ -0,0 +1,18 @@
+# Copyright (c) 2014 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+- name: "Patching the ssh client configuration to avoid timeout issues."
+  sudo: yes
+  script: files/patch-ssh-client-config.sh