WinRM: fatal errors when executing a long-running command

[virtuald]

ISSUE TYPE

• Bug Report

COMPONENT NAME

windows/winrm

ANSIBLE VERSION

ansible --version
ansible 2.3.2.0
  config file = 
  configured module search path = [u'/path/to/my/ansible/library']
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

CONFIGURATION

• I'm using various custom filters/actions/modules, but none of them are related to the underlying connection

OS / ENVIRONMENT

• Control node: Ubuntu 16.04
• Hosts: Windows Server 2012 R2

SUMMARY

I've been able to consistently get fatal errors when installing .NET 4.5.1 using Ansible 2.3.1.0 on Windows 2012 that doesn't have .NET 4.5.2. On Ansible 2.3.2.0 it seems like something got fixed, and now sometimes I will get the error message but ansible will continue despite that. However, sometimes it does not continue and just fails the playbook.

I also noticed that often when the error occurs, if I immediately try to do an ansible action on the machine after the failure, it will also fail. My theory is that WinRM is just locked up until the process that I started previously finishes?

Potentially related to #25532 and #23320

STEPS TO REPRODUCE

- name:
  win_command: NDP451-KB2858728-x86-x64-AllOS-ENU.exe /q /norestart

EXPECTED RESULTS

ACTUAL RESULTS

 [WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last):   File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 267, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last)   File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 248, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq))   File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 207, in send_message     return self.transport.send_message(message)   File "/usr/local/lib/python2.7/dist-
packages/winrm/transport.py", line 202, in send_message     raise WinRMTransportError('http', error_message) WinRMTransportError: (u'http', u'Bad HTTP response returned from server. Code 500')

fatal: [host0]: FAILED! => {"failed": true, "msg": "ssl: (u'http', u'Bad HTTP response returned from server. Code 500')"}

[nitzmahone]

I'm going to close this one- you're using WinRM to upgrade core WinRM/Powershell components. While I've occasionally had success doing this, in most cases, the installer basically rips the rug out from underneath WinRM by terminating services or otherwise making new .NET/Powershell calls nonfunctional until a reboot has occurred... I don't believe this is a scenario that Microsoft supports or tests, either, so really the issue lies in their installer- there's not much we can do about it since it's our rug that's getting unceremoniously removed at high speed... You might want to have a go with win_psexec or win_scheduled_task...

[virtuald]

@nitzmahone the reason I cited the .NET installer is because it's a publicly available artifact that makes it pretty easy to show this issue.

Though, I admit, the error I run into when installing a very large internal MSI package is similar, but not totally the same (using either win_package or win_command and the relevant arguments):

The full traceback is:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 522, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 45, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars, wrap_async=wrap_async))
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 734, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 883, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 339, in exec_command
    result = self._winrm_exec(cmd_parts[0], cmd_parts[1:], from_exec=True, stdin_iterator=stdin_iterator)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 279, in _winrm_exec
    response = Response(self.protocol.get_command_output(self.shell_id, command_id))
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 333, in get_command_output
    self._raw_get_command_output(shell_id, command_id)
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 352, in _raw_get_command_output
    res = self.send_message(xmltodict.unparse(req))
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 207, in send_message
    return self.transport.send_message(message)
  File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 184, in send_message
    response = self.session.send(prepared_request, timeout=self.read_timeout_sec)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='172.20.1.196', port=5986): Read timed out. (read timeout=30)

fatal: [host0]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

I feel in the prior versions that I ran into the 500 error when installing large packages in the past... but I could be wrong.