Elastic Defend Installation Failed on Windows Server 2022

[simlecp]

Dear Support Team,

I hope this message finds you well. I am encountering an issue while attempting to install the Elastic Endpoint Security plugin on a Windows Server 2022 managed by Fleet Server within a Kubernetes standalone cluster.

The specific server in question is hosted on a dedicated cloud server. Unfortunately, I am facing an obstacle as the installation of the aforementioned plugin repeatedly fails on this particular server.

Key points to note:

• Server Details: Windows Server 2022 hosted on a dedicated cloud server.
• Issue Description: Elastic Endpoint Security plugin installation fails consistently.
• Version: 8.8.2 (Elastic agent - Elasticsearch - Fleet Server), Elastic Defend integrations 8.8.0

State: DEGRADED
Message: 1 or more components/units in a failed state
Fleet State: HEALTHY
Fleet Message: Connected
Components:
  * endpoint         (FAILED)
                     failed install endpoint service: 2023-11-21 14:35:31: info: InstallLib.cpp:531 Install failed and no rollback package exists: exit status 21
  * windows/metrics  (HEALTHY)
                     Healthy: communicating with pid '9900'
  * log              (HEALTHY)
                     Healthy: communicating with pid '11760'
  * iis/metrics      (HEALTHY)
                     Healthy: communicating with pid '1240'
  * winlog           (HEALTHY)
                     Healthy: communicating with pid '3212'
  * system/metrics   (HEALTHY)
                     Healthy: communicating with pid '5528'
  * http/metrics     (HEALTHY)
                     Healthy: communicating with pid '7616'
  * filestream       (HEALTHY)
                     Healthy: communicating with pid '6944'
  * beat/metrics     (HEALTHY)
                     Healthy: communicating with pid '6096'

It's important to highlight that this issue does not affect other servers within the same network and same Elastic Agent configuration (they're managed by fleet to). Additionally, attempts to restart and reinstall the Elastic Agent have not resolved the issue.

Additionaly i can provide elasict-agent json logs:

{"log.level":"error","@timestamp":"2023-11-21T14:35:31.166Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: Main.cpp:394 Verifying existing installation","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.166Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:602 Endpoint is not installed","context":"command output","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T14:35:31.194Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":173},"message":"after check if endpoint service is installed, err: 2023-11-21 14:35:31: info: InstallLib.cpp:602 Endpoint is not installed: exit status 1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T14:35:31.194Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":176},"message":"failed check endpoint service: 2023-11-21 14:35:31: info: InstallLib.cpp:602 Endpoint is not installed: exit status 1, try install","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: Main.cpp:368 Upgrading existing installation (protected)","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:490 Attempting to create a rollback package","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:503 Failed to create rollback package, attempting full uninstall","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: debug: Service.cpp:817 PPL is supported. This process is unprotected. (TrustLevelSid: absent)","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Service.cpp:329 OpenServiceW(ElasticEndpointDriver) failed with error 1060","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Service.cpp:187 DeleteService(ElasticEndpoint) failed with error 1072","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: warning: Util.cpp:1384 Endpoint service scheduled for deletion at next reboot.","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Service.cpp:178 OpenServiceW(ElasticEndpointDriver) failed with error 1060","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Util.cpp:1407 Endpoint driver service was unable to be deleted or scheduled for deletion.","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Service.cpp:329 OpenServiceW(ElasticELAMDriver) failed with error 1060","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Service.cpp:178 OpenServiceW(ElasticELAMDriver) failed with error 1060","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: error: Util.cpp:1433 ELAM driver service was unable to be deleted or scheduled for deletion.","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: warning: InstallLib.cpp:288 System reboot required to finish uninstall","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:507 Proceeding with install","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:206 Installing from e","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.256Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: Internal.cpp:201 Extracting installation artifacts","context":"command output","ecs.version":"1.6.0"}
"log.level":"error","@timestamp":"2023-11-21T14:35:31.461Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: debug: Util.cpp:1462 Endpoint service already in use.","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.461Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-21 14:35:31: info: InstallLib.cpp:531 Install failed and no rollback package exists","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.492Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/conn_info_server.go","file.line":45},"message":"failed accept conn info connection: accept tcp 127.0.0.1:6788: use of closed network connection","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.492Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":978},"message":"Component state changed endpoint-default (STARTING->FAILED): failed install endpoint service: 2023-11-21 14:35:31: info: InstallLib.cpp:531 Install failed and no rollback package exists: exit status 21","log":{"source":"elastic-agent"},"component":{"id":"endpoint-default","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.492Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":978},"message":"Unit state changed endpoint-default-8f30f029-3636-410e-818a-d97eec04ec95 (STARTING->FAILED): failed install endpoint service: 2023-11-21 14:35:31: info: InstallLib.cpp:531 Install failed and no rollback package exists: exit status 21","log":{"source":"elastic-agent"},"component":{"id":"endpoint-default","state":"FAILED"},"unit":{"id":"endpoint-default-8f30f029-3636-410e-818a-d97eec04ec95","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-21T14:35:31.492Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":978},"message":"Unit state changed endpoint-default (STARTING->FAILED): failed install endpoint service: 2023-11-21 14:35:31: info: InstallLib.cpp:531 Install failed and no rollback package exists: exit status 21","log":{"source":"elastic-agent"},"component":{"id":"endpoint-default","state":"FAILED"},"unit":{"id":"endpoint-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

i didn't find any issue to help me (It's not impossible that I haven't found the right topic. If there's one related to this subject, I'd appreciate being directed to it).

I would greatly appreciate any guidance or assistance you can provide to resolve this matter promptly. If further information or logs are required, please do not hesitate to reach out.

Thank you for your attention to this issue.
SD

[simlecp]

Hello,

I'm coming to the news,
Is there anything anyone can tell me about this?

Do you know of anything that might be useful to me? Is it a bug, a configuration problem?

I still don't have a solution to this problem...

Best regards,
SD