SSH Scan Failed for ed25519-only servers

[532910]

Scan Failed: The Mozilla SSH Observatory scans from sshscan.rubidus.com at 45.55.176.164.
Many systems are configured with firewalls that block SSH access. To successfully scan with the SSH Observatory, access must be granted to Mozilla's scanning system.

[floatingatoll]

Hi, is there a question we can answer about this text?

[532910]

@floatingatoll if you asking me, then sorry, I didn't understand the question

[floatingatoll]

@532910 Hi, you've pasted an error message and help text from the Observatory website. Is there an issue you're trying to report? Is there something we can help you with? Do you have a question about this error message that we can help answer?

[532910]

The error I've pasted is the issue I'm trying to report.
I have no question, just like to report that SSH Observatory doesn't work on https://observatory.mozilla.org

[532910]

Aaa, it wants to say that ssh access to my host is firewalled --- it's lie, my ssh is widely opened!

[floatingatoll]

It works for the public SSH site at 'github.com':

https://observatory.mozilla.org/analyze/github.com#ssh

Without knowing the hostname you're attempting to scan, it's hard to say what's wrong. Is your SSH host configured in IPv6-only mode, or to use a port other than 22?

[532910]

I've tried 3 different hosts all with ssh on 22 port and without firewall. All of them gives the same error.
Two of them are dual-stack, one ipv4 only.

[532910]

outerface.net boo.outerface.net zip.outerface.net g.outerface.net

[532910]

https://www.isitblockedinrussia.com says sshscan.rubidus.com and 45.55.176.164 are not blocked in russia.

[532910]

% curl -X POST "https://sshscan.rubidus.com/api/v1/scan?target=outerface.net"                      
{"uuid":"2dd65519-9e3d-41e5-aff8-352995fa66fc"}
 curl "https://sshscan.rubidus.com/api/v1/scan/results?uuid=2dd65519-9e3d-41e5-aff8-352995fa66fc"
{"status":"UNKNOWN"}

[claudijd]

I no longer actively work on this project, but as an effort to help you solve the problem. I did a quick test using the ssh_scan binary that the SSH Observatory uses as it's engine...

$ ./bin/ssh_scan -t outerface.net

and I was able to get a successful scan result. My suspicion here is that the server infra is experiencing a disk usage issue, I'll pop on the server quick and see if I can address this.

[claudijd]

The web server appears to be serving static content here:

https://sshscan.rubidus.com/

[claudijd]

I was able to run the example client's that utilize the production API endpoint here to run a scan for ssh.mozilla.com:

$ python client.py
[+] Submitting scan request for ssh.mozilla.com
[+] Got 200 OK {"uuid":"1a35c92e-bc33-4391-9f9a-4bf951d7bb9b"}
[+] Checking for scan results
[+] Backing off for a half sec to let scan to complete
[+] Checking for scan results
[+] Backing off for a half sec to let scan to complete
[+] Checking for scan results
[+] Backing off for a half sec to let scan to complete
[+] Checking for scan results
[+] Backing off for a half sec to let scan to complete
[+] Checking for scan results
{"ssh_scan_version":"0.0.35","ip":"63.245.208.111","hostname":"ssh.mozilla.com","port":22,"server_banner":"SSH-2.0-OpenSSH_7.4","ssh_version":2.0,"os":"unknown","os_cpe":"o:unknown","ssh_lib":"openssh","ssh_lib_cpe":"a:openssh:openssh:7.4","key_algorithms":["[email protected]","ecdh-sha2-nistp521","ecdh-sha2-nistp384","ecdh-sha2-nistp256","diffie-hellman-group-exchange-sha256"],"encryption_algorithms_client_to_server":["[email protected]","[email protected]","[email protected]","aes256-ctr","aes192-ctr","aes128-ctr"],"encryption_algorithms_server_to_client":["[email protected]","[email protected]","[email protected]","aes256-ctr","aes192-ctr","aes128-ctr"],"mac_algorithms_client_to_server":["[email protected]","[email protected]","[email protected]","hmac-sha2-512","hmac-sha2-256","[email protected]"],"mac_algorithms_server_to_client":["[email protected]","[email protected]","[email protected]","hmac-sha2-512","hmac-sha2-256","[email protected]"],"compression_algorithms_client_to_server":["none","[email protected]"],"compression_algorithms_server_to_client":["none","[email protected]"],"languages_client_to_server":[],"languages_server_to_client":[],"auth_methods":["publickey"],"fingerprints":{},"duplicate_host_key_ips":[],"compliance":{"policy":"Mozilla Modern","compliant":true,"recommendations":[],"references":["https://wiki.mozilla.org/Security/Guidelines/OpenSSH"],"grade":"A"},"start_time":"2020-05-27 20:12:02 +0000","end_time":"2020-05-27 20:12:04 +0000","scan_duration_seconds":1.748478882,"status":"COMPLETED"}

[claudijd]

Using the ruby client, I tried outerface.net:

$ ruby client.rb outerface.net
[+] Submitting scan request for outerface.net
[+] Got 200 {"uuid":"94b36633-74aa-4e5f-99e4-036608239a89"}
[+] Checking for scan results
[+] Backing off for a half sec to let scan to complete
...

But it never returns a successful response.

[claudijd]

Additionally, the stats endpoint appears to be returning content:

https://sshscan.rubidus.com/api/v1/stats

[claudijd]

I was able to conduct a scan using the ssh_scan binary on the ssh_scan_api server for outerface.net...

ssh_scan -t outerface.net

[
{
"ssh_scan_version": "0.0.35",
"ip": "2001:470:28:295::1",
"hostname": "outerface.net",
"port": 22,
...

Note that the host in question is resolving a v6 address and the testing of ssh_scan_api against IPv6 enabled assets doesn't have good test coverage. I could take a peek at the API code and see if there's something obviously wrong here.

[532910]

zip.outerface.net has no ipv6 but has the same issue

[claudijd]

Thanks for that, I just figured out what the issue is...

"error": "could not settle on host_key algorithm"

This error is inserted with this code within the SSH Scan engine:

https://github.com/mozilla/ssh_scan/blob/master/lib/ssh_scan/scan_engine.rb#L95-L120

[claudijd]

I get that error for outerface and zip.outerface hosts. I'm wondering what host-key algorithms you are using now, as this could be a gap in Ruby Net:SSH's support.

[claudijd]

Here's some insight into the error from raw Net::SSH using presumably net-ssh 5.2.0, I see they have a 6 release they are working on, I could try to see if that has better support for your host-key algos.

repro.rb

require 'net/ssh'

net_ssh_session = Net::SSH::Transport::Session.new(
'outerface.net',
:port => 22,
:timeout => 5,
:paranoid => false
)
auth_session = Net::SSH::Authentication::Session.new(
net_ssh_session, :auth_methods => ["none"]
)
auth_session.authenticate("none", "test", "test")
result.auth_methods = auth_session.allowed_auth_methods
net_ssh_session.close

$ gem list | grep net-ssh
net-ssh (5.2.0, 5.0.2, 4.2.0)
$ ruby repro.rb
/Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/algorithms.rb:371:in negotiate': could not settle on host_key algorithm (Net::SSH::Exception) from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/algorithms.rb:347:in negotiate_algorithms'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/algorithms.rb:227:in proceed!' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/algorithms.rb:167:in accept_kexinit'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:210:in block in poll_message' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:190:in loop'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:190:in poll_message' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:225:in block in wait'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:223:in loop' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:223:in wait'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:90:in initialize' from repro.rb:3:in new'
from repro.rb:3:in `

'

[claudijd]

Here's the issue, which is more clearly articulated in the 6.x version of Ruby Net:SSH...

Server host_key preferences: ssh-ed25519
Client host_key preferences: [email protected],[email protected],[email protected],ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,[email protected],[email protected],ssh-rsa

$ ruby repro.rb
/Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/algorithms.rb:394:in negotiate': could not settle on host_key algorithm (Net::SSH::Exception) Server host_key preferences: ssh-ed25519 Client host_key preferences: [email protected],[email protected],[email protected],ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,[email protected],[email protected],ssh-rsa from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/algorithms.rb:368:in negotiate_algorithms'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/algorithms.rb:245:in proceed!' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/algorithms.rb:185:in accept_kexinit'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:210:in block in poll_message' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:190:in loop'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:190:in poll_message' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:225:in block in wait'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:223:in loop' from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:223:in wait'
from /Users/jclaudius/.rvm/gems/ruby-2.4.2/gems/net-ssh-6.0.2/lib/net/ssh/transport/session.rb:90:in initialize' from repro.rb:3:in new'
from repro.rb:3:in `

'

[claudijd]

More backstory here:

net-ssh/net-ssh#476

[532910]

Yep, only ed25519 is enabled on the all hosts above.

[claudijd]

I'm looking at workarounds to equip the docker images that net-ssh/net-ssh#476 suggests as a fix

[claudijd]

This PR mozilla/ssh_scan#519 should address the ed25519 gap in native net-ssh and adds the necessary requirements in ssh_scan, which is a core dependency for ssh_scan_api.

[claudijd]

I just landed mozilla/ssh_scan#519 in ssh_scan v0.0.43 release, which can be found here: https://github.com/mozilla/ssh_scan/releases/tag/0.0.43

This will still need to get deployed to ssh_scan_api, of which I'll send a PR for that, but the deployment may lag.

[claudijd]

Here's the PR for the ssh_scan_api engine bump: mozilla/ssh_scan_api#171

This still needs to be manually deployed to prod, but it's moving.

[532910]

Thank you, Jonathan (: