ISPDB: Recreate the ISPDB

[benbucksch]

1. Go through the list of largest domains
2. For each domain, look up the DNS MX record. Look at the MX hostname. Find who is the hoster. Also find out which clusters they have. E.g. Yahoo and Outlook.com have US and European clusters, and we need to differentiate them. But we want to merge live.de, live.fr, live.it, outlook.de, hotmail.de etc. into the same config, if possible. What matters is the mail storage cluster, but you can typically see that from the MX hostnames.
3. Find the correct IMAP, POP3 and SMTP configuration, using the customer help or support documents that they publish. You can typically search for " thunderbird IMAP hostname" or a similar search. But important make sure that your are the official website of the provider. Or that the config hostnames match the provider's main domain (e.g. imap.sekan.net for [email protected]).
4. We need: hostname, port number, TLS or STARTTLS (which one of the two) - each for IMAP, POP3 and SMTP. Also username form ("[email protected]" (preferred) or "foo" or something else?), and whether an app-specific password is required.
5. Write these into an Excel or LibreOffice Calc sheet or a text file.
6. Make a super simple hackish little script that reads your notes and writes the autoconfig XML file format, as defined by https://benbucksch.github.io/autoconfig-spec/draft-ietf-mailmaint-autoconfig.html
7. Check the configs using a node.js script. Use the IMAP server survey script that you created and used a few months ago, to check the configs. Let it read the autoconfig XML file, using the Mustang code, or using direct TCP connection. Obviously, login will not work, but we should at least be able to connect and capture the CAPABILITIES before login. Check that the TLS certificate does not error. Check each server config, including all IMAP, POP3 and SMTP configs listed.
8. Correct the configs based on the survey results.
9. Send me the autoconfig XML files.

But IMPORTANT: You are NOT ALLOWED to look at any other ISP config DB, neither the original ISP DB (which I, Ben Bucksch, originally created), nor at copycat websites. The entire purpose of the excersize is to re-create the information from scratch, from original sources, not from other ISP DBs. This is important for legal reasons. This is a "clean room" re-creation of the DB.

[benbucksch]

autoconfig.ods
(Copyright and database rights reserved)

Intermediate result for the database.

[benbucksch]

@jermy-c Thank you! That completes step 5 above. Please proceed with steps 6 to 9.

[benbucksch]

Form Jeremy
err.csv
output.zip
(c) and DB rights Ben Bucksch

[benbucksch]

https://github.com/mustang-im/ispdb/
🥳🎊🎉🥳🎊

[benbucksch]

@jermy-c Feedback:

output.csv

• 3 files had a trailing space
• I've added the .xml file extension (it will be removed by the deployment script when pushing to the web server)
• I've seen some invalid data, e.g. <username/> or even completely empty configs. Please check all the files manually.
• Please check why your validation script didn't catch these errors, and add more validation to check for such mistakes.
• If we have an SMTP server on port 465 or 587, do not add the one on port 25.
• Remove the servers that were failing, as recorded by err.csv
  • If the error was a wrong certificate, the error msg often tells you the right hostname. If (and only if) the other hostname is matching the ISP domain, please try with that other hostname again.
• In general, use good judgement. If there is a reasonable configuration that matches the ISP's hostnames and has SSL, that's sufficient. We don't need 2 servers per ISP, only 1 good one.

err.csv

This is excellent, thank you. Are these all failures? That's not a lot, I had expected more. Maybe the situation improved in the last 15 years.

False errors:

• Why is imap.gmail.com listed as error with a self-signed cert? That's clearly working, right? :-)
• Likewise, imap.orange.fr, port 143, starttls, is working fine

[benbucksch]

Please capture all information in the autoconfig XML. Esp. the URL of the document where you got the information. In the end, we will keep only the XML files, so all info must be in there. There is a specific field for the doc URL.

Please also capture your notes. If there is no field for something, please add an XML comment. <!-- Example -->

[benbucksch]

If the ISP offers configs that are reserved to "pro" users, use the "free" config. (But SSL is required in any case.)

[benbucksch]

Did you find any ISP that is not offering SSL?

[benbucksch]

Did you find any ISP that is not offering IMAP? (e.g. only POP3, or no server at all?)
I know of Proton, tuta, hey and a few others.