Formalizing freeform notes

[stuartpb]

See opws/opws-dataset#137 and opws/opws-validate#2: right now, many profiles have notes fields in arbitrary locations, which are specially ignored by validation.

The sustainable solution here is to understand ways in which notes are used, and put them in predictable locations, with codified semantics around how they would be presented in given use cases.

[stuartpb]

OK, I'm going through the list of all "notes" fields that are in profiles now, and I'm breaking them down into a few general categories:

Errata

• password.notes for 37signals.com ("Password editing suggests using at least 1 number but does not require it.")
• password.value.notes for c9.io ("Password form rejects passwords that are "not strong enough" based on an arbitrary character-class-based formula, unrelated to the displayed "strength-o-meter" (which is just a progress bar of the password's length up to 24 characters).")
• password.notes for codehs.com ("You can technically set a zero-length password, though the login page treats password input as "required".")
• password.value.length.notes for coderbyte.com ("Password change form will accept passwords of any length, but login input has a validation maximum of 40.")
• password.notes for doublefine.com ("Password input includes a list of what makes a "good" password, but these are not the actual password requirements.")
• password.notes for facebook.com ("Resetting a password claims that passwords must not be the same as a previous password (with "must not" in bold), but there doesn't actually seem to be a previous-password prohibition in place.")
• password.notes for filestack.com ("Password must be at least 6 characters when first registering.")
• password.value.notes for gocurb.com ("14-character limit when registering on Android")
• password.value.length.notes for gog.com ("The system accepts passwords longer than 100 characters, but does not recognize them when logging in or changing passwords.")
• password.notes for heroku.com ("Setting a password after registration only accepts passwords that contain a combination of letters, numbers, and/or symbols (consisting of more than one of these). This restriction is not enforced when changing the password after registration.")
• username.contents.notes for name.com ("Registration lists backslash as an accepted username character (it is not).")
• password.value.notes for netflix.com ("Registration lists the maximum password length as 50 characters.")
• password.value.length.notes for okcupid.com ("Registration restricts password length to 5; password reset and changing restricts password length to 6.", "Trying to change your password to anything longer than 256 characters fails with the error message "Update Failed! Could not update database; please try again later."")
• password.value.length.notes for yammer.com ("Resetting a password only enforces a length limit of 6.")

Sort of like errata, but harmless, and pretty much an irrelevant detail

• password.reset.flow.submit.destination.notes for linkedin.com ("The stub page includes a "Sign In" link, even though you are signed in after submitting the new password.")

Citations of data that is (mostly?) represented in the schema

• password.value.notes for ebay.com ("http://pages.ebay.com/help/new/contextual/create_password.html")
• username.value.notes for ebay.com ("http://pages.ebay.com/help/policies/identity-user-ids.html", "http://pages.ebay.com/help/reg/contextual/createuserid.html")
• password.value.notes for ubi.com ("https://connect.ubi.com/Default/Help?appId=c5393f10-7ac7-4b4f-90fa-21f8f3451a04#pass")
• username.value.notes for ubi.com ("https://connect.ubi.com/Default/Help?appId=c5393f10-7ac7-4b4f-90fa-21f8f3451a04#user")

Directions beyond what's in the schema

• password.reset.flow.request.notes for bountysource.com ("Enter email, then a "Forgot?" link will appear by the password field.")
• password.change.notes for c9.io ("The user settings page features a "Request password" button that initiates a password reset request.")
• password.reset.flow.request.notes for crunchbase.com ("Click the "Forgot password?" button.")
• password.change.notes for crunchbase.com ("Click the "Password" button.")
• password.change.notes for fluidui.com ("Click the "Home" menu in the upper-right corner, then click "Account Settings", then "Password" to open the password change dialog.")
• password.change.notes for hrblock.com ("Click "Account Settings" after logging in")
• password.reset.notes for tarsnap.com ("https://www.tarsnap.com/faq.html#forgotten-password")
• password.change.notes for workflowy.com ("Click the upper-right-hand corner menu, then "Settings", then "Change password" in the gray Account box.")

Directions so bad they're practically errata

• password.reset.flow.notes for pave.com ("Following the link from the password reset email directs to https://www.pave.com/password-reset, which fails to load with a "Page not found" error. However, as following the reset link also logs you in, and password change does not require the original password, you can use your account settings to change the password.")
• notes for readthedocs.org ("No way to change password once logged in.")

Nuanced contradictions of what the codified data suggests per schema

• username.notes for google.com (""Username" here describes the name for Gmail addresses. Users do not have to set up a Gmail account name if they use their own email address.\n")
• registration.notes for google.com ("Registration accepts either an existing email address or a new Gmail address to register.")
• registration.notes for ultrahook.com (""username" is technically "webhook namespace".\n")

Circumstantial UI

• password.reset.flow.request.notes for cloudflare.com ("Captcha only appears after multiple reset requests.")
• password.change.notes for crunchbase.com (""Current password" input is not displayed when the account has no password set (ie. when it was registered via third-party authentication).")
• password.reset.flow.request.notes for gog.com ("Captcha only appears after multiple reset requests.")
• password.change.notes for humblebundle.com ("Password re-authentication is via prompt after entering the password, and only needs to be entered the first time the password is changed in a session.")
• registration.form.captcha.notes for reddit.com ("Captcha is presented after entering email address.")
• registration.form.notes for twitter.com ("Username selection comes after entering full name, email address or phone number, and password.")

Further requirements and restrictions

• password.value.notes for coinbase.com ("Passwords must be rated "good" and not "too weak" by an opaque judgment algorithm.")
• registration.form.captcha.notes for doublefine.com ("Also has a CAPTCHA question ("Name a Double Fine game").")
• registration.form.notes for fluidui.com ("Must also specify an initial project name and platform.")
• registration.form.notes for godaddy.com ("Also requires a 4-digit "Support PIN" that cannot be a single repeated digit or sequential digits.")
• password.value.blacklist.notes for google.com (""Recently-used" passwords are rejected as well.\n")
• password.reset.flow.request.notes for namesilo.com ("Also requires the answer to the user's security question.")
• password.reset.onetime.request.form.notes for orcacard.com ("Requires first name, last name, email address, Orca Card number and three-digit card Verification Number, and answer to a user-knowledge security question.")
• registration.form.notes for redditgifts.com ("Also requires a shipping address.")
• password.value.notes for simple.com ("Simple has its own passphrase strength grading system, and a user's passphrase must be at least a "grade C" to pass.")
• password.reset.notes for simple.com ("Sends tokens via email and SMS, both are needed.")
• password.change.notes for upwork.com ("Security question answer required to authorize account changes from new machine.")
• username.value.notes for upwork.com ("Starting a username with a hyphen returns the error "Special characters are not allowed."")
• username.reminder.notes for wellsfargo.com ("Requires Social Security number (or ITIN) and password.")

Entirely unusual requirement

• password.reset.replacement.notes for wellsfargo.com ("Requires username or Social Security number (or ITIN), account/loan/card number (or birthdate, for mortgage or employer-sponsored retirement plan customers), and Debit/ATM card number and PIN. Password change form is presented immediately after entering all of these.")

Alternatives

• password.reset.flow.request.form.notes for hrblock.com ("Also accepts a combination of first name, last name, Social Security number, and date of birth.")
• username.reminder.request.form.notes for hrblock.com ("Also accepts a combination of first name, last name, Social Security number, and date of birth.")
• password.reset.flow.request.form.notes for netflix.com ("Also accepts a combination of first name, last name, and credit card number.")

Trivia and details outside of what's expected

• registration.notes for aur.archlinux.org ("Registering an account sends a password reset email to set a password.")
• password.reset.flow.response.email.notes for mammothhq.com ("Sender address is a long random name starting with "bounce" and ending with @email.mammothhq.com")
• password.reset.flow.notes for mammothhq.com ("This is actually a link to be signed in without a password, with which you can then change the password normally.")
• password.reset.flow.submit.sessions.notes for linkedin.com ("The form to submit the password includes a checkbox to control whether other sessions should be logged out or not.")
• password.change.sessions.notes for linkedin.com ("The form to submit the password includes a checkbox to control whether other sessions should be logged out or not.")
• registration.notes for verizonwireless.com ("Registration requires verification by text message first, then presents a form for all other registration data.")
• registration.notes for yammer.com ("Registration starts with an entered email address, then is completed after following a link sent to that email address.")

Sort of an important detail

• registration.form.terms.notes for steampowered.com ("Checkbox also agrees that you are over 13 years of age.")

---

So, from this, here's what I'm thinking:

• Add fields for errata, directions, and docs (or documentation or citations or whatever) in draft/v0.2. (This will be its own pull request, to debate where each of these should apply.)
• Figure out what to do with the last two kinds of notes (see next comment).
• Ship v0.2 (this being the main issue with v0.1, I'm fine making this the only change in v0.2, apart from the incredibly-minor JSON Schema draft-06 bump).
• Pull out the exception for notes in the validator.
• Migrate the dataset to v0.2 by merging a PR that converts all notes to the appropriate type of note.

[stuartpb]

The way I see it, there are two options for dealing with the long tail of notes encapsulated by the last two kinds of note:

• Introduce nuances and/or also (or alsomust or further or further.requirements or something that conveys this list is for additional requirements) as field types.
• File a series of pull requests to remove the last and/or second-to-last kind of notes from profiles, documenting what's removed in the pull request, and maybe filing / pinging issues for those details here as they're removed. (This would happen before proposing changes to the v0.2 draft.)

Of the two, I'm leaning more toward the latter, since the whole discussion around these notes was that they're bad catch-alls for things that should probably be described structurally (for example, the nuance of CAPTCHAs only appearing after a few failed requests should probably be represented by a dedicated "transient" field or something like that), and having a long-form place for them introduces awkward situations where consumers are trained to read for something that will eventually potentially disappear.

Basically, as a better lifecycle infrastructure emerges, we can start putting these details in pull requests / commit comments and referring out to them in schemata issues (and vice versa), so there's less harm in just leaving these aspects of the site completely undefined until the appropriate structure to accommodate them within the schema exists, where endpoint consumers can make decisions based on their information.

That's really the fundamental guiding principle to how each of these will be codified: does their presence send a clear signal to the program about how to proceed with its use case? If not, it's not worth codifying until it can be structured in such a way that does.

[stuartpb]

Perhaps a circumstances field for describing forms where certain things may not appear? The use case there is that they should be displayed as an info list if rendering "You'll need this"... maybe that means they need to be attached to specific inputs, then.

[stuartpb]

Also, I figure stuff like Google's username weirdness merits a "clarification" field for the explanation, and other sites may have similar need for clarification (like when "username" is used to refer to "customer number"). But, then, I don't know how to handle that registration caveat.

[stuartpb]

I think a few of these unusual registration flows could be described by having a circumstances field on the form level itself.

[stuartpb]

I'm just going to propose circumstances for fields themselves for now: putting circumstances on every form would be a pretty major change, and I'm not convinced is even the right model for describing a radically-different path.

[stuartpb]

Okay, now I'm going to propose further.requirements for forms and further.restrictions for passwords/usernames. I've got some thoughts on whether or not it should be merged, but I'll leave that debate for the pull request.

EDIT: Actually, I'm going to keep it to just further.requirements: any "further restrictions" on passwords described in the current notes should be conveyable through a combination of {value,contents}.{must,mustnot} and errata.

[stuartpb]

I'm proposing further.requirements as well (#7): it feels wrong to me to describe some things that are required to do a task but not others, just because they don't fit the schema. further.requirements is broad and it has the whole problem of "describing in prose things that should be considered for inclusion in the schema structure", but its presence still at least conveys something decision-worthy (whether or not someone with the values listed in the profile can just be handed off to registration without further prompting), so I figure it's worth including like this, at least for v0.2.

[stuartpb]

Based on the list above (with some recategorizations and reconsiderations), my plan for migrating issues to the new structure:

errata

• 37signals.com: password.errata <= password.value.notes ("Password editing suggests using at least 1 number but does not require it.")
• c9.io: password.errata <= password.value.notes ("Password form rejects passwords that are "not strong enough" based on an arbitrary character-class-based formula, unrelated to the displayed "strength-o-meter" (which is just a progress bar of the password's length up to 24 characters).")
  • this should arguably be mirrored into a password.value.must rule, but I'll let future issues / PRs sort that out (Update c9.io.yaml opws-dataset#152 covered how bad the strength formula is, it really is so bad it's essentially a mistake in and of itself)
• codehs.com: password.errata <= password.notes ("You can technically set a zero-length password, though the login page treats password input as "required".")
• coderbyte.com: password.errata <= password.value.length.notes ("Password change form will accept passwords of any length, but login input has a validation maximum of 40.")
• doublefine.com: password.errata <= password.notes ("Password input includes a list of what makes a "good" password, but these are not the actual password requirements.")
• facebook.com: password.errata <= password.notes ("Resetting a password claims that passwords must not be the same as a previous password (with "must not" in bold), but there doesn't actually seem to be a previous-password prohibition in place.")
• filestack.com: password.errata <= password.notes ("Password must be at least 6 characters when first registering.")
• gocurb.com: password.errata <= password.value.notes ("14-character limit when registering on Android")
• gog.com: password.errata <= password.value.length.notes ("The system accepts passwords longer than 100 characters, but does not recognize them when logging in or changing passwords.")
• heroku.com: password.errata <= password.notes ("Setting a password after registration only accepts passwords that contain a combination of letters, numbers, and/or symbols (consisting of more than one of these). This restriction is not enforced when changing the password after registration.")
• name.com: username.errata <= username.contents.notes ("Registration lists backslash as an accepted username character (it is not).")
• netflix.com: password.errata <= password.value.notes ("Registration lists the maximum password length as 50 characters.")
• okcupid.com: password.errata <= password.value.length.notes ("Registration restricts password length to 5; password reset and changing restricts password length to 6.", "Trying to change your password to anything longer than 256 characters fails with the error message "Update Failed! Could not update database; please try again later."")
• readthedocs.org: password.errata <= notes ("No way to change password once logged in.")
• upwork.com: username.errata <= username.value.notes ("Starting a username with a hyphen returns the error "Special characters are not allowed."")
• yammer.com: password.errata <= password.value.length.notes ("Resetting a password only enforces a length limit of 6.")

directions

• bountysource.com: password.reset.flow.request.directions <= password.reset.flow.request.notes ("Enter email, then a "Forgot?" link will appear by the password field.")
• c9.io: password.change.directions <= password.change.notes ("The user settings page features a "Request password" button that initiates a password reset request.")
• crunchbase.com: password.reset.flow.request.directions <= password.reset.flow.request.notes ("Click the "Forgot password?" button.")
• crunchbase.com: password.change.directions <= password.change.notes ("Click the "Password" button.")
• fluidui.com: password.change.directions <= password.change.notes ("Click the "Home" menu in the upper-right corner, then click "Account Settings", then "Password" to open the password change dialog.")
• hrblock.com: password.change.directions <= password.change.notes ("Click "Account Settings" after logging in")
• mammothhq.com: password.reset.flow.change.directions <= password.reset.flow.notes ("This is actually a link to be signed in without a password, with which you can then change the password normally.")
• pave.com: password.reset.flow.change.directions <= password.reset.flow.notes ("Following the link from the password reset email directs to https://www.pave.com/password-reset, which fails to load with a "Page not found" error. However, as following the reset link also logs you in, and password change does not require the original password, you can use your account settings to change the password.")
• tarsnap.com: password.reset.replacement.directions <= password.reset.notes ("https://www.tarsnap.com/faq.html#forgotten-password")
• verizonwireless.com: registration.directions <= registration.notes ("Registration requires verification by text message first, then presents a form for all other registration data.")
• workflowy.com: password.change.directions <= password.change.notes ("Click the upper-right-hand corner menu, then "Settings", then "Change password" in the gray Account box.")
• yammer.com: registration.directions <= registration.notes ("Registration starts with an entered email address, then is completed after following a link sent to that email address.")

documentation

• ebay.com: password.documentation <= password.value.notes ("http://pages.ebay.com/help/new/contextual/create_password.html")
• ebay.com: username.documentation <= username.value.notes ("http://pages.ebay.com/help/policies/identity-user-ids.html", "http://pages.ebay.com/help/reg/contextual/createuserid.html")
• ubi.com: password.documentation <= password.value.notes ("https://connect.ubi.com/Default/Help?appId=c5393f10-7ac7-4b4f-90fa-21f8f3451a04#pass")
• ubi.com: username.documentation <= username.value.notes ("https://connect.ubi.com/Default/Help?appId=c5393f10-7ac7-4b4f-90fa-21f8f3451a04#user")

circumstances

• aur.archlinux.org: registration.form.password.circumstances <= registration.notes ("Registering an account sends a password reset email to set a password.")
• cloudflare.com: password.reset.flow.request.form.captcha.circumstances <= password.reset.flow.request.notes ("Captcha only appears after multiple reset requests.")
• crunchbase.com: password.change.form.oldpassword.circumstances <= password.change.notes (""Current password" input is not displayed when the account has no password set (ie. when it was registered via third-party authentication).\n")
• gog.com: password.reset.flow.request.form.captcha.circumstances <= password.reset.flow.request.notes ("Captcha only appears after multiple reset requests.")
• humblebundle.com: password.change.form.oldpassword.circumstances <= password.change.notes ("Password re-authentication is via prompt after entering the password, and only needs to be entered the first time the password is changed in a session.")
• reddit.com: registration.form.captcha.circumstances <= registration.form.captcha.notes ("Captcha is presented after entering email address.")
• twitter.com: registration.form.username.circumstances <= registration.form.notes ("Username selection comes after entering full name, email address or phone number, and password.")

further.requirements

• doublefine.com: registration.form.further.requirements <= registration.form.captcha.notes ("Also has a CAPTCHA question ("Name a Double Fine game").")
• fluidui.com: registration.form.further.requirements <= registration.form.notes ("Must also specify an initial project name and platform.")
• godaddy.com: registration.form.further.requirements <= registration.form.notes ("Also requires a 4-digit "Support PIN" that cannot be a single repeated digit or sequential digits.")
• namesilo.com: password.reset.flow.request.further.requirements <= password.reset.flow.request.notes ("Also requires the answer to the user's security question.")
  • Wouldn't this be more like cause for a "knowledge" field to be added to the spec for this form?
• orcacard.com: password.reset.onetime.request.form.further.requirements <= password.reset.onetime.request.form.notes ("Requires first name, last name, email address, Orca Card number and three-digit card Verification Number, and answer to a user-knowledge security question.")
• redditgifts.com: registration.form.further.requirements <= registration.form.notes ("Also requires a shipping address.")
• simple.com: password.reset.flow.change.further.requirements <= password.reset.notes ("Sends tokens via email and SMS, both are needed.")
• steampowered.com: registration.form.further.requirements <= registration.form.terms.notes ("Checkbox also agrees that you are over 13 years of age.") see below
• upwork.com: password.change.form.further.requirements <= password.change.notes ("Security question answer required to authorize account changes from new machine.")
• wellsfargo.com: password.reset.replacement.further.requirements <= password.reset.replacement.notes ("Requires username or Social Security number (or ITIN), account/loan/card number (or birthdate, for mortgage or employer-sponsored retirement plan customers), and Debit/ATM card number and PIN. Password change form is presented immediately after entering all of these.")
• wellsfargo.com: username.reminder.further.requirements <= username.reminder.notes ("Requires Social Security number (or ITIN) and password.")

pre-existing fields

• coinbase.com: password.value.must <= password.value.notes ("Passwords must be rated "good" and not "too weak" by an opaque judgment algorithm.")
• google.com: password.value.mustnot <= password.value.blacklist.notes (""Recently-used" passwords are rejected as well.\n")
• simple.com: password.value.must <= password.value.notes ("Simple has its own passphrase strength grading system, and a user's passphrase must be at least a "grade C" to pass.")

removed, pending future accommodation

• google.com: username.notes (""Username" here describes the name for Gmail addresses. Users do not have to set up a Gmail account name if they use their own email address.\n")
• google.com: registration.notes ("Registration accepts either an existing email address or a new Gmail address to register.")
• hrblock.com: password.reset.flow.request.form.notes ("Also accepts a combination of first name, last name, Social Security number, and date of birth.")
• hrblock.com: username.reminder.request.form.notes ("Also accepts a combination of first name, last name, Social Security number, and date of birth.")
• linkedin.com: password.reset.flow.submit.destination.notes ("The stub page includes a "Sign In" link, even though you are signed in after submitting the new password.")
• linkedin.com: password.reset.flow.submit.sessions.notes ("The form to submit the password includes a checkbox to control whether other sessions should be logged out or not.")
• linkedin.com: password.change.sessions.notes ("The form to submit the password includes a checkbox to control whether other sessions should be logged out or not.")
• mammothhq.com: password.reset.flow.response.email.notes ("Sender address is a long random name starting with "bounce" and ending with @email.mammothhq.com")
• netflix.com: password.reset.flow.request.form.notes ("Also accepts a combination of first name, last name, and credit card number.")
• ultrahook.com: registration.notes (""username" is technically "webhook namespace".\n")

[stuartpb]

Actually, I'm gonna take the fairly radical step of classifying the two instances where a site has an opaque "password strength" requirement as errata rather than rules, because it fails the main litmus test of a rule, in that there is no way the user can tell whether a password would satisfy the rule or not. (There are a couple other profiles that have rules of "must satisfy the following formula", but, as bad as that is, it's not the complete failure that having opaque strength limits is, because that's at least small enough to read and understand.)

All password rules are problematic, but within the scope of the rules defined in this spec, they're at least problems that can be evaluated. Anything that makes it impossible to definitively evaluate what password may have been derived for that site's use qualifies as "errata".

[stuartpb]

OK, all the lingering notes have been removed (opws/opws-dataset#309), time to finish this:

• merge the pending proposals against this draft
• ship the draft as v0.2
• update the validator to recognize v0.2 as a SCHEMA_VERSION and no longer ignore notes (resolving Giving "notes" a magic-ectomy opws-validate#2)
• push a branch making all the note moves described above and bumping SCHEMA_VERSION to v0.2
• merge that migration

If done in this order, this should all cooperate well enough with CI.

[stuartpb]

Actually, I'm going to cut the note about the terms checkbox for steampowered.com since, after consulting the Steam Subscriber Agreement, being at least 13 years of age is a requirement for agreeing anyway (they just highlight it, presumably to deter all the 12-year-olds that sign up for Steam anyway).

[stuartpb]

OK... now I need to update the builder so it knows how to handle schema versions later than v0.1.