Supplemental Resources usability findings and recommendations

Contributors to these findings

• Pam Drouin, Lead Researcher
• Jamielyn Smith, Lead Designer
• Britta Gustafson, Product Manager

About these findings

• Date presented to the eRegs team: 2021-11-08
• Report exported from Dovetail (CMCS Dovetail account required 🔒)
• Note: video snippets not included in this version for privacy reasons

Next Steps

Our product manager will take the recommendations/key points and identify which stories are already addressing this, and what stories are missing.

---

Background

The eRegulations team has been adding supplemental content to the (CMCS-internal-for-now) eRegulations website. We want to learn what works and what doesn’t work well for intended users and why, so we can decide what to do about it.

• October 2021 Research purpose, goals, methodology (CMCS Dovetail account required 🔒)

Key Takeaways

• What worked well

  • Using search to navigate to regulations, something that was not possible during our last usability study!

  • The categories, labels, and descriptions were clear and interacting with them was smooth and intuitive

• Where people struggled

  • Figuring out how to finding subregulatory guidance was confusing

  • Understanding the connection between reg text and supplemental resources

  • It was unclear that the sidebar had resources at both the part and subpart level

  • Identifying relevant guidance (explicitly related to the reg section) vs implicitly related guidance

---

Task metrics overview

Tasks	Success	Partial	Error
1: Find a familiar reg using Figma	6/6	-	-
2: Find familiar reg using live site	6/6	-	-
3: Find known subreg guidance doc	3/6	-	3/6
4: Find a less familiar reg	4/4	-	-
5: Understand “Latest version as of”	-	1/1	-

Task 1: All 6 participants were able to identify a viable strategy for finding a familiar reg using Figma

• 5/6 participants said they would use the Table of Contents (ToC) first, with jump-to feature being second.

• 1/6 said they would use jump-to feature for known sections, but then use keyword search if they were looking for something they didn’t know by heart.

Task 2: All 6 participants were able to successfully find a reg they were familiar with using the live site

• 4/6 participants used the ToC on the homepage for navigation

• 2/6 used search: P4 typed "435.831" and P5 typed “community first” and both used search results page links for navigation

Task 3: All participants used the search box to look for a subregulatory guidance document, but only half were able to successfully find a subregulatory guidance document

• 2 of the 3 participants who did locate a document expressed doubt on how they would find something that is not tied to a reg (the policy is in the statute and not in regulation).

• The 3 participants who were unable to locate a document were hoping to see the document they hand in mind in the search results. 1 of those 3 seemed confused by the eCFR link on the right-hand side of the search results page.

Task 4: 4/4 participants were able to locate a reg they were less familiar with.

• They used a variety of strategies: going to eRegs Home Page ToC, using the sidebar navigation’s jump-to feature, or scrolling down the sidebar navigation to another section.

• The remaining 2 were not asked due to time constraints.

Task 5: Only one person was asked what they thought “Latest version as of” meant due to time constraints.

• The answer they gave was a reasonable interpretation, however it was not the user’s fault that it was incorrect. The latest change was due to a typographic update, not an updated rule, and this differentiation is not displayed in the user interface.

---

Primary findings

Finding #1: The connection between reg text and subpart resources isn't clear, resulting in information overload and extra work from participants to reduce or avoid that overload.

INSIGHT SUMMARY

We asked participants a few questions around browsing the sidebar content and also looking for a specific piece of subregulatory guidance they used recently. Our purpose was to test how usable the sidebar content was for policy experts.

We found it was difficult for some participants to see the connection between the reg text and the resources on the right side — not so much because it was confusing, but because it didn’t match their expectations or needs. Participants expected to see subregulatory guidance that was directly applicable to the section they were looking at. It was also not readily apparent that subregulatory guidance was already assigned by topic, as some participants expressed wanting to see and filter guidance by topic.

While subregulatory guidance documents are technically organized by topic because they are manually linked to regs that they are topically related to (whether explicitly or implicitly), how eRegulations surfaces (or doesn’t surface) this connection between reg text and resources resulted in an experience that is neither clear nor usable to participants.

We observed issues with a participant (P4) who starts with the regulation section first and then looks for guidance, as well as a participant who starts with the guidance first and then looks up the relevant regs (P2). This issue also dovetails with P5 looking for a subregulatory guidance document that’s not tied to a reg (policy is only in statute not regulations).

The impact of this problem is an information overload that forces the user to perform additional tasks to sift what’s explicitly relevant to the section or topic they are researching from what’s implicitly relevant to their research, such as combing through all the guidance in the sidebar or turning to a colleague instead.

SUPPORTING EVIDENCE

Problem: an unclear connection between the reg text and resources

When asked to browse the sidebar resources, P2 was excited about seeing the available subregulatory guidance, but remarked that it was not really connecting to what they were seeing in the middle of the screen.

P2 explained that there is not enough information in the sidebar for them to determine whether a sidebar item is connected to a citation. Specifically with State Medicaid Director Letters (SMDLs), they wouldn’t know which letter among the list of letters is connected to the specific citation they are looking for, so it would still require effort to determine that connection.

We confirmed with P2 our understanding that when going to an exact citation, they expected to only see resource items that only related to that specific citation or section.

When asking P3 to find a familiar piece of subregulatory guidance, they remarked that it was nice to have the subregulatory guidance there but hard to know what is there. They said this in the context of being within a reg. P3 stressed that [paraphrased because not recorded]:

1. It’s more important to have a topic than the document title or vehicle (SMDL, SHO, CIB, etc.)
2. Internal and external audiences almost always go by topic - mostly go search by topic.

P3 said to the degree that subregulatory guidance is linked to the reg could be helpful, but if they didn’t know some guidance was put out, they wouldn’t know how to go about looking for it on eRegulations. They would go to Medicaid.gov where they could search by topic.

Simply displaying the subregulatory guidance next to the related reg text wasn't enough to communicate to P3 or P2 that it was already topically arranged by regulation structure.

Impact: too much noise, not enough signal

Sidebar resources are tied to regulations through explicit or implicit relationships. Participants preferred that explicit relationships be made visible when looking up specific regulations in order to find relevant information quickly and efficiently. Not differentiating between what’s explicitly related and what’s implicitly related led P4 to believe that most resources are not relevant to the particular section they were looking at.

They anticipated that they would have to reach out to a colleague who might have the relevant information they need instead.

An alternate way of solving this information overload problem (instead of turning to a colleague) is manually going through everything in the subregulatory guidance category, but "that's a lot."

An additional confusion point is that, within subpart resources, P3 in particular struggled to distinguish between the part and the subpart.

Participant suggestions to help us clarify the problem

A few participants described solutions to the problems they had with the disconnectedness between reg text and subpart resources sidebar and the information overload they encountered within the resources sidebar.

P5 asked if subregulatory guidance that isn’t directly related would be made visually distinct somehow. We asked if that would be helpful and they said that while they’d like that level of detail to be teased out, they know that the subject matter is so nuanced it might not be worth the effort.

P4 talked about needing to see which sections are mentioned in this guidance with tags, or if there were different sections, being able to filter down to a specific section. They’re highlighting a need to quickly identify whether something in the sidebar mentions the section they are looking at, and also the ability to filter everything that’s ever been written that, for example, mentions §435.831(g).

P2 described the disjointed feeling they were experiencing where they were in the reg text (looking at the navigation sidebar) and trying to make a connection to the subpart resources. They expressed a need for a greater visual connection between everything they were seeing by using a metaphor familiar to them: an email inbox that reads from left to right (sidebar navigation, subpart resources, reg text).

IMPLICATIONS

While the original eRegulations was structured by section because it worked best for the kind of regulations it supported, early on we determined that structuring the reg text by subpart was the best solution for intended users. We wanted to emphasize the reg reading experience and then, with this study, test the resource-access experience of subpart resources.

However, this research demonstrates that structuring resources by subpart is too broad to be useful. A good reading experience for the reg is not at the same level of granularity for accessing resources related to the reg. We had come to assuming that people would need all reg views (full, subpart, and section) but now we definitively know that in addition to a multi-view reading experience, people also need to see when a reg section has any guidance that pertains to it, and when guidance explicitly mentions a reg.

If we surface to users those explicit connections, as well as provide resources that implicitly relate to the reg text in a de-emphasized way, this will likely help people review appropriateness of connections while in the reg text, as well as increase confidence in SME curation process.

RECOMMENDATIONS

Explore ways of making the relationship between the reg text and the content within the subpart resources sidebar more concrete to users

Explore ways to increase the signal–to–noise ratio, such as differentiating explicitly relevant content from less relevant content, as it will lessen the information overload participants experience

---

Secondary findings

Finding #2: Limiting the scope of search to only regulations does not meet user needs or expectations

INSIGHT SUMMARY

In previous studies, participants wanted to search within the resources sidebar, in addition to using the global search feature for both regulation text keyword queries, and direct navigation to citations.

When asked to find a subregulatory guidance document they recently referenced, all six participants expected to be able to use search to quickly find the document.

Because there’s currently only one prominent search bar on the website, all participants used the search bar in the header to look for the document, but only half (P3, P4, and P5) were able to successfully find the document they searched for.

• Two of the three participants (P4 and P5) who successfully located a document expressed doubt in how they would find something that wasn’t tied to a regulation.

• The three participants (P1, P2, and P6) who were unable to locate a document were hoping to see the document they had in mind on the search results page. P6 was further confused by the eCFR link in the right sidebar on the search results page.

• Two participants (P2 and P5) wanted the ability to do a “reverse search,” where they start with the subregulatory guidance, then look up relevant regulations.

• Two participants (P1 and P4) expected to see a separate search bar in the right sidebar that would specifically search within the supplemental content resources.

Since it’s already easy to find regulations online, within eRegulations and in other tools like eCFR and Google, introducing the ability to search within the supplemental content resources would make eRegulations more useful to people. See the “Next steps” section for details.

SUPPORTING EVIDENCE

Problem: the scope of the search feature is too limited

The scope of the search bar in the header was unclear to some, but all participants wanted to search and saw that as their only option.

Because the search bar said “Search Regulations,” P4 wondered if it only searched for text within regulations or if there was also a way to search for subregulatory guidance.

• P1 wanted to search, but didn’t see an obvious place to search.
• P1 also expected to see a search bar in the right sidebar. In the absence of a separate search bar, they expected the search bar in the header to clearly state that it searched both subregulatory guidance and regulations.
• Some participants prefer to start with the subregulatory guidance first, then look up relevant regulations.
• P2 described doing a “reverse search” and wanted the ability to start their search at any point, as opposed to being required to start with the regulation text.
• P5 saw value in both pathways: starting with the regulation first and starting with subregulatory guidance first.

Impact: participants are unable to quickly and easily find resources

Once participants realized the search feature was limited to searching the regulation text, they were unsure how to search for subregulatory guidance documents that weren’t explicitly connected to the regulation text.

P4 and P5 didn't know what search terms to use and after executing a search, they were unable to find relevant search results. P4 and P5 both said they wouldn't know how to find a document that was not tied to a regulation.

P2 mentioned it’s generally more challenging to find resources than regulations, so the site would be more useful if it included a resources search feature. The pointed out that regulations are easily found in Google, but the resources are much more difficult to find. THey were hoping eRegulations would make it a little easier for them to find.

Feedback was mixed on whether or not it was more useful to have a single search experience encompassing both supplemental content and regulation text, or two separate search experiences.

While some participants wanted to see everything at once, there was some concern that a single search experience could serve too many results to be useful.

Filters and some sort of advanced search could help with this. This requires further study.

RECOMMENDATIONS

• Expand the scope of the search feature to include resources in addition to regulation text
• Determine whether we need search to include rule names?
• Explore ways to enable searching and filtering within the resources sidebar
• Introduce a direct pathway to the resources from the homepage to enable “reverse searching” and browsing the full resource library, without having to start with the regulation text first
• Remove the eCFR link in the right sidebar on the search results page

Finding #3: The resources sidebar would be more useful with a few content and interaction additions

INSIGHT SUMMARY

The majority of participants preferred the empty state design that displayed all categories, combined with category-specific empty state messages (Option B), over the design that only displayed categories with content (Option A).

• P1, P3, P5, P6 thought Option B was clear, informative, and assuring.
• P4 liked the confirmation that there was nothing to display offered in Option B, but had a slight preference for Option A because it was less cluttered. They also mentioned that after spending more time using the site, they would understand why certain categories weren’t being displayed.
• P2 wasn’t asked to evaluate the empty state designs due to time constraints.

Although Option B was preferred, there was some concern over the ambiguity of the empty state copy.

• P1 wanted confirmation that the resources were comprehensive and questioned why the copy was vague.

We can confidently move forward with Option B. We should also explore ways to make the empty state copy more precise and supplement this copy with more explanation on the home / about page.

SUPPORTING EVIDENCE

It’s more clear, informative, and assuring to display all category labels along with category-specific empty state messages

P1, P3, P5, P6 preferred Option B.

P4 had a slight preference for Option A because it was less cluttered.

More precise empty state copy would be helpful

P1 questioned why the copy stated, “No related statutes compiled for this subpart” as opposed to something more precise like, “No related statutes have been issued for this subpart.”

The empty state message reassured P4 that they weren't simply overlooking a resource. However, they mentioned wanting to verify that there wasn't a resource.

RECOMMENDATIONS

• Move forward with Option B
• Explore ways to make the empty state copy more precise and supplement this copy with more explanation on the home / about page

Finding #4: When there are no resources to display, it's more informative and assuring to display all category labels along with category-specific empty state messages

INSIGHT SUMMARY

Participants found that interacting with the resources sidebar through opening and closing the category menus to be smooth and intuitive. However, they also wanted the ability to see all resources at once.

• P1 said being able to view everything at once would help with scannability.
• P2 and P5 said they don’t always know the specific resource format (e.g., SMDL or SHO) they’re looking for, but they have a keyword or phrase in mind, so being able to view all would allow them to quickly search the resources using Ctrl + F.
• P4 liked the idea of having more than one way to view the resources.

Participants would consider the resources to be more comprehensive if they also included the State Medicaid Manual and final rules from before 1994.

• P4 and P5 understood why the State Medicaid Manual wasn’t included in the resources sidebar. And although parts of the State Medicaid Manual have become obsolete, they still consider it to be a helpful resource.
• P4 described referencing final rules from as far back as 1978, so including resources from before 1994 would be useful.
• In the absence of that older content, P1 and P4 said it would be helpful if the site clearly stated how historically comprehensive the resources were.

There are a few different ways to introduce the ability to see all resources at once. This relates to the insight about participants expecting to use search for both resources and regulation text keyword queries, and should be explored in tandem. We’re already looking into adding older resources. We’re also in the process of writing more precise copy for empty states / resource comprehensiveness, and exploring ways to supplement this copy with more explanation on the home / about page.

SUPPORTING EVIDENCE

Although interacting with the resources sidebar was smooth and intuitive, it would also be helpful to see all resources at once

P1 said being able to view everything at once would help with scannability.

P2 and P5 said they don’t always know the specific resource format (e.g., SMDL or SHO) they’re looking for, but they have a keyword or phrase in mind, so being able to view all would allow them to quickly search the resources using Ctrl + F.

P4 liked the idea of having more than one way to view the resources.

Although interacting with the resources sidebar was smooth and intuitive, it would also be helpful to see all resources at once

P4 frequently references the SMM, although they “proceed with caution” because it’s not up to date. This is a request we’ve heard in past usability studies as well.

P5 said it would be helpful to include the SMM because it’s technically published guidance. An added benefit of its inclusion is that it might force CMS to update it.

Although interacting with the resources sidebar was smooth and intuitive, it would also be helpful to see all resources at once

P4 described referencing resources from as far back as 1978, so including resources from before 1994 would be useful.

In the absence of that older content, P1 and P4 said it would be helpful if the site clearly stated how historically comprehensive the resources were.

They weren't confident eRegulations would have all of the resources they need, partly because it's a new tool, and partly because CMS doesn't always catalog content comprehensively.

Miscellaneous content additions

When asked about content additions, P1 suggested the following, but they weren't absolute necessities:

• Definitions for each sidebar category
• States managed care programs (state by state information)
• GAO and OIG studies

P2 also had a few gentle suggestions:

• State plan amendment templates / reference material for other states
• Mailboxes where states submit questions (how to request technical assistance)
• PRA resources

P5 suggested including ARA memos.

RECOMMENDATIONS

• Explore ways to introduce the ability to see all resources at once. This relates to the insight about participants expecting to use search for both resources and regulation text keyword queries, and should be explored in tandem.
• Continue looking into adding older resources.
• Consider adding the State Medicaid Manual and other content suggestions.
• Explore ways to write more precise copy for empty states / communicating resource comprehensiveness and explore ways to supplement this copy with more explanation on the home / about page.

---

Tertiary findings

Finding #5: Further wayfinding data

P2 experienced an issue we’ve identified in the decision/problem doc we have on our wiki: Wayfinding while reading. They found it difficult to tell which subpart they were in after jumping to a specific section.

Finding #6: making eRegulations' value prop more prominent

P1 had participated in an earlier study, and noticed one day that the Part ToC sidebar wasn’t there (unlike the prototype we were testing). They made the good point of noting that if they didn’t know subreg guidance wasn’t inside the reg text, they likely would not click into the reg text because they already are familiar with it.

A good reminder to surface the key value proposition of subregulatory guidance alongside the regulations more consistently.

---

Likelihood of use

Moderator question: After everything you’ve seen, is eRegulations in its current form enough to motivate you to come back tomorrow?

P1: Oh, for sure! […] it's so easy to find […] you've got the final regs and I don't have the final reg for 2020 printed out. So I know immediately where to go, to find it on this website, if I don't want to go hunting in my paper copy, or for some reason, I don't know where it is.

P2: Yes. I think I will. This will be something that will save on my favorites and I will go to it pretty often.

P3: [paraphrased] A little hard to say since eCFR is new so still learning that. […] Would everything still be on Medicaid.gov? Then people just have options for researching for regs and two places for subreg guidance. Not sure which I would use.  Might use eRegs if have more searchability

P6: I like the potential. Let me just phrase it that way. There's the potential to be really helpful here and that's exciting […] I like the idea that it's in one place to be able to do all of that searching and researching questions. There's a lot of utility there.

---

Recap of recommendations

Relationship between reg text and sidebar resources

• Explore ways of making the relationship between the reg text and the content within the subpart resources sidebar more concrete to users
• Explore ways to increase the signal–to–noise ratio, such as differentiating explicitly relevant content from less relevant content, as it will lessen the information overload participants experience.

Searching, filtering, and viewing resources

• Expand the scope of the search feature to include resources in addition to regulation text
  • Open question: Do we need search to include rule names?
• Explore ways to enable searching and filtering within the resources sidebar
• Remove the eCFR link in the right sidebar on the search results page
• Explore ways to introduce the ability to see all resources at once
• Introduce a direct pathway to the resources from the homepage to enable “reverse searching” and browsing the full resource library, without having to start with the regulation text first

Content additions

• Continue looking into adding older resources
• Consider adding the State Medicaid Manual and other content suggestions

Communicating empty states

• Move forward with empty state design Option B
• Explore ways to write more precise copy for empty states / communicating resource comprehensiveness and explore ways to supplement this copy with more explanation on the home / about page.