Round 2 research findings

Round two research findings – prototype testing

Recommendations

1. Make it easier for agency implementers to view requirements within the context of their originating policies.
2. Users should be able to navigate to the full policy from a requirement.
3. Allow agency implementers to both link directly to a specific requirement as well as to the entire policy.
4. Clarify, minimize, or remove the policy and requirement IDs on the public interface.
5. Create a way for agency implementers to browse through a full list of the topics that are available, and navigate to relevant policies and requirements through that list.
6. Allow agency implementers to easily find related documents (for example, guidance or clarifications) to a given policy or requirement.

Questions for future research

• How do different agency implementers understand and think about policies?
• How do different agency implementers prioritize policies and requirements?
• How do different types of users want to navigate the site?
• Why do implementers expect a PDF, what value it provides them and what difficulties they encounter?

Comprehension

The requirement/policy IDs don’t make sense to users.

“I don’t know what that number [requirement ID] means.”

[Referring to the policy ID] "What does that number mean?"

‘It [requirement ID] has a prominent place. Feel like it should mean more than it is.’

Prioritization

Prioritization and finding things are intrinsically linked – agency implementers want to be able to filter out what doesn’t apply to them (low priority), and surface the most important or urgent requirements and policies that do.

Users have different ways of deciding what is and isn’t a priority. Some users believe everything is a priority because that’s what they’d be audited on, others feel that some policies are outdated or irrelevant and want to focus on requirement due dates or policy recency.

“The older stuff is already known and doesn’t apply sometimes. OMB doesn’t care about some of those older ones so why would I.”

“[Showing what’s most important is] not a thing. Auditors don’t care. If there is an imperative and we don’t do it then it’s just as much a black mark no matter where it is.”

“Knowing what is due first would allow me to prioritize. Sort by date is a big thing for me to do.”

“[One of the reason’s I research policy is] to see what’s new. Be great to see if I’ve missed anything. Very difficult to keep recent on anything that’s going on.”

“Maybe add a filter when guidance was added; last year, etc… It filters out the noise.”

“Sorting by newest would help filter out older stuff.”

Users need to understand the context around a requirement or policy.

“Where context is missing it creates a lot of FUD.” (Fear, uncertainty, and doubt)

“There is 4000+ policies for our services. On top of that OMB will issue its own guidance and used for any audit. Doesn’t matter my interpretation on something if OMB says something different. Greater detail could be tactical.”

This is a new way of presenting and organizing policies, and that unfamiliarity initially confuses users.

“First I’d be looking for a “what is this” Welcome, what is this, etc.”

“[What are the blocks of text on the page?] Doesn’t look familiar to me. Regulations? Doesn’t look familiar. Sunset seem like maybe rule or law. Draft policy?”

“I don’t really see the requirements as separate from policy.”

“We think about the law and the policy as the same.”

Finding things

Users would like to be able to see a full list of the tags available so they don’t have to guess at what search terms to use. This way they can be sure they’re not missing relevant topics and policies.

“Other way [I’d search] is the keywords in a title. Fitara… not finding anything. Software lifecycle. GFE.”

“ In terms of opening screen I’d prefer a dashboard that has an “organize by”, “related to”, “search by” or some sort of an explore.”

“Topic was clear but Keyword was not. I would assume it checkboxes and I wouldn’t have to guess on the type ahead.” (emphasis added)

“I expected some sort of tag cloud. Popular/more frequent topics. Expected this on the left side.”

Users use a lot of different techniques to find policies and requirements that pertain to them.

“Is there a bookmark capability? I would bookmark it in the browser? Not in the app. Maybe a button to trigger a bookmark.”

“Ctrl-F due to policy titles having very little to do with the contents. Hope that the terms I care about are in the descriptions.”

“This is a filtering action. That’s nice.”

Sharing

Agency implementers send others both an entire policy (for context) as well as specific parts of the policy.

“What I was hoping for was a permalink to a requirement. Context is still important. But I would love to hyperlink to just the requirement. Context of where it relates in the policy document.”

[How would you share lists of certain requirements? Say if you’ve filtered something down.] “What I do today: get the pdf, highlight in the pdf, mail that. […] Ability to share whole policy would be great.”

[Would it be important to reference where the requirement sat in the list (context) [when sharing] or just one requirement?] “Generally, I’m looking for a specific requirement, not a series. Often individual requirements have nothing to do with what’s above or below.“

“I only have access to the pdf. I can just do a pull quote and link to the overall document.“

Format

Agency implementers expect to read a policy as a PDF, but some express a preference for a web page.

[What did you think that url to be?] “Machine readable PDF. But a nicely formatted web page would be ideal.“

[What do you think read-the-whole policy would take you?] “PDF. ‘The actual document’”.

“I’d much rather send folks to a website and not just a document.”