Rounds 3 4 research findings

Round 3-4 research findings, prototype testing

Recommendations

Consistent with Round 2 research findings

1. Find ways to help agency implementers understand the context for a particular requirement.

2. Clarify, minimize, or remove the policy and requirement IDs on the public interface.

3. Create a way for agency implementers to browse through a full list of the topics that are available so that they don't need to know what keywords are available in advance.

New

1. Clarify the difference between keyword search and text search on the homepage.

Design opportunities

1. How might we alert users when there are changes to policies or guidelines that impact their work, without bothering them with notifications for policies that don’t affect them?
2. How might we help users understand what specific, often-used phrases in policy means?
3. Relatedly, how might we help policy writers use phrases and terms in consistent ways?

Questions for future research

Are users so discouraged by the disclaimer language that they don’t think they can trust this source for any use?

Findings

Notifications

There doesn’t appear to be one clear source for agency implementers to find out about new policies or changes to existing ones, but this is an unmet need.

“[How do you normally interact with OMB policies?] I didn’t look them up a ton but looked at them when they were emailed to me.”

“They [OMB policies] come into my awareness when I try to find them. There is no centralized tool. Sometimes I’d be having conversations with people, or a press release, or an email blast. Once acquired something relevant, I would put into a personal tracker but became unruly and irrelevant. I’d tag withs scope and relevancy, pages, hyperlink (likely 404ing). Third step is how are we going to incorporate this.”

[How do you find out about new policies?] “Most of the time it comes through fed CIO council has a distribution network. I usually get those. If not that, sometimes OMB will solicit certain agencies for feedback and I will see that via email. Sometimes it’s honestly Twitter believe it or not.”

“The things I want to know are… What’s new and what has been modified, removed, etc?”

“[What are the top 3 things we should be sure to get right?] #1 Notifications x 3”

“New policies… I like the idea but the first one [in the prototype] is a little dated but it would be useful. It be nice if it was personalized. Past search, or tailored to your interest. Pros and cons to that.”

Disclaimer

The language about ‘unofficial guidance’ works in giving users’ pause before relying on the information in the policy library, but it may erode trust to the point that it discourages any use.

“From the top, the note about “unofficial guidance” kind of threw me. […] To me it’s the canonical version no matter where it’s happening.”

“‘Includes excepts’ – is it not the whole document? [tester describes app]. It gives me pause, the excepts are useful, but sometimes there is a story to be told in the language in the memo. Those that have been around awhile can read between the lines based on the language that is chosen.”

“‘Should be considered unofficial’ that’s curious. Is the intent that it would not be hosted by a federal agency? I’d assume that on something that is trying to dispense government data but it’s the definitive source. That sentence would give me pause and give me reason to suspect accuracy of data or freshness of data.”

“There is a different meaning between unofficial and just explaining plainly and clearly what the purpose is. I can understand why they are prefacing it but it might be helpful to disclaim to read the whole document. The frequency note is also weird. If the beta is front loaded then there is enough info that it might not be definitive. Explain what more of the constraints are when in beta.”

Comprehension

Users need to understand the context around a requirement or policy (further validation from Round 2 findings).

“I want to view all requirements to see what they are.”

“This req statement isn’t useful without the context of the policy.”

“I still want to read around it [the requirement] in the policy to see what else it is trying to say.” [top three things we should get right] “#2 In-document permalinks. Not PDF wizardry but in a HTML document that will bring me back to the originating requirement of discussion.”

This is a new way of presenting and organizing policies, and that unfamiliarity initially confuses users. (further validation from Round 2)

“Not sure I’m clear what a req is inside and OMB policy. A lot of times they are very nuanced and directed in how they are done. There are different times that a req means different things within the policy.”

“This one [requirement] actually makes sense because it’s instructing me what to do.”

“The use of the term requirements throws me off. It’s really not req. It makes make me think of ‘this, this, this’ but separate from the policy they are not all written as req. These are more sections that you should be aware of. Req is such a ‘loaded’ word.”

“I guess I did a search for Digital Services under Requirements and not policies? Just reading through what came up. It looks like it's one of four pages of results. I guess you can search by requirements and this tells us how many req match the search.”

Understanding policy also depends on understanding the language.

“What would be helpful is to know why some policy titles are named the way they are and do they mean I need to do something over just something that is informative. I’d probably go by titles and see which one is closest to what I’m trying to do.”

There may be multiple ways users understand what applies to them.

“At more old office, the policy might have different actions within different scenarios. It may or may not apply to my specific scenario.”

Finding things

Users would like to be able to see a full list of the tags available so they don’t have to guess at what search terms to use. This way they can be sure they’re not missing relevant topics and policies. (further validation from round 2)

“#3 Having a site map or browse function, a way to layout everything it has. I can maybe intuit out but it be nice to see all the different areas on one page. I created my own tags but if I could see a list of the topics that would be helpful. Personal helpful notify. Topic map to click around.”

Users want to be able to find and view related documents so they can understand the full picture.

[How do you know relationship now?] “Sometimes reference is in footnote or body by saying it replaces or supersedes. With the DCLI memorandum. [google search and goes to datacenters.cio.gov] view pdf but it’s unavailable because moved to archive. Then I search again to try to find where document lives. Link roulette. [pulls up pdf] FITARA footnote that refer to other items and then have to search again.”

“It’s fair to say [my agency] interprets OMB policy. It’s not exactly 1 to 1. Generally they are reflective of OMB policy.”

“I would love to see interrelationship between documents, that would be nice to see. A relationship of documents with a term that relate back to originating document. Like the tools that do inference tools that will show you the graph format of things that are related like a tree like structure in how they are related. For me being able to discern relationship would be useful with you having to read them all. Show other documents that are related to the ones you are look at to help guide you as you explore a collection of documents.”

“If it grew to include GIA(?) audits then it would be awesome to have outstanding audits where treasury has been cited. Being a cabinet level agency pretty much everything applies to us.”

Users are confused about how search is currently presented.

“Why are you asking what agency I work for? Would the tool filter based on where I worked? Would it bring to the surface things relevant to me or is it for tracking purposes? Maybe it doesn't apply to non cabinet agencies… I’d mostly be interested in the topic.”

“Little confusing that there are two search areas and there is this (site) search area. If I had to guess I’d assume top allows me to search site … don’t know. I assume the topic search is the one in the middle.”

“The next part that threw me was the double search. Oh okay they are exposing the kind of setting I would expect in an advanced search.”

“I am always the person who is going to the advanced search anyway. I’m not a typical user. What I would expect to be a clear delineation around what I can join around commands, If I go to advanced search in google it clearly tells me the dials aI can tweak.”

Weaker signals

Users want to be able to quickly refer back to key phrases and sections within a policy.

“I’d love to be able to save my state and highlight sections of policy. Much like how I mark up the pdf. Yellowline it. Sometimes I’ll use the highlight to react to things instead of rereading the whole thing.”

Some Agency Implementers also provide feedback on proposed policies.

“Often I’m the one to formulate a response [to IT policy]. I do a lot of web searching to figure out background of this policy. Interact with OMB a lot and sometime help formulate some policies and memos.”

“The whole LRM process by which they are sent out for feedback and commentary is a nice thing to know around the structure.”