PHASE 1: Workshop results and user stories

Notice & Comment Workshop 3/8

Main question: Based on our user research, what do we want to build?

Research findings / readout

We went through the research findings we’ve found through the last few weeks of user research interviews.

Who did we speak to?

• 11 internal agency staff across EPA, CFPB, and the FCC
• 18 external users split across the spectrum of users:

[image coming soon]

Internal staff problem areas

• Sorting comments - What do we do with all these comments?
• Responding to comments - Making sure everyone is heard.

These two processes are the most time consuming for the agency staff. Sorting comments is the biggest problem area because contractors are usually pulled in to help with that process, but there isn’t one fully effective way of sorting all comments into buckets that make it easiest for agency staff to respond to the comments.

External user problem areas

• Missing context - Finding something to comment on.

  • It is very difficult to throw someone into this environment without context. Questions that arise:
    • Why a regulation vs an act of congress?
    • What’s in the FR vs the eCFR vs regs.gov?
    • What stage is the rule in?
    • Plus they need to understand the regulation itself to understand how it is changing.

• Forms of commenting - How do I comment?

  • Browser comments are looked down upon.
  • PDFs provide users with control, but are very hard to sort through and break up on the agency side.

• Very little feedback loop - Post-comment submission.

  • The general public especially doesn’t know what the checkpoints are in the commenting process:
    • What happened to my comment?
    • Did the agency hear me?
    • Did the agency respond to my comment?

Areas of improvement noted from both groups

• Open discussion - Should commenters discuss amongst themselves?
• Effective commenting - What makes a good comment? How do we teach that?

Exercise - Design studio

Set up: Based on the problem areas listed above, everyone in the room and on the phone was asked to sketch ideas for 7 minutes. When those 7 minutes were up, we went issue by issue around the room to discuss the sketches and ideas that came out of the exercise.

Some prominent sketches:

Feedback loop

[image coming soon]

Feedback loop sketches revolved around giving the commenter more context as to what they should expect next in the process beyond a receipt notice. In addition to the sketch above, we had discussion of:

• “Upon completion of comment, a notice would be provided that states: A helpful explanation would go here, particularly for novice commenters which provides info re: next steps in the process so they don’t feel left in the dark.”

• Give every commenter a tracking number they can check back with. They could also potentially click a box that allows them to be notified when the action/rule reaches the next stage. “e.g. If comment is on a proposed rule, they are notified by email (w/ hyperlink) when the final rule and response to comment come out.”

Forms of commenting

[image coming soon]

Forms of commenting sketches revolved around a few different areas:

• Make it really easy for the user to comment on specific sections of the rule and allow for them to connect their own comment to parts of the rule.

• If we are asking users to change their current processes, we should explain to them the benefits of this new way and how it will help the agency in responding to their comments faster.

• We cannot give them the same amount of control that Word can give them, but we can try to help commenters make their comments look good using rich text formats.

• We can ask more direct questions. We know PRA and other rules make this difficult, but we discussed how pulling out questions in the rule may help commenters respond.

Missing context

[image coming soon]

We can give users context in many ways. A few we discussed were around using the right sidebar to pull out important aspects of the rule along with highlighting where users are asked to comment.

[image coming soon]

In addition, we could potentially show commenters and agency staff visually what parts of the rule are getting the most comments.

Effective comments

[image coming soon]

To promote effective commenting, we could provide a template or checklist for users to read/use before, during, or after they write their comment.

[image coming soon]

Similarly, we could pull the parts of the preamble or rule they are commenting on into their comment to help avoid getting side-tracked.

Goals that stemmed from the group discussion:

• We want to make it easier for everyone, but especially, more of the general public to comment. However, we want to guide them through the process so that their comments add value to the agency’s decision making process.

• Helping to sort comments is the biggest win and selling point for agencies. However, the goal/scope of this project is focused on helping to enhance the commenting experience for industry and the public.

• On-boarding is important. Users should not need to be trained by agency staff to use this tool.

Prioritization and success metrics

Using the list of problem areas identified by the user research, we voted on what areas we thought would have the most impact on in the next 2 months of the project.

• Forms of commenting (15 votes)
• Missing context (8 votes)
• Effective commenting (6 votes)
• Feedback loop (1 vote)

Success metrics

• Time/cost spent processing comments for a similar “sized” rule (in the past) compared to the proposed eManifest User Fee rule.
• Number of substantive comments
• How many repeat users use it (long term)
• Do you feel like you were heard? (Net promoter score style rating)

User stories

[coming soon]

---

General background

• Phase 1: Generative research
  • Original research plan
  • Findings (Link TBD)
  • Workshop with EPA results and user stories < You are here!
• Phase 2: Testing the tool
  • Findings (Link TBD)
• Phase 3: Success?
  • Comment period research plan