Skip to content
This repository has been archived by the owner on Dec 8, 2017. It is now read-only.

PHASE 1: Workshop results and user stories

Jen E edited this page Oct 24, 2016 · 9 revisions

Notice & Comment Workshop 3/8

Main question: Based on our user research, what do we want to build?

Research findings / readout

We went through the research findings we’ve found through the last few weeks of user research interviews.

Who did we speak to?

  • 11 internal agency staff across EPA, CFPB, and the FCC
  • 18 external users split across the spectrum of users:

Spectrum of Users

Internal staff problem areas

  • Sorting comments - What do we do with all these comments?
  • Responding to comments - Making sure everyone is heard.

These two processes are the most time consuming for the agency staff. Sorting comments is the biggest problem area because contractors are usually pulled in to help with that process, but there isn’t one fully effective way of sorting all comments into buckets that make it easiest for agency staff to respond to the comments.

External user problem areas

  • Missing context - Finding something to comment on.

    • It is very difficult to throw someone into this environment without context. Questions that arise:
      • Why a regulation vs an act of congress?
      • What’s in the FR vs the eCFR vs regs.gov?
      • What stage is the rule in?
      • Plus they need to understand the regulation itself to understand how it is changing.
  • Forms of commenting - How do I comment?

    • Browser comments are looked down upon.
    • PDFs provide users with control, but are very hard to sort through and break up on the agency side.
  • Very little feedback loop - Post-comment submission.

    • The general public especially doesn’t know what the checkpoints are in the commenting process:
      • What happened to my comment?
      • Did the agency hear me?
      • Did the agency respond to my comment?

Areas of improvement noted from both groups

  • Open discussion - Should commenters discuss amongst themselves?
  • Effective commenting - What makes a good comment? How do we teach that?

Exercise - Design studio

Set up: Based on the problem areas listed above, everyone in the room and on the phone was asked to sketch ideas for 7 minutes. When those 7 minutes were up, we went issue by issue around the room to discuss the sketches and ideas that came out of the exercise.

Some prominent sketches:

Feedback loop

Feedback loop sketch

Feedback loop sketches revolved around giving the commenter more context as to what they should expect next in the process beyond a receipt notice. In addition to the sketch above, we had discussion of:

  • “Upon completion of comment, a notice would be provided that states: A helpful explanation would go here, particularly for novice commenters which provides info re: next steps in the process so they don’t feel left in the dark.”

  • Give every commenter a tracking number they can check back with. They could also potentially click a box that allows them to be notified when the action/rule reaches the next stage. “e.g. If comment is on a proposed rule, they are notified by email (w/ hyperlink) when the final rule and response to comment come out.”

Forms of commenting

Forms of commenting sketches 1

Forms of commenting sketches 2

Forms of commenting sketches revolved around a few different areas:

  • Make it really easy for the user to comment on specific sections of the rule and allow for them to connect their own comment to parts of the rule.

  • If we are asking users to change their current processes, we should explain to them the benefits of this new way and how it will help the agency in responding to their comments faster.

  • We cannot give them the same amount of control that Word can give them, but we can try to help commenters make their comments look good using rich text formats.

  • We can ask more direct questions. We know PRA and other rules make this difficult, but we discussed how pulling out questions in the rule may help commenters respond.

Missing context

Missing context sketches 1

We can give users context in many ways. A few we discussed were around using the right sidebar to pull out important aspects of the rule along with highlighting where users are asked to comment.

Missing context sketches 2

In addition, we could potentially show commenters and agency staff visually what parts of the rule are getting the most comments.

Effective comments

Effective comments sketching 1

To promote effective commenting, we could provide a template or checklist for users to read/use before, during, or after they write their comment.

Effective comments sketching 2

Similarly, we could pull the parts of the preamble or rule they are commenting on into their comment to help avoid getting side-tracked.

Goals that stemmed from the group discussion:

  • We want to make it easier for everyone, but especially, more of the general public to comment. However, we want to guide them through the process so that their comments add value to the agency’s decision making process.

  • Helping to sort comments is the biggest win and selling point for agencies. However, the goal/scope of this project is focused on helping to enhance the commenting experience for industry and the public.

  • On-boarding is important. Users should not need to be trained by agency staff to use this tool.

Prioritization and success metrics

Using the list of problem areas identified by the user research, we voted on what areas we thought would have the most impact on in the next 2 months of the project.

  • Forms of commenting (15 votes)
  • Missing context (8 votes)
  • Effective commenting (6 votes)
  • Feedback loop (1 vote)

Success metrics

  • Time/cost spent processing comments for a similar “sized” rule (in the past) compared to the proposed eManifest User Fee rule.
  • Number of substantive comments
  • How many repeat users use it (long term)
  • Do you feel like you were heard? (Net promoter score style rating)

User stories

Forms of commenting

  • As a commenter, I want to be able to view and read the entire preamble of the Notice of Proposed Rulemaking so that I can fully understand the proposal.
  • As a commenter, I want to be able to view the changes between the current regulation and the new proposed changes so that I can see what has changed and focus my energy on the differences.
  • As a commenter, I want to be able to jump around through sections of the rule when one is referenced by another so that I can quickly navigate through the sections.
  • As a commenter, I want to be able to answer a direct question the agency is asking in the rule so that I can provide effective feedback to the regulator’s needs and to make my voice heard.
  • As a commenter, I want to be able to comment on any part of the proposed rule that I want so that I can register my voice clearly on the proper part of the regulation or proposed rule. This will provide clear context for regulators to understand.
  • As a member of the agency who has to sort comments, I want to know where in the rule the commenter was when they submitted their comments so that I can better sort the comments into buckets.
  • As a professional commenter, I want to be able to string different comments together so that I can tell one overall story through different issues.
  • As a member of the agency staff, I want to collect some information on the commenter/organization so that I can understand whom these comments are coming from.
  • As a rule writer, I want to present the proposal in a way that clearly points out the part of the proposal I’m looking for comment on so that it is easier for commenters to understand what I’m asking and stay on track in their comments.
  • As a commenter, I want to be able to upload other documents such as excel/csv files so that I can support my claims with data.

Missing context

  • As a commenter, I want to be able to see documents that are referenced in the rule so that I can understand the context and evidence, specially if they are being used as foundations to the rule’s argument or referenced for historic reasons.
  • As a commenter, I want to know when the comment period ends, and how much time I have left to comment, so that I can plan when to drop comments.
  • As a commenter, I want to understand what other supporting documents the agency has included in their docket so that it can help me understand the context and motivations around changes in the rule.
  • As a commenter, I want to understand the current published regulation so that I can understand what’s changing in the proposal.
  • As a commenter, I want to know how the agency plans to organize and track my comments so that I can write my comment to make it easiest for them to understand my views.
  • As a member of the agency who has to sort these comments, I want a tool that will help automatically organize them using Natural Language Processing or a similar technology so that I can focus on analysis.
  • As a member of the agency who has to sort these comments, I want the ability to digitize and cross reference all comments to remove duplicates sent in paper, CD and through the web form so that I can organize them, save time and focus on analysis and responding to comments.
  • As a commenter or as an analyst, I want to be able to sort through the comments easily so I can see what different people and groups are saying on the subject.

Effective commenting

  • As a member of the agency who has to respond to comments, I want less comments that express short opinions (i.e. “This sucks.” or “I argee.”) and more substantive comments so that we improve the signal to noise ratio and get valuable, constructive feedback.
  • As a commenter, I want to understand how to write an effective comment so that the agency hears and uses what I have to say.
  • As a member of the agency who manages comments, I want to not have to do any extra work or training to get members of the public to intuitively understand how to comment, so that I can do my job easier and faster and more effectively.

Feedback loop

  • As a commenter, I want to be notified that my comment was submitted successfully to the agency, so that I feel my voice has been heard.
    • Informative confirmation screen
  • As a commenter, I want to be notified when my comment has been cleared and posted, so that I can be assured my voice has been heard and I know other commenters can see it.
  • As a commenter, I want to see my comment in relation to everyone else’s comments, so that I can see how other’s feel in relation to my views.
  • As a commenter, I want to know when the comment period is closing so that I’ll know when my comment will be posted and when the deadline to submit is by.
  • As a commenter, I want to know when the next action or rule is published so that I check to see how the agency responded to my comment..

Open discussion (out of scope for this pilot)

  • As an employee of an advocacy group, I want to be able to send members a link to my organization’s comment so that they can respond and add their thoughts and the agency can get a more personalized view of my member’s thoughts on the subject and I don’t have to create a separate form letter tool to collect these comments.
  • As a commenter, I want to be able to comment on other people’s comments so that I can have an open discussion on the issue and help the agency understand multiple sides to one issue.
  • As a commenter, I want to see popular discussions float to the top so I can see which parts of the rule is the most interesting or controversial.

General background