Merge pull request #622 from gsa-oes/2307B

2307b

_projects/2307B-form-modifications-multiple-income-households.md

@@ -1,5 +1,5 @@
 ---
-title:  Evaluating form modifications to make it easier for multiple income households to report income on benefits applications
+title: Modifying income reporting on benefits applications — Evidence from households with multiple jobs
 permalink: /projects/2307B-form-modifications-multiple-income-households/
 tags: 
  - project
@@ -8,17 +8,29 @@ image: /assets/img/project-images/2307B-image.webp
 image_alt_text: A person using a calculator
 image-credit: https://www.pexels.com/photo/a-person-sitting-a-wooden-table-with-a-notepad-and-laptop-using-a-calculator-6963847/
 analysis-plan: /assets/analysis/2307B-analysis-plan.pdf
-abstract: 
+abstract-web: /results/income-reporting-mods-multiple-income/
 year: 2024  
 domain:
  - Government Operations
 type: Impact evaluation of program change
 agency: Agriculture
-status: Ongoing
-summary: Analysis Plan Registration
+status: Complete
+summary: Providing applicants with a more structured way to report their income did not increase submission rates for an online application for SNAP benefits
 ---
-This evaluation is currently being implemented. We have created this project page as a mechanism to pre-specify what data will be collected, what we plan to measure, and how we’ll conduct our analysis. We believe this is a critical component of conducting transparent, replicable, and high-quality research; and aim to share our Analysis Plans whenever possible.
+## What is the agency priority?
+The Supplemental Nutrition Assistance Program (SNAP) is America’s largest nutrition program. In fiscal year 2023, SNAP provided an average of nearly $200 a month in food assistance to 1 in 8 Americans. To receive those benefits, individuals apply on behalf of their household with their local county or state SNAP office. SNAP applications can be long and burdensome to complete, meaning applicants may struggle to make it through the process to receive benefits for which they are eligible. Reducing barriers in access to SNAP is a priority of USDA.
 
-The Analysis Plan at the right indicates the date locked, and you can verify our upload date <a class="usa-link usa-link--external" href="https://github.com/gsa-oes/office-of-evaluation-sciences/commits/master/assets/analysis/2307B-analysis-plan.pdf">on Github</a>. 
+## What did we evaluate?
+We collaborated with Code for America (CfA) on an evaluation to make it easier for SNAP applicants to report their income on a “digital assister” version of the application, designed by CfA.
 
-Check back for results!
+Together, we designed an alternative income reporting option that provides more structure to how applicants enter their income in the application. The goal of the evaluation was to make it easier for applicants to report their income by splitting a difficult task into smaller components.
+
+## How did the evaluation work?
+The evaluation ran from June 6th – August 15th, 2024. All SNAP applicants who made it to the income reporting module during the study and who reported having more than one job (<i>N</i>=20,549) were randomly assigned to one of two income reporting options. In the unstructured reporting condition (<i>n</i>=10,179), applicants were asked to report their total household income from the last 30 days in a single text box. In the structured reporting condition (<i>n</i>=10,370), applicants were asked to report the income from each job for each earner in their household. For example, in a household with two earners who each have two jobs, applicants would start with one earner, enter the income from each of that person’s jobs, and then do the same for the second earner. For each job, applicants would select the pay frequency and enter the amount earned per pay period. Applicants in the structured reporting condition also had the option to estimate the income of each earner in the household if they preferred not to enter their income from each job.
+
+## What did we learn?
+We found that application submission rates were lower in the structured reporting condition (57.9%) compared with the unstructured reporting condition (58.9%), but this difference was not statistically significant. However, some subgroups of applicants (those who had one household member and those who applied on a mobile device) were less likely to submit the application when they were assigned to the structured reporting condition. Overall, applicants in the structured reporting condition also reported significantly more income on average. As a result, these applicants  were more likely to report income that was over the threshold for estimated SNAP eligibility.
+
+The results indicate that the structure of income reporting did not have a meaningful overall effect on application submission rates for applicants in households with multiple jobs. However, adding more structure did reduce application submission for some groups of applicants, and overall, it led applicants to report more income on average.
+
+Verify the upload date of our analysis plan <a class="usa-link usa-link--external" href="https://github.com/gsa-oes/office-of-evaluation-sciences/commits/master/assets/analysis/2307B-analysis-plan.pdf">on Github</a>.

_projects/results/2307B-abstract.md

@@ -0,0 +1,61 @@
+---
+title: Modifying income reporting on benefits applications - Evidence from single-income households
+layout: page
+permalink: /results/income-reporting-mods-multiple-income/
+tags: 
+  - abstract
+share_image: /assets/img/project-images/2307B-image.png
+image: /assets/img/project-images/2307B-image.webp 
+image_alt_text: Hands typing on laptop
+summary:
+
+---
+<br>
+<a class="usa-button" href="https://oes.gsa.gov/assets/abstracts/2307B-multiple-income-reporting-mods-abstract.pdf" target="_blank">View the PDF Version</a>
+
+## Key findings
+Providing applicants with a more structured way to report their income from multiple jobs did not meaningfully affect application completion rates in a “digital assister” SNAP application.<sup>1</sup> However, applicants in the structured income reporting condition reported more income on average and, as a result, were 3.5 percentage points more likely to report income above the threshold for estimated SNAP eligibility.
+
+## Agency priority
+The Supplemental Nutrition Assistance Program (SNAP) is America’s largest nutrition program, providing an average of nearly $200 a month in food assistance to 1 in 8 Americans.<sup>2</sup> SNAP is a U.S. Department of Agriculture (USDA) program that is administered by state agencies. To receive these benefits, individuals apply on behalf of their household with their local county or state SNAP office. SNAP applications can often be completed online for ease of access, but even online applications can be long and difficult to complete. Administrative burdens experienced by applicants throughout the benefits application process may prevent them from applying for and receiving valuable financial support.<sup>3</sup>
+
+Eliminating barriers in access to SNAP is <a class="usa-link usa-link--external" href="https://www.usda.gov/sites/default/files/documents/usda-learning-agenda.pdf">a priority of USDA</a>. Reducing burdens in forms and applications is one pathway towards improving the delivery of federal services.
+
+## Program change description
+Code for America (CfA, our collaborator for this work) created a digital assister to help SNAP applicants apply for benefits more easily in one state administering SNAP. In the business-as-usual digital assister, SNAP applicants with income from multiple jobs are asked to report their total household income from the last 30 days in a single, open text box. 
+
+In an analysis of six months of baseline data (from September 2023 to February 2024), we found that SNAP applicants whose households receive income from multiple jobs were 12.6 percentage points less likely to submit the application compared to those with income from only one job. Applicants with income from multiple jobs may face unique barriers with unstructured income reporting, such as not knowing exact income amounts from each job, and having to sum their income across different sources and pay schedules.<sup>4,5</sup> 
+
+We worked with CfA to design an alternative income reporting option for applicants with multiple jobs that provides more structure for applicants navigating the income reporting process. Our primary aim was to make it easier for applicants to report their income by splitting an otherwise difficult task into smaller components. Additionally, providing more structure to income reporting can reduce errors that come from rounding or omitting some sources of income.<sup>6</sup>
+
+## Evaluation design
+The evaluation ran from June 6th – August 15th, 2024. All SNAP applicants who made it to the income reporting module during the study and who reported having more than one job (N=20,549) were randomly assigned to one of two income reporting options.<sup>7</sup>
+
+In the <b>unstructured reporting condition</b> (<i>n</i>=10,179), applicants were asked to report their total household income from the last 30 days in a single text box. In the <b>structured reporting condition</b> (<i>n</i>=10,370), applicants were asked to report the income from each job for each earner in their household. For example, in a household with two earners who each have two jobs, applicants would start with one earner, enter the income from each of that person’s jobs, and then do the same for the second earner. For each job, applicants would select the pay frequency and enter the amount earned per pay period. Applicants in the structured reporting condition also had the option to estimate the income of each earner in the household if they preferred not to enter their income from each job.
+
+## Analysis of existing data
+CfA collects all responses to its SNAP digital assister, along with indicators of condition assignment, time spent on the application, and whether an applicant ultimately submitted their application. The primary outcome of interest was whether the applicant submitted the application.
+
+## Results
+We found that application submission rates were lower in the structured reporting condition (57.9%) compared with the unstructured condition (58.9%), but this difference was not statistically significant (p=.135, 95% CI [-2.38, 0.32]).<sup>8</sup>
+
+We ran additional analyses comparing the unstructured and structured reporting conditions on a set of exploratory outcomes. Applicants in the structured reporting condition reported an average of $223.90 more income (p<.001, 95% CI [153.62, 294.19]). These applicants were also 3.5 percentage points more likely to report income that was over the threshold for SNAP eligibility (p<.001, 95% CI [0.02, 0.05]).<sup>9</sup>We were not able to determine whether there was a difference in approvals for SNAP benefits between the unstructured and structured reporting conditions.<sup>10</sup>
+
+The exploratory analyses also indicate that application submission rates among some subgroups of applicants differed between the structured and unstructured reporting conditions. Applicants with only one household member and applicants who used the digital assister on a mobile device were significantly less likely to submit the application when they were in the structured reporting condition compared with the unstructured reporting condition. Among applicants with more than one household member and applicants who used a desktop device, there was no difference between the structured and unstructured reporting conditions.
+
+## Implications
+Our results indicate that the structure of income reporting in a digital assister does not have a meaningful overall effect on application submission rates for applicants in households with multiple jobs. This suggests that other barriers such as gathering and verifying income information make it difficult for these applicants to complete federal forms and benefits applications. For example, applicants may not know the exact earnings of other members of their household, or may have a hard time keeping track of earnings records from multiple jobs, making it difficult for them to complete the application — regardless of how income is reported.
+
+Although assignment to the structured income reporting condition did not appear to affect submission rates on average, it did lead applicants to report more income. As a result, applicants in the structured reporting condition were more likely to be over the digital assister threshold for SNAP eligibility. This may be due to the structured reporting condition leading people to more accurately and fully report their income, resulting in more accurate determinations of ineligibility. On the other hand, the structured reporting condition may have led people to inadvertently over-report their income, resulting in incorrect determinations of ineligibility. Future evaluations that are able to leverage verified income data would better be able to determine whether structured income reporting options can increase reporting accuracy.
+
+## Notes:
+1. A digital assister is a tool that builds an easier-to-use interface on top of an existing form, somewhat akin to a graphical user interface.
+2. “Program Data Featured Reports, Fiscal Year 2023” U.S. Department of Agriculture Food and Nutrition Service, accessed July 18, 2024. <a class="usa-link usa-link--external" href="https://www.fns.usda.gov/pd/overview">https://www.fns.usda.gov/pd/overview</a>.
+3. Moynihan, Donald, Pamela Herd, and Hope Harvey. “Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions.” <i>Journal of Public Administration Research and Theory</i> 25, no. 1 (2015): 43 – 69.
+4. Marquis, Kent, and Jeffrey Moore. 1990. “Measurement Errors in SIPP Program Reports.” Working Paper SIPP-WP-113. U.S. Census Bureau. <a class="usa-link usa-link--external" href="https://www.census.gov/library/working-papers/1990/demo/SIPP-WP-113.html">https://www.census.gov/library/working-papers/1990/demo/SIPP-WP-113.html</a>.
+5.  Hurd, Michael, F. Thomas Juster, and James P. Smith. 2003. “Enhancing the Quality of Data on Income: Recent Innovations from the HRS.” <i>Journal of Human Resources</i> XXXVIII (3): 758–72. <a class="usa-link usa-link--external" href="https://doi.org/10.3368/jhr.XXXVIII.3.758">https://doi.org/10.3368/jhr.XXXVIII.3.758</a>.
+6. Wu, Qiong, and Liping Gu. "Comparing single-and multiple-question designs of measuring family income in China family panel studies." <i>Sociological Methods & Research</i> 53.2 (2024): 872-897.
+7. This included applicants who themselves had income from more than one job and/or who had income from more than one member of their household.
+8. For our confirmatory analysis, we ran an unadjusted model as pre-specified in our <a href="https://oes.gsa.gov/assets/analysis/2307B-analysis-plan.pdf">analysis plan</a>. The results hold in a Lin-adjusted model with a pre-specified list of covariates.
+9. Data on income and predicted eligibility are only available for applicants who made it to each of these stages of the application, meaning that these estimates do not represent the full sample and may be affected by selection bias. Notably, completion of these sections of the application does not differ between conditions.
+10. We received county approval data from CfA indicating whether or not applicants who submitted applications via the digital assister were ultimately approved or denied for SNAP. This county dataset was missing approval data for many clients, and we were unable to determine the reason for this missingness. We ran multiple models with different subsets of applicants (e.g., including all applicants, restricting the sample to applicants who were in the approval dataset) and did not find any significant estimates of the treatment effect. We may receive additional data to follow up on whether the treatment affected ultimate SNAP approval.