+
+We ran additional analyses comparing the unstructured and structured reporting conditions on a set of exploratory outcomes. Applicants in the structured reporting condition reported an average of $223.90 more income (p<.001, 95% CI [153.62, 294.19]). These applicants were also 3.5 percentage points more likely to report income that was over the threshold for SNAP eligibility (p<.001, 95% CI [0.02, 0.05]). We were not able to determine whether there was a difference in approvals for SNAP benefits between the unstructured and structured reporting conditions.
+
+The exploratory analyses also indicate that application submission rates among some subgroups of applicants differed between the structured and unstructured reporting conditions . Applicants with only one household member and applicants who used the digital assister on a mobile device were significantly less likely to submit the application when they were in the structured reporting condition compared with the unstructured reporting condition. Among applicants with more than one household member and applicants who used a desktop device, there was no difference between the structured and unstructured reporting conditions.
+
+## Implications
+Our results indicate that the structure of income reporting in a digital assister does not have a meaningful overall effect on application submission rates for applicants in households with multiple jobs. This suggests that other barriers such as gathering and verifying income information make it difficult for these applicants to complete federal forms and benefits applications. For example, applicants may not know the exact earnings of other members of their household, or may have a hard time keeping track of earnings records from multiple jobs, making it difficult for them to complete the application — regardless of how income is reported.
+
+Although assignment to the structured income reporting condition did not appear to affect submission rates on average, it did lead applicants to report more income. As a result, applicants in the structured reporting condition were more likely to be over the digital assister threshold for SNAP eligibility. This may be due to the structured reporting condition leading people to more accurately and fully report their income, resulting in more accurate determinations of ineligibility. On the other hand, the structured reporting condition may have led people to inadvertently over-report their income, resulting in incorrect determinations of ineligibility. Future evaluations that are able to leverage verified income data would better be able to determine whether structured income reporting options can increase reporting accuracy.
+
+## Notes:
+1. A digital assister is a tool that builds an easier-to-use interface on top of an existing form, somewhat akin to a graphical user interface.
+2. “Program Data Featured Reports, Fiscal Year 2023” U.S. Department of Agriculture Food and Nutrition Service, accessed July 18, 2024. https://www.fns.usda.gov/pd/overview.
+3. Moynihan, Donald, Pamela Herd, and Hope Harvey. “Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions.” Journal of Public Administration Research and Theory 25, no. 1 (2015): 43 – 69.
+4. Moore, Jeffrey C., Linda L. Stinson, and Edward J. Welniak. “Income measurement in surveys: A Review.” Journal of Official Statistics 16.4 (2000): 331-362.
+5. De La Rosa, Wendy, Abigail B. Sussman, Eric Giannella, and Maximilian Hell. “Communicating Amounts in Terms of Commonly Used Budgeting Periods Increases Intentions to Claim Government Benefits.” Proceedings of the National Academy of Science 119, no. 37 (2022): e2205877119.
+6. Devoe, Sanford E., Byron Y. Lee, and Jeffrey Pfeffer. “Hourly versus salaried payment and decisions about trading time and money over time.” ILR Review 63, no. 4 (2010): 627 – 640.
+7. Iyengar, Sheena S., and Mark R. Lepper. “When choice is demotivating: Can one desire too much of a good thing?.” Journal of Personality and Social Psychology 79, no. 6 (2000): 995 – 1006.
+8. Applicants were included in the evaluation if they reported having a single job and stable income that did not fluctuate monthly, and if they were not self-employed.
+9. We evaluated this hypothesis using an F-test of joint significance, as well as pairwise tests of differences in completion rates between pairs of intervention arms.
+10. Applicable forms may be found in applications to means-tested programs, including those on this list, as well as other situations in which the public reports income to the federal government, such as tax forms and associated digital assisters.
From fc3129386f957e4b594bc16b8c3e3025fa543c69 Mon Sep 17 00:00:00 2001
From: Amanda Girard <107217322+amandagirard@users.noreply.github.com>
Date: Fri, 15 Nov 2024 12:17:36 -0500
Subject: [PATCH 03/10] Update 2307B-abstract.md
---
_projects/results/2307B-abstract.md | 33 +++++++++++++----------------
1 file changed, 15 insertions(+), 18 deletions(-)
diff --git a/_projects/results/2307B-abstract.md b/_projects/results/2307B-abstract.md
index b677e6a8a..0badde4c2 100644
--- a/_projects/results/2307B-abstract.md
+++ b/_projects/results/2307B-abstract.md
@@ -14,22 +14,22 @@ summary:
View the PDF Version
## Key findings
-Providing applicants with a more structured way to report their income from multiple jobs did not meaningfully affect application completion rates in a “digital assister” SNAP application. However, applicants in the structured income reporting condition reported more income on average and, as a result, were 3.5 percentage points more likely to report income above the threshold for estimated SNAP eligibility.
+Providing applicants with a more structured way to report their income from multiple jobs did not meaningfully affect application completion rates in a “digital assister” SNAP application.1 However, applicants in the structured income reporting condition reported more income on average and, as a result, were 3.5 percentage points more likely to report income above the threshold for estimated SNAP eligibility.
## Agency priority
-The Supplemental Nutrition Assistance Program (SNAP) is America’s largest nutrition program, providing an average of nearly $200 a month in food assistance to 1 in 8 Americans. SNAP is a U.S. Department of Agriculture (USDA) program that is administered by state agencies. To receive these benefits, individuals apply on behalf of their household with their local county or state SNAP office. SNAP applications can often be completed online for ease of access, but even online applications can be long and difficult to complete. Administrative burdens experienced by applicants throughout the benefits application process may prevent them from applying for and receiving valuable financial support.
+The Supplemental Nutrition Assistance Program (SNAP) is America’s largest nutrition program, providing an average of nearly $200 a month in food assistance to 1 in 8 Americans.2 SNAP is a U.S. Department of Agriculture (USDA) program that is administered by state agencies. To receive these benefits, individuals apply on behalf of their household with their local county or state SNAP office. SNAP applications can often be completed online for ease of access, but even online applications can be long and difficult to complete. Administrative burdens experienced by applicants throughout the benefits application process may prevent them from applying for and receiving valuable financial support.3
Eliminating barriers in access to SNAP is a priority of USDA. Reducing burdens in forms and applications is one pathway towards improving the delivery of federal services.
## Program change description
Code for America (CfA, our collaborator for this work) created a digital assister to help SNAP applicants apply for benefits more easily in one state administering SNAP. In the business-as-usual digital assister, SNAP applicants with income from multiple jobs are asked to report their total household income from the last 30 days in a single, open text box.
-In an analysis of six months of baseline data (from September 2023 to February 2024), we found that SNAP applicants whose households receive income from multiple jobs were 12.6 percentage points less likely to submit the application compared to those with income from only one job. Applicants with income from multiple jobs may face unique barriers with unstructured income reporting, such as not knowing exact income amounts from each job, and having to sum their income across different sources and pay schedules.,
+In an analysis of six months of baseline data (from September 2023 to February 2024), we found that SNAP applicants whose households receive income from multiple jobs were 12.6 percentage points less likely to submit the application compared to those with income from only one job. Applicants with income from multiple jobs may face unique barriers with unstructured income reporting, such as not knowing exact income amounts from each job, and having to sum their income across different sources and pay schedules.4,5
-We worked with CfA to design an alternative income reporting option for applicants with multiple jobs that provides more structure for applicants navigating the income reporting process. Our primary aim was to make it easier for applicants to report their income by splitting an otherwise difficult task into smaller components. Additionally, providing more structure to income reporting can reduce errors that come from rounding or omitting some sources of income.
+We worked with CfA to design an alternative income reporting option for applicants with multiple jobs that provides more structure for applicants navigating the income reporting process. Our primary aim was to make it easier for applicants to report their income by splitting an otherwise difficult task into smaller components. Additionally, providing more structure to income reporting can reduce errors that come from rounding or omitting some sources of income.6
## Evaluation design
-The evaluation ran from June 6th – August 15th, 2024. All SNAP applicants who made it to the income reporting module during the study and who reported having more than one job (N=20,549) were randomly assigned to one of two income reporting options.
+The evaluation ran from June 6th – August 15th, 2024. All SNAP applicants who made it to the income reporting module during the study and who reported having more than one job (N=20,549) were randomly assigned to one of two income reporting options.7
In the unstructured reporting condition (n=10,179), applicants were asked to report their total household income from the last 30 days in a single text box. In the structured reporting condition (n=10,370), applicants were asked to report the income from each job for each earner in their household. For example, in a household with two earners who each have two jobs, applicants would start with one earner, enter the income from each of that person’s jobs, and then do the same for the second earner. For each job, applicants would select the pay frequency and enter the amount earned per pay period. Applicants in the structured reporting condition also had the option to estimate the income of each earner in the household if they preferred not to enter their income from each job.
@@ -37,14 +37,11 @@ In the unstructured reporting condition (n=10,179), applicants were asked to rep
CfA collects all responses to its SNAP digital assister, along with indicators of condition assignment, time spent on the application, and whether an applicant ultimately submitted their application. The primary outcome of interest was whether the applicant submitted the application.
## Results
-We found that application submission rates were lower in the structured reporting condition (57.9%) compared with the unstructured condition (58.9%), but this difference was not statistically significant (p=.135, 95% CI [-2.38, 0.32]).
+We found that application submission rates were lower in the structured reporting condition (57.9%) compared with the unstructured condition (58.9%), but this difference was not statistically significant (p=.135, 95% CI [-2.38, 0.32]).8
-Figure 1. Application submission rates did not differ between the unstructured and structured reporting conditions
-
+We ran additional analyses comparing the unstructured and structured reporting conditions on a set of exploratory outcomes. Applicants in the structured reporting condition reported an average of $223.90 more income (p<.001, 95% CI [153.62, 294.19]). These applicants were also 3.5 percentage points more likely to report income that was over the threshold for SNAP eligibility (p<.001, 95% CI [0.02, 0.05]).9We were not able to determine whether there was a difference in approvals for SNAP benefits between the unstructured and structured reporting conditions.10
-We ran additional analyses comparing the unstructured and structured reporting conditions on a set of exploratory outcomes. Applicants in the structured reporting condition reported an average of $223.90 more income (p<.001, 95% CI [153.62, 294.19]). These applicants were also 3.5 percentage points more likely to report income that was over the threshold for SNAP eligibility (p<.001, 95% CI [0.02, 0.05]). We were not able to determine whether there was a difference in approvals for SNAP benefits between the unstructured and structured reporting conditions.
-
-The exploratory analyses also indicate that application submission rates among some subgroups of applicants differed between the structured and unstructured reporting conditions . Applicants with only one household member and applicants who used the digital assister on a mobile device were significantly less likely to submit the application when they were in the structured reporting condition compared with the unstructured reporting condition. Among applicants with more than one household member and applicants who used a desktop device, there was no difference between the structured and unstructured reporting conditions.
+The exploratory analyses also indicate that application submission rates among some subgroups of applicants differed between the structured and unstructured reporting conditions. Applicants with only one household member and applicants who used the digital assister on a mobile device were significantly less likely to submit the application when they were in the structured reporting condition compared with the unstructured reporting condition. Among applicants with more than one household member and applicants who used a desktop device, there was no difference between the structured and unstructured reporting conditions.
## Implications
Our results indicate that the structure of income reporting in a digital assister does not have a meaningful overall effect on application submission rates for applicants in households with multiple jobs. This suggests that other barriers such as gathering and verifying income information make it difficult for these applicants to complete federal forms and benefits applications. For example, applicants may not know the exact earnings of other members of their household, or may have a hard time keeping track of earnings records from multiple jobs, making it difficult for them to complete the application — regardless of how income is reported.
@@ -55,10 +52,10 @@ Although assignment to the structured income reporting condition did not appear
1. A digital assister is a tool that builds an easier-to-use interface on top of an existing form, somewhat akin to a graphical user interface.
2. “Program Data Featured Reports, Fiscal Year 2023” U.S. Department of Agriculture Food and Nutrition Service, accessed July 18, 2024. https://www.fns.usda.gov/pd/overview.
3. Moynihan, Donald, Pamela Herd, and Hope Harvey. “Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions.” Journal of Public Administration Research and Theory 25, no. 1 (2015): 43 – 69.
-4. Moore, Jeffrey C., Linda L. Stinson, and Edward J. Welniak. “Income measurement in surveys: A Review.” Journal of Official Statistics 16.4 (2000): 331-362.
-5. De La Rosa, Wendy, Abigail B. Sussman, Eric Giannella, and Maximilian Hell. “Communicating Amounts in Terms of Commonly Used Budgeting Periods Increases Intentions to Claim Government Benefits.” Proceedings of the National Academy of Science 119, no. 37 (2022): e2205877119.
-6. Devoe, Sanford E., Byron Y. Lee, and Jeffrey Pfeffer. “Hourly versus salaried payment and decisions about trading time and money over time.” ILR Review 63, no. 4 (2010): 627 – 640.
-7. Iyengar, Sheena S., and Mark R. Lepper. “When choice is demotivating: Can one desire too much of a good thing?.” Journal of Personality and Social Psychology 79, no. 6 (2000): 995 – 1006.
-8. Applicants were included in the evaluation if they reported having a single job and stable income that did not fluctuate monthly, and if they were not self-employed.
-9. We evaluated this hypothesis using an F-test of joint significance, as well as pairwise tests of differences in completion rates between pairs of intervention arms.
-10. Applicable forms may be found in applications to means-tested programs, including those on this list, as well as other situations in which the public reports income to the federal government, such as tax forms and associated digital assisters.
+4. Marquis, Kent, and Jeffrey Moore. 1990. “Measurement Errors in SIPP Program Reports.” Working Paper SIPP-WP-113. U.S. Census Bureau. https://www.census.gov/library/working-papers/1990/demo/SIPP-WP-113.html.
+5. Hurd, Michael, F. Thomas Juster, and James P. Smith. 2003. “Enhancing the Quality of Data on Income: Recent Innovations from the HRS.” Journal of Human Resources XXXVIII (3): 758–72. https://doi.org/10.3368/jhr.XXXVIII.3.758.
+6. Wu, Qiong, and Liping Gu. "Comparing single-and multiple-question designs of measuring family income in China family panel studies." Sociological Methods & Research 53.2 (2024): 872-897.
+7. This included applicants who themselves had income from more than one job and/or who had income from more than one member of their household.
+8. For our confirmatory analysis, we ran an unadjusted model as pre-specified in our analysis plan. The results hold in a Lin-adjusted model with a pre-specified list of covariates.
+9. Data on income and predicted eligibility are only available for applicants who made it to each of these stages of the application, meaning that these estimates do not represent the full sample and may be affected by selection bias. Notably, completion of these sections of the application does not differ between conditions.
+10. We received county approval data from CfA indicating whether or not applicants who submitted applications via the digital assister were ultimately approved or denied for SNAP. This county dataset was missing approval data for many clients, and we were unable to determine the reason for this missingness. We ran multiple models with different subsets of applicants (e.g., including all applicants, restricting the sample to applicants who were in the approval dataset) and did not find any significant estimates of the treatment effect. We may receive additional data to follow up on whether the treatment affected ultimate SNAP approval.
From 2f9f9f6d1916ca449a2358a2f534119216127cf3 Mon Sep 17 00:00:00 2001
From: Amanda Girard <107217322+amandagirard@users.noreply.github.com>
Date: Fri, 15 Nov 2024 12:31:30 -0500
Subject: [PATCH 04/10] Update 2307B-abstract.md
---
_projects/results/2307B-abstract.md | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/_projects/results/2307B-abstract.md b/_projects/results/2307B-abstract.md
index 0badde4c2..a36ec9659 100644
--- a/_projects/results/2307B-abstract.md
+++ b/_projects/results/2307B-abstract.md
@@ -52,10 +52,10 @@ Although assignment to the structured income reporting condition did not appear
1. A digital assister is a tool that builds an easier-to-use interface on top of an existing form, somewhat akin to a graphical user interface.
2. “Program Data Featured Reports, Fiscal Year 2023” U.S. Department of Agriculture Food and Nutrition Service, accessed July 18, 2024. https://www.fns.usda.gov/pd/overview.
3. Moynihan, Donald, Pamela Herd, and Hope Harvey. “Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions.” Journal of Public Administration Research and Theory 25, no. 1 (2015): 43 – 69.
-4. Marquis, Kent, and Jeffrey Moore. 1990. “Measurement Errors in SIPP Program Reports.” Working Paper SIPP-WP-113. U.S. Census Bureau. https://www.census.gov/library/working-papers/1990/demo/SIPP-WP-113.html.
-5. Hurd, Michael, F. Thomas Juster, and James P. Smith. 2003. “Enhancing the Quality of Data on Income: Recent Innovations from the HRS.” Journal of Human Resources XXXVIII (3): 758–72. https://doi.org/10.3368/jhr.XXXVIII.3.758.
-6. Wu, Qiong, and Liping Gu. "Comparing single-and multiple-question designs of measuring family income in China family panel studies." Sociological Methods & Research 53.2 (2024): 872-897.
+4. Marquis, Kent, and Jeffrey Moore. 1990. “Measurement Errors in SIPP Program Reports.” Working Paper SIPP-WP-113. U.S. Census Bureau. https://www.census.gov/library/working-papers/1990/demo/SIPP-WP-113.html.
+5. Hurd, Michael, F. Thomas Juster, and James P. Smith. 2003. “Enhancing the Quality of Data on Income: Recent Innovations from the HRS.” Journal of Human Resources XXXVIII (3): 758–72. https://doi.org/10.3368/jhr.XXXVIII.3.758.
+6. Wu, Qiong, and Liping Gu. "Comparing single-and multiple-question designs of measuring family income in China family panel studies." Sociological Methods & Research 53.2 (2024): 872-897.
7. This included applicants who themselves had income from more than one job and/or who had income from more than one member of their household.
-8. For our confirmatory analysis, we ran an unadjusted model as pre-specified in our analysis plan. The results hold in a Lin-adjusted model with a pre-specified list of covariates.
+8. For our confirmatory analysis, we ran an unadjusted model as pre-specified in our analysis plan. The results hold in a Lin-adjusted model with a pre-specified list of covariates.
9. Data on income and predicted eligibility are only available for applicants who made it to each of these stages of the application, meaning that these estimates do not represent the full sample and may be affected by selection bias. Notably, completion of these sections of the application does not differ between conditions.
-10. We received county approval data from CfA indicating whether or not applicants who submitted applications via the digital assister were ultimately approved or denied for SNAP. This county dataset was missing approval data for many clients, and we were unable to determine the reason for this missingness. We ran multiple models with different subsets of applicants (e.g., including all applicants, restricting the sample to applicants who were in the approval dataset) and did not find any significant estimates of the treatment effect. We may receive additional data to follow up on whether the treatment affected ultimate SNAP approval.
+10. We received county approval data from CfA indicating whether or not applicants who submitted applications via the digital assister were ultimately approved or denied for SNAP. This county dataset was missing approval data for many clients, and we were unable to determine the reason for this missingness. We ran multiple models with different subsets of applicants (e.g., including all applicants, restricting the sample to applicants who were in the approval dataset) and did not find any significant estimates of the treatment effect. We may receive additional data to follow up on whether the treatment affected ultimate SNAP approval.
From 6af3000058493141c1f33512064d827043ae2408 Mon Sep 17 00:00:00 2001
From: Amanda Girard <107217322+amandagirard@users.noreply.github.com>
Date: Fri, 15 Nov 2024 12:34:10 -0500
Subject: [PATCH 05/10] Update 2307B-abstract.md
---
_projects/results/2307B-abstract.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_projects/results/2307B-abstract.md b/_projects/results/2307B-abstract.md
index a36ec9659..725582cd8 100644
--- a/_projects/results/2307B-abstract.md
+++ b/_projects/results/2307B-abstract.md
@@ -31,7 +31,7 @@ We worked with CfA to design an alternative income reporting option for applican
## Evaluation design
The evaluation ran from June 6th – August 15th, 2024. All SNAP applicants who made it to the income reporting module during the study and who reported having more than one job (N=20,549) were randomly assigned to one of two income reporting options.7
-In the unstructured reporting condition (n=10,179), applicants were asked to report their total household income from the last 30 days in a single text box. In the structured reporting condition (n=10,370), applicants were asked to report the income from each job for each earner in their household. For example, in a household with two earners who each have two jobs, applicants would start with one earner, enter the income from each of that person’s jobs, and then do the same for the second earner. For each job, applicants would select the pay frequency and enter the amount earned per pay period. Applicants in the structured reporting condition also had the option to estimate the income of each earner in the household if they preferred not to enter their income from each job.
+In the unstructured reporting condition (n=10,179), applicants were asked to report their total household income from the last 30 days in a single text box. In the structured reporting condition (n=10,370), applicants were asked to report the income from each job for each earner in their household. For example, in a household with two earners who each have two jobs, applicants would start with one earner, enter the income from each of that person’s jobs, and then do the same for the second earner. For each job, applicants would select the pay frequency and enter the amount earned per pay period. Applicants in the structured reporting condition also had the option to estimate the income of each earner in the household if they preferred not to enter their income from each job.
## Analysis of existing data
CfA collects all responses to its SNAP digital assister, along with indicators of condition assignment, time spent on the application, and whether an applicant ultimately submitted their application. The primary outcome of interest was whether the applicant submitted the application.
From fb6393af3aba14c53f9ffcd04b24bac213fe5c96 Mon Sep 17 00:00:00 2001
From: Amanda Girard <107217322+amandagirard@users.noreply.github.com>
Date: Fri, 15 Nov 2024 12:38:44 -0500
Subject: [PATCH 06/10] Add files via upload
---
...multiple-income-reporting-mods-abstract.pdf | Bin 0 -> 187243 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 assets/abstracts/2307B-multiple-income-reporting-mods-abstract.pdf
diff --git a/assets/abstracts/2307B-multiple-income-reporting-mods-abstract.pdf b/assets/abstracts/2307B-multiple-income-reporting-mods-abstract.pdf
new file mode 100644
index 0000000000000000000000000000000000000000..ea940487443ce1d67d5726f55e34dd217d346349
GIT binary patch
literal 187243
zcmd?PWmJ$|w*?AKaM^_%7BVLEa*;udYddQ;a8q6V2K$|6I>(R^b
zVPZ*A8lNF{Ec?RLVMXWTgz7+;3XRq&H*F^URl4)S(5?Zbh*UGdCWk;vQ!}S!IbpUd
zDp+vafKs+DXDPO#6+b1MaysGxT#y3?};ZKb3>XnAEIbD4_gqBAJv#O^<+a}58z
zZ@s9QrGt?jov5XrgORY2fwiI0f9*Zb#QHBX>6!&_3IO!EbJrU|6s}PCtSLO9gOgvi
z?r2x06*pIoXvl5O+l&6`M}u;timYo|FQM^?5rdMJlm~0K=ntdiC!h25uq!pdxVoB^
zDp3Q<54PK;Uk5wzv_52ub)Y7$kxAXCj
zTiNyA{YWE8K~jQ4%uD$Go0u#Ab6-IpQV-Oecd^^SdpCKk{+)~D1-WD`=C&d(_x12H
z3cbHaQE~qE2=ao#i}0jg(97ogkwn(3e-Ae=0QodO5Gwo-{ef{9V$l*3(Z`d*-zz9n
z4E7FN*aw*3IuHKv_eF{ddx3V;;pfZFdpjH1<%j(e2pv5cnm70~{JUBhcc@B`r_Zz8
zlKO=&gK@#e?ry{L)$)?}#hAeu=kCYvYZVx0{SEnU07rr^J2w}$gZ+w43^E;$tIOr)
zWBEj-8T74orLU2r`x